Inclusion of spline tables in container

[mlincett]

Currently, only the spline tables used by millipede_original are included in the Skymap Scanner docker image and I believe there are a few points that should be sorted out:

• the spline tables for millipede_original are fetched upstream, as prescribed by the Dockerfile of IceTray. It would make more sense to fetch them as part of the Dockerfile of Skymap Scanner;
• millipede_wilks and splinempe require additional spline tables; right now for millipede_wilks we override $I3_DATA in the client starter script so it points to CVMFS, however this does not work in CI;
• there is a Dockerfile no_cvmfs: shouldn't this become the default, at least for CI purposes?

[ric-evans]

see https://github.com/icecube/skymap_scanner/issues/148#issuecomment-1496459268

Preferably, I'd like to see a no-CVMFS pattern for all our use cases if possible.

[mlincett]

see #148 (comment)

Preferably, I'd like to see a no-CVMFS pattern for all our use cases if possible.

I guess by "no-CVFMS" you mean an image not including data from CVMFS? Because the current "no-CVFMS" Dockerfile is the opposite: a container that does not need CVMFS access at runtime.

So I guess we would be fetching the spline data at runtime and mount their location as a volume, right? I agree with making the "code-only" image the default (also because it is then trivial to add data on top of that if there are special requirements).

[ric-evans]

Right, I believe we agree--a dataless image.

We either mount as a volume or have the container's process fetch the data and write within the container. Then following processes would look inside the container instead of refetching.

[mlincett]

Side note: since we are going to clean up the Dockerfile, there is a bunch of commented out import statements for pieces of realtime that are no longer required and can be safely removed.

[mlincett]

After a brief chat with @ric-evans , maybe we could reconsider IceTray filestagers on the client side.

It seems to be the path of least resistance.

[mlincett]

Now that #174 is merged I suggest we make the upstream IceTray image "dataless", what do you think @dsschult ?

[dsschult]

Now that #174 is merged I suggest we make the upstream IceTray image "dataless", what do you think @dsschult ?

I agree.

https://github.com/WIPACrepo/docker-icecube-icetray/pull/10

That look good?

[mlincett]

Now that #174 is merged I suggest we make the upstream IceTray image "dataless", what do you think @dsschult ?

I agree.

WIPACrepo/docker-icecube-icetray#10

That look good?

Seems fine to me!

[dsschult]

Just checking, but how do we handle the base gcd lookups? Some similar principle? Or do we even do those anymore?

[mlincett]

Just checking, but how do we handle the base gcd lookups? Some similar principle? Or do we even do those anymore?

I have not touched that part anymore but the DataStager was designed with this also in mind. For now we still expect them to be inside the container, but the staging can/will be implemented (are GCD files on prod-exe?)

[dsschult]

Just checking, but how do we handle the base gcd lookups? Some similar principle? Or do we even do those anymore?

I have not touched that part anymore but the DataStager was designed with this also in mind. For now we still expect them to be inside the container, but the staging can/will be implemented (are GCD files on prod-exe?)

One helping thing is #150 , and then we don't need the base gcds in the client container, just at the server. That has CVMFS available for most scenarios.

I don't think the base GCD files are on prod-exe, as they are in /data/user/followup/baseline_gcds/, but that could be fixed fairly easily for a fallback stager.