search

icecube / skymap_scanner

A distributed system that performs a likelihood scan of event directions for IceCube real-time alerts using CPU cluster(s) and queue-based message passing.
5 stars 2 forks source link

CI Tests for Milipede Wilks #148

Closed ric-evans closed 1 year ago

ric-evans commented 1 year ago

It will be useful to have CI tests using millipede_wilks (nside=1). We may even save runtime.

mlincett commented 1 year ago

Aside from what I mentioned in #166 (CI images need to include all the required spline tables), we need to change the test structure so that each reco has its own test data.

I also suggest that the test succeeds if test data are not provided for a given reco (when a new reconstruction is introduced, we do not have test data for it until we decide it's mature enough to produce a baseline).

tianluyuan commented 1 year ago

I think the new cascade splines were removed from the Dockerfile to reduce space requirements. But there is the no_cvmfs Dockerfile, which I suppose could be used for CI? What are the constraints there?

mlincett commented 1 year ago

@ric-evans @dsschult can we switch CI to use the no_cvmfs image or do we risk hitting some storage limit for GitHub workflows?

dsschult commented 1 year ago

We might hit the storage limit. I know for a whlie we were at the limit, before some cleanup of the images.

Also, larger images slow down CI by a noticeable amount.

mlincett commented 1 year ago

We might hit the storage limit. I know for a whlie we were at the limit, before some cleanup of the images.

Also, larger images slow down CI by a noticeable amount.

I understand. For CI this means we may need a different Dockerfile for each reco_algo, since each reco needs a different set of spline tables. This requires a heavy restructuring of the current workflows, I guess.

For reco-specific development branches it is still possible to amend the Dockerfile to fetch all the required data as long as the branch is being developed, and revert it back just before merging. It is not ideal but suffices for me right now.

dsschult commented 1 year ago

The other option is to download the spline tables at runtime in the "lite" image. Which wouldn't be much different size-wise than downloading/building that layer in the image, but is more customizable to the algorithm.

mlincett commented 1 year ago

The other option is to download the spline tables at runtime in the "lite" image. Which wouldn't be much different size-wise than downloading/building that layer in the image, but is more customizable to the algorithm.

I guess we could begin with removing the spline tables from the IceTray image and fetching them in the Skymap Scanner image, see my remark in #166.

ric-evans commented 1 year ago

I'm not a fan of keeping sizeable data inside the image--unless we need to version-track that data. If we can do an at-runtime pattern, I'd like that better

mlincett commented 1 year ago

Thanks to #166 and the quick fix to #193 I think I can easily run a test scan and export the pickle files now. I am considering including this in #189 .