search

c3-time-domain / SeeChange

A time-domain data reduction pipeline (e.g., for handling images->lightcurves) for surveys like DECam and LS4
BSD 3-Clause "New" or "Revised" License
0 stars 4 forks source link

Measurements test fails on GA but not locally #306

Closed guynir42 closed 5 months ago

guynir42 commented 5 months ago

The test tests/models/test_measurements.py::test_filtering_measurements consistently fails on GA but passes locally.

The place where it fails is that only 6 out of 7 measurements that pass all cuts have positive flux. On my local machine there are 8 out of 8 measurements that have positive flux.

I have added some printouts to find where the differences appear. It looks like the new image and the images used to build the reference image are the same locally and on GA. But the coadd image used as a reference is different (sampling some random pixels we see slightly different values).

There is probably something different in the way alignment (or coaddition) works on the dockerized version vs my local machine.

I have a hunch it is due to the new alignment settings, maybe being passed in through a config file that I am missing.

For now I'm marking this test as skip, so I can continue developing, but we need to figure this out.

guynir42 commented 5 months ago

I think the reason for the differences is because I have older versions of SCAMP and SWARP on my machine (and no, I was not able to apt-get install the newer versions despite having updated my dpkg and everything).

The tests were borderline passing because of having many spurious detections. After having added proper background subtraction to the images going into the coaddition and subtraction codes, the number of bad detections has dropped by about x2 and the tests seem to be passing both locally and on GA.

I think we should be trying to write more robust tests and avoid matching to specific values (that may be different on different machines).

I'm closing this for now as I don't think there's much more to do here.