search

c3-time-domain / SeeChange

A time-domain data reduction pipeline (e.g., for handling images->lightcurves) for surveys like DECam and LS4
BSD 3-Clause "New" or "Revised" License
0 stars 4 forks source link

Secondary RefSet creation #318

Open guynir42 opened 1 week ago

guynir42 commented 1 week ago

The current code for the RefMaker is going to assume you have exactly one set of parameters for how to choose images that go into a reference.

There would be, however, situations where we may want to have a secondary configuration (or tertiary, etc). The reason is that you might find a place in the sky where many images had bad seeing and you'd rather make an exception and use fewer images. This could still be the same RefSet, but you'd want to load a reference from the secondary only when the primary is missing.

The RefSet has multiple Provenances attached to it, in the order they were created, so that you can run once with parameter set 1 (defined in the config, maybe) and then run it again with parameter set 2 (override from kwargs, perhaps). The second time you'd call RefMaker.run() will append the new provenance to the RefSet.

To actually get the Reference with the secondary config (and secondary provenance) you'd have to call RefMaker.run() two times. The first time will either raise a not enough images found, or it will return a reference but that reference will be marked by is_bad and the external calling scope will simply call the RefMaker again with new parameters. The RefMaker doens't do this loop internally, it assume the parameters it is given are the only ones.

Note that if you change the parameters of the image query code, that picks which images get loaded, then there's no limit on the number of provenances that can be attached to a RefSet (I would recommend not adding many). But if you try to add different parameters for how those images were created (or their products), e.g., by changing the extraction or coaddition parameters, this will not work. The RefMaker will not let you add those new parameters to the ref set. The reason is that those parameters go into the subtraction upstreams, and we want to make sure the downstream provenances are uniquely defined by the name of the requested ref set (e.g., there's a one-to-one relationship between RefSet and final product Provenance).

guynir42 commented 1 week ago

Important note: suppose you've run with parameter set 1 and failed to make a reference (or if that reference is marked bad for some reason) and you've decided to run parameter set 2 and gotten a reference.

The next time you'd call RefMaker.run(), for that part of the sky, regardless of the parameter set you use, you will continue to get back the reference from parameter set 2. Even if you call it with parameter set 1, and even if new images were added in the meantime, which allow you to create a (better) reference with parameter set 1.

This is because you want to only ever have one reference for each reference set for each place in the sky. You don't want to suddenly have a deeper reference in the middle of the survey. If you want to deliberately make new, better references, you should create a new ref set and switch the provenances of all the outputs to use that one instead (by changing the ref set name in the subtraction parameters).