c3-time-domain / SeeChange

A time-domain data reduction pipeline (e.g., for handling images->lightcurves) for surveys like DECam and LS4
BSD 3-Clause "New" or "Revised" License
0 stars 4 forks source link

Allow different code_versions #317

Open guynir42 opened 1 week ago

guynir42 commented 1 week ago

In principle we can have different code versions for each process step, e.g., code version "extraction_v1.0.1" and "coaddition_v2.1.3" can coexist and they would just make different provenances.

The advantage is that if you update the code for a downstream provenance (e.g., for "measuring") you don't have to make new provenances (and new data) for all the upstream products like images, subtractions, etc.

The provenance model handles this ok, but there are a lot of places where we make the provenances, where we just grab the code version from the upstream provenance, or grab the "latest" code version. All these things will get more complicated, and probably require that we make some function that grabs the "latest" code version for a specific process step (is that going to be a new column on CodeVersion maybe?).

Also, if you don't want the "latest" version, you'd have to have a way to pass that requirement into the function that grabs the code version and be able to specify some other version. This can come up, for example, if you are deliberately trying to recreate a provenance tree of older data. It would not be enough to know the parameters, you'd have to specify the code version explicitly, too. I don't know how often this would come up...