search

LSSTDESC / rail_base

Base classes for RAIL
MIT License
0 stars 1 forks source link

Check file existence when reading yaml #6

Open jfcrenshaw opened 2 years ago

jfcrenshaw commented 2 years ago

When RAIL is reading the yaml to set up the pipeline, it should check (at the very beginning!) if the stated output files already exist and refuse to overwrite them, unless flag OVERWRITE=True

OliviaLynn commented 1 year ago

I want to clarify how this should behave with respect to "resume" mode, which I understand to be a flag one can set so the pipeline will "ignore stages if all of their output already exists" (described in "Initialize the pipeline" step of Pipe_Example.ipynb, and included as a value set in pipe_example.yml). This mode seems pretty similar to me.

If we add a flag a user can toggle when reading and running a pipeline, should this just completely override the value of resume mode specified in the pipeline's .yml when the OVERWRITE flag has been set to either true or false, but default to the pipeline's resume mode if we chose not to include overwrite as a keyword?

@aimalz I remember we spoke about this but only briefly - I was wondering if you have any thoughts.

Edit to add: maybe this just needs unit tests to confirm that this flag is actually working?

aimalz commented 1 year ago

This might be a good place to also check if input files exist where they're expected to, and it might be related to resolving occurrences of stages needing their input files to be in src/rail.