rickyegeland
commented
12 months ago It would be safer to avoid overwriting a key, expensive data output like SPHINX_dataframe at all.
Maybe --resume should write a pair of outputs – one with only new assessments, and one with the original and the new concatenated. Timestamped filenames would help the workflow management.
For example, imagine starting with outdir/pkl/SPHINX_dataframe_20231010.pkl from a previous run. Calling sphinxval --resume outdir/pkl/SPHINX_dataframe_20231010.pkl with new forecasts/observations until 2023-11-01 could produce the files ouddir/pkl/SPHINX_new_20231101.pkl with only the new assessments from this run, plus outdir/pkl/SPHINX_dataframe_20231101.pkl containing everything.
This scheme would allow us to go back and re-run from a previous checkpoint, which might be useful if isolated errors are found that are fixed in a minor bugfix release.
Dramatic version changes with new features SPHINX would start an entirely new results timeline, of course.
This ignores all the other SPHINX outputs. Perhaps the checkpoint timestamp needs to be in a directory path, not the filename.
ktindiana
When using the resume function, the SPHINX Dataframe (output/csv/SPHINX_dataframe.csv, etc) is rewritten after the new set of forecasts are processed. This version of the dataframe contains only the new forecasts and is not the desired output. In a later step, all of the results (previously input dataframe and new dataframe) are combined and then written out to output/csv/SPHINX_dataframe.csv, etc along with all the validation results. It seems non-optimal to have this writing step in the middle which could potentially lead to confusion or loss of results.
When using resume, should not write out the dataframe before combining with the previous results.