Preamble:
The thing-currently-named-wrapper (wrapper.py) works for testing fre-cli, when we have all history files already available at the time of running. However, when we get to running production jobs, the models will be sending over bundles of history files and post-processing them in parallel. This breaks some of the logic currently in the wrapper flow - in particular, the assumptions that there's not already a pre-existing experiment belonging to the same user that the current set of history files is being added to, and the assumption that there's not already an experiment with that name running.
The logic we need is encapsulated in a flowchart at the end of this issue; this breaks it down by tool.
The tool:
Fre pp validate does not have any logic for preexisting experiments. We need to add a new logic branch that checks to see that the config (pp.yaml) matches between the prior and current experiments and updates the experiment's pp.yaml if they do not. This is the behavior that we want to have when a user discovers partway through an experiment that the config was wrong but only for parts of the experiment that have not yet run; it should come with loud warning messages about reproducing past results.
fre pp validate:
[ ] Is the pp.yaml for your prior experiment up-to-date with the current configuration?
[ ] corresponding test: not much more than a diff command
[ ] If not, update pp.yaml for the prior experiment (and save off the old config for reference)
[ ] corresponding test: pretty simple, probably involves copying around a file with the "wrong" config
Preamble: The thing-currently-named-wrapper (wrapper.py) works for testing fre-cli, when we have all history files already available at the time of running. However, when we get to running production jobs, the models will be sending over bundles of history files and post-processing them in parallel. This breaks some of the logic currently in the wrapper flow - in particular, the assumptions that there's not already a pre-existing experiment belonging to the same user that the current set of history files is being added to, and the assumption that there's not already an experiment with that name running.
The logic we need is encapsulated in a flowchart at the end of this issue; this breaks it down by tool.
The tool: Fre pp validate does not have any logic for preexisting experiments. We need to add a new logic branch that checks to see that the config (pp.yaml) matches between the prior and current experiments and updates the experiment's pp.yaml if they do not. This is the behavior that we want to have when a user discovers partway through an experiment that the config was wrong but only for parts of the experiment that have not yet run; it should come with loud warning messages about reproducing past results.
fre pp validate: [ ] Is the pp.yaml for your prior experiment up-to-date with the current configuration?