ctrl-C or other interruption during the saving of experiment.json to disk can cause corruption leading to a broken, unrecoverable experiment. Solution: saving the experiment.json should be COW-esque, i.e., we write a new experiment.json to a temp file, then atomically replace the previous experiment.json with the new one.
ctrl-C or other interruption during the saving of experiment.json to disk can cause corruption leading to a broken, unrecoverable experiment. Solution: saving the experiment.json should be COW-esque, i.e., we write a new experiment.json to a temp file, then atomically replace the previous experiment.json with the new one.