Checkpoint/backup in case of crash

[tuanpham96]

This is a great package! I was just wondering if there is any way in the package that supports checkpoints. Sometimes, there could be a crash/error running at a certain parameter set. It would be nice if there is a checkpoint file so that re-running the parameter sweeping would not re-run the already-run sweep.

[elcorto]

Hi, thanks for your interest in the project.

I'd see this as a 2-step post-processing task. You could use a bool field named "crashed" to mark crashed runs. There are two ways to generate that.

• Add a "crashed" check to func after the workload has exited and add the "crashed" field to the dict returned by func
• Have a post-processing script add the field to the database, iterate over all psets, parse the run's output (disk) and update the pset's "crashed" value (database record)

I'd use the latter since here you have full control over post-processing you data (i.e. the criterion for what counts as crashed or how to detect a crashed state might change). You can re-run post-processing and setting the "crashed" flag as often as is needed.

Then have a second driver script (see this part of the README), that filters only crashed=True runs from the old database and re-runs those. Repeat that until no crashed runs remain.

See also this run script and this data analysis script for worked out examples (adding a column, parse output, update field values).

In your final database, you will have all crashed runs and the final good run for a specific pset (different _run_ids). In a final data evaluation script, simply use all crashed=False runs only.