Closed tuanpham96 closed 4 years ago
Hi, thanks for your interest in the project.
I'd see this as a 2-step post-processing task. You could use a bool field named "crashed" to mark crashed runs. There are two ways to generate that.
func
after the workload has exited and add the "crashed" field to the dict returned by func
pset
s, parse the run's output (disk) and update the pset
's "crashed" value (database record)I'd use the latter since here you have full control over post-processing you data (i.e. the criterion for what counts as crashed or how to detect a crashed state might change). You can re-run post-processing and setting the "crashed" flag as often as is needed.
Then have a second driver script (see this part of the README), that filters only crashed=True
runs from the old database and re-runs those. Repeat that until no crashed runs remain.
See also this run script and this data analysis script for worked out examples (adding a column, parse output, update field values).
In your final database, you will have all crashed runs and the final good run for a specific pset
(different _run_id
s). In a final data evaluation script, simply use all crashed=False
runs only.
This is a great package! I was just wondering if there is any way in the package that supports checkpoints. Sometimes, there could be a crash/error running at a certain parameter set. It would be nice if there is a checkpoint file so that re-running the parameter sweeping would not re-run the already-run sweep.