New command-line tool for extending Monte Carlo runs?

[etneil]

A common workflow request for lattice simulations is just to extend an existing HMC run (gather more statistics!) This would usually amount to extending an existing Monte Carlo chain with identical parameters, or sometimes (when thermalizing e.g.) with one or two parameter changes. This seems like a nice candidate for one of the command-line tools.

On measurements: if there are measurements attached, it's probably useful to have those duplicated by default as well. So the tool would have to:

• Find the end of a particular HMC stream (is there a nice way to specify these from the command line? Task number?)
• Read the HMC object and make a new one continuing from it
• Read any measurements attached and add them as well (?)

(If this is getting too involved, then we could avoid the last step; once measurements get involved it's probably easier for the user to just edit the script they used to generate the jobs in the first place in order to extend. But something simple to extend and tweak runs while thermalizing might be nice...)

[dchackett]

One possible implementation of this would be allowing the user to run the run-specification script again with N increased. The desired behavior would be that, when the user does this, instead of specifying a whole new longer stream, re-running the script only adds the new jobs to the dispatch, augmenting the existing stream. This should avoid any weird special-case behavior when trying to replicate measurements.