example for continuing sampling from a periodically saved trace

[unaschneck]

Hi,

Thank you for making this! I am a little unclear from the readme documentation how I would go about reloading the saved trace to continue an interrupted sampling. Ideally, I'd like to determine the total number of sampling and burn steps ahead of time and then have an ongoing save after each sample in case of a power outage or the job being interrupted in the server and reload from previously saved trace.

I am trying to implement something like

ch_client = clickhouse_driver.Client("localhost")
backend = mcbackend.ClickHouseBackend(ch_client)
trace = mcbackend.pymc.TraceBackend(backend)

continue_run = 0 # here is where I would activate a flag if the last run was not completed because the job was interrupted

with my_model:
if continue_run:
trace_id = pm.sample(***restart sampling from saved trace***)
else:
trace_id = pm.sample(draws=number_samples, tune=number_burn_samples, chains=number_walkers, cores=number_processors, return_inferencedata=True, target_accept=0.95,trace=trace) # save ongoing trace to backend

[michaelosthege]

Hi @unaschneck this is a frequently asked-for thing, but unfortunately it's not easy to restore the state of an MCMC sampler. I built McBackend to be useful for this, but it requires digging into the way how sampler state & random number generators work in PyMC. Even if you don't care about reproducibility, you'll still need to restore the tuned parameters, such as mass matrix settings of NUTS.

Also see my previous comment here: https://discourse.pymc.io/t/resuming-sampling-from-a-previous-trace/11245/5

Also, this is 100 % something that needs to be done on the PyMC side. I'll still leave this issue open so other people find it more easily.