search

joshspeagle / dynesty

Dynamic Nested Sampling package for computing Bayesian posteriors and evidences
https://dynesty.readthedocs.io/
MIT License
357 stars 77 forks source link

API to save result #144

Closed gapplef closed 5 years ago

gapplef commented 5 years ago

127

If you want to save the state of the sampler, just pickle-ing the whole thing should work.

So current, the result of dynesty can only be pickled and saved manually. Is it possible to have an API similar to the Backends new in emcee 3.0 to save the state in HDF5 files.

joshspeagle commented 5 years ago

I think it's a good idea to implement like this that would dump the results to disk via HDF5 as you go. I have something similar in some other code I use (and have appreciated it immensely), and this seems like a nice quality-of-life feature.

I'm currently traveling, but hopefully I can get to this sometime early next month.

gapplef commented 5 years ago

Have a nice trip! Thanks a lot for your great job on dynesty.

joshspeagle commented 5 years ago

So I dug into this. Unfortunately, I can't get a proper "backend" up and running due to issues with how I structured the code internally (i.e. it's not nearly as clean as emcee). I might be able to get something that stores some elements of the sampler as an HDF file that dumps on the fly, but anything that would allow you to, e.g., reconstruct the sampler at iteration X and/or continue sampling where you left off after reading in from memory is extremely difficult. Not totally impossible, but something that would require a significant time investment to set up. I might return to this in the future, but for now I'll have to indefinitely punt on this. Sorry...