Resuming of interrupted runs

[segasai]

This a preliminary version of code that allows resuming of dynesty runs. The current interface is this is to run:

    dns.run_nested(checkpoint_file='xx.pkl')

And to resume

   dns = dds.DynamicSampler.restore('xx.pkl')
    dns.run_nested(resume=True)

This was used to test it

import dynesty
import dynesty.dynamicsampler as dds
import numpy as np
import multiprocessing as mp
import os, time

def like(x):
    return -.5 * np.sum(x**2)

def ptform(x):
    return 20 * x - 10

def interrupter(pid, dt):
    time.sleep(dt)
    os.kill(pid, 9)

def start_interrupter(pid, dt):
    pp = mp.Process(target=interrupter, args=(pid, dt))
    pp.start()
    return pp

def fit():
    ndim = 2
    dns = dynesty.DynamicNestedSampler(like,
                                       ptform,
                                       ndim,
                                       nlive=100,
                                       rstate=np.random.default_rng(1))
    dns.run_nested(checkpoint_file='xx.pkl')

def fit_resume():
    dns = dds.DynamicSampler.restore('xx.pkl')
    print('resuming')
    dns.run_nested(resume=True)

def doall():
    pp = mp.Process(target=fit)
    pp.start()
    fit_pid = pp.pid
    delay = 15
    pp = start_interrupter(fit_pid, delay)
    time.sleep(delay + 1)
    fit_resume()

doall()

There are quite a few things that need fixing.

• Implement the same for a static sampling
• add an actual test to the test suite (including the test that we get exactly the same results if interrupted)
• make sure that the checkpoint file is not saved every iteration ( but every N iteratios or time)
• check whether the interface is good
• prevent resuming if dynesty version doesnt match

I only implemented interruption for run_nested interface. I don't plan to implement interruption of an iterator interface.

[coveralls]

Pull Request Test Coverage Report for Build 2957311805

• 319 of 340 (93.82%) changed or added relevant lines in 7 files are covered.
• 2 unchanged lines in 1 file lost coverage.
• Overall coverage increased (+0.04%) to 90.567%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
py/dynesty/dynamicsampler.py	130	133	97.74%
py/dynesty/utils.py	41	49	83.67%
py/dynesty/dynesty.py	132	142	92.96%
<!--	Total:	319	340	93.82%	-->

Files with Coverage Reduction	New Missed Lines	%
py/dynesty/sampler.py	2	91.16%
<!--	Total:	2		-->

Totals
Change from base Build 2911341179:	0.04%
Covered Lines:	3831
Relevant Lines:	4230

---

💛 - Coveralls

[segasai]

This is starting to get ready for review.

• The static nested sampling also now supports resuming
• The test verifies that the exact logz doesn't change if restarted
• Warning is emitted if the versions don't match.

[segasai]

Thanks for taking a look!

• I agree the docs/examples are currently completely missing. I'll aim to work on that given the main part of the patch is done
• I also agree on prominent warnings that this may a breaking change for some use cases (although none of the existing tests required changes)
• I think this will require a major version change (at least 1.3 )
• Given a major change I'll also think about other significant changes I had in mind. One key thing I've been keeping delaying is adding a Results.samples_equal() method to allow getting equally weighted samples directly from Results class. I've had some other thoughts on refactorings (specifically segregating each 'inner sampler' into a separate classes I.e. Rwalk() Rslice(), Slice() so it'd be easier to create new ones and understand what is the specific interface needed there and avoid keeping adding options like slices= walks= etc to dynesty methods. But I probably won't have time for this refactoring.

[segasai]

I think this is ready for a final review and merge.

If we feel that more docs/examples are needed that may be done after the merge. The docs for the branch should be available here https://dynesty.readthedocs.io/en/resuming/

[segasai]

The only thing extra I can think of here, is of some way of usilng say dill instead of pickle. I don't quite know if it's better to just always use dill or think of an option to switch the pickler. (but this stuff should go after this is merged)

[segasai]

Hi @joshspeagle If you could take a look/rereview the PR that would be great, otherwise, I will aim to merge this early next week. Thanks

[segasai]

Thanks, @joshspeagle ! Merging