Massive performance bug with dynamic sampling

[segasai]

If I am correct there is a massive performance issue with one single line of code in the dynamic sampler https://github.com/joshspeagle/dynesty/blob/583cac5b0dfb8fd62451821e4347543f6ae95017/py/dynesty/dynamicsampler.py#L1357 This line was introduced in this PR by me : ) https://github.com/joshspeagle/dynesty/pull/326

Basically the seemingly innocuous line apparently leads to the 'iteration' counter to be high from the very beginning of the sampler that does the batch. And because the ncalls over number of iterations is evaluated as efficiency and that is one of the parameters determining whether the bounds will be updated or not, that leads to progressively slower and slower batches due to much less frequent bound updates.

This is really bad. I.e. this program runs 7 time faster with the line commented out.

import numpy as np
import dynesty
import time

nlive = 100
ndim = 5
size = 100

def loglike(x):
    r2 = np.sum(x**2)
    ret = -0.5 * r2
    time.sleep(.0001)
    return ret

def prior_transform(x):
    return (2 * x - 1) * size

def testx():
    rstate = np.random.default_rng(1)
    sampler = dynesty.DynamicNestedSampler(loglike,
                                           prior_transform,
                                           ndim,
                                           nlive=nlive,
                                           rstate=rstate,
                                           sample='unif')
    sampler.run_nested(print_progress=True, maxbatch=0)
    for i in range(5):
        sampler.add_batch(mode='full', print_progress=True)
    res = sampler.results
    print(res.logz[-1], res.logzerr[-1])

if __name__ == '__main__':
    testx()

A couple takes

• I wonder if we need some kind of performance test to catch those kind of issue, as this was embarrassing. Maybe we need an alternative way of counting the efficiency, as relying on internal counters .ncall , .it as it is potentially error-prone.
• I think the boundary update code as well as boundary tracking needs some refactoring.

[segasai]

closed with #416

[nstarman]

https://github.com/airspeed-velocity/asv used to be (is still?) popular for CI Benchmarks.

[segasai]

Yes, that sounds like a good idea. The testing infrastructure is only a year/two old, but it is now probably quite robust to start tracking.