search

ejhigson / dyPolyChord

Super fast dynamic nested sampling with PolyChord (Python, C++ and Fortran likelihoods).
http://dypolychord.readthedocs.io/en/latest/
MIT License
22 stars 5 forks source link

Functionality - performance #2

Closed andrewfowlie closed 6 years ago

andrewfowlie commented 6 years ago

Hi there,

For my review, I am trying to check off this requirement,

  • [ ] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

How can I verify the performance gains achieved by your code? Is there a particular example that illustrates them?

ejhigson commented 6 years ago

Thanks for pointing out this requirement to me! There are empirical tests of dyPolyChord in the dynamic nested sampling paper which I think should be sufficient. If required, the code for reproducing these tests is publicly available at https://github.com/ejhigson/dns.

I have added both a reference to these results and a link to the code to reproduce them in the submitted paper. I have also added a performance section to the documentation which discusses this.

If you are happy with this I will close the issue.

andrewfowlie commented 6 years ago

OK. @arfon Can you clarify whether the below requirement means the reviewer (me) has confirmed the claims? or that the authors have confirmed the claims?

  • [ ] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

The authors have fairly thoroughly investigated the performance of their algorithm (and in fact the implementation submitted to JOSS) in a separate publication, but the submission to JOSS doesn't itself contain examples of the performance and I haven't verified their findings.

@ejhigson If I must verify the performance claims myself, if possible, could you prepare a minimal example that shows orders of magnitude gains in computational efficiency? Preferably, the example shouldn't require great deal of CPU time.

ejhigson commented 6 years ago

@ejhigson If I must verify the performance claims myself, if possible, could you prepare a minimal example that shows orders of magnitude gains in computational efficiency? Preferably, the example shouldn't require great deal of CPU time.

I am afraid running such a test (without a somewhat contrived problem or choice or priors) is computationally expensive as the big gains are in highish dimensions and computational cost with PolyChord scales with dimension D as ~D^3. Plus to do so rigorously you need a lot of runs to measure the speedup accurately and an analytic value for the result to check you have not reduced the variation of results by introducing biases.

For this reason in the JOSS paper I only claim dynamic nested sampling (in general) "allows order-of-magnitude increases in efficiency" and that dyPolyChord (this implementation) offers "state-of-the-art performance" - i.e. better than PolyChord. I think these claims are well verified in https://arxiv.org/abs/1704.03459, and I don't think you need to rerun the tests yourself (they are computationally expensive) although I of course defer to @arfon on this.

andrewfowlie commented 6 years ago

I think I misunderstood the software paper as claiming orders of magnitude improvement by your code. Upon re-reading it, I now see it doesn’t say that. I’m happy to accept the arXiv paper as a demonstration of the weaker claim of state-of-the-art performance.