Draft: Results dictionary explanation

[jacopok]

Hi, this is a draft PR to solve #119; it is still incomplete, also due to some trouble I had when building the documentation.

The first point is that I don't know if the formatting is correct since I cannot build the docs: after installing everything and running make html I get an error in the example-line.ipynb notebook:

ValueError                                Traceback (most recent call last)
Cell In[14], line 5
      2 bins=np.linspace(0.01, 0.2, 64+1)
      3 scatter_samples = result['samples'][:,2]
----> 5 pdf, _ = fastKDE.pdf(scatter_samples, axes=(bins,))
      6 plt.plot(bins, pdf, color='k')
      8 from ultranest.plot import PredictionBand

ValueError: too many values to unpack (expected 2)
ValueError: too many values to unpack (expected 2)

You can ignore this error by setting the following in conf.py:

    nbsphinx_allow_errors = True

If I allow errors in the conf.py, however, the API docs seem not to get built at all... I am on the latest version of ultranest as well as fastkde (2.0.1) so I don't quite understand what the problem is (the pdf from fastKDE seems to only yield one array).

Anyway, about the actual dictionary entries:

• where may I find some information about the MWW test?
• what is logz_single?
• where may I find more information/a reference for the bootstrap logz estimate?

I think the rest should be mostly OK.

[JohannesBuchner]

Looks like fastkde changed API from <2.0 to 2.0. Maybe install <2.0?

[jacopok]

That fixed the error, but I'm getting another issue: the docs are built correctly, except for the API ones and the document ultranest.html (and, with it, all the auto-generated part of the docs, which would include the content of this PR). The warnings I'm seeing are:

/home/jacopo/Documents/UltraNest/docs/API.rst:44: WARNING: toctree contains reference to nonexisting document 'ultranest'
/home/jacopo/Documents/UltraNest/docs/modules.rst:4: WARNING: toctree contains reference to nonexisting document 'ultranest'
/home/jacopo/Documents/UltraNest/docs/priors.ipynb:201: WARNING: duplicate label priors:specifying priors, other instance in /home/jacopo/Documents/UltraNest/docs/priors.ipynb
/home/jacopo/Documents/UltraNest/docs/using-ultranest.ipynb:226: WARNING: File not found: 'ultranest.rst#ultranest.integrator.ReactiveNestedSampler'
/home/jacopo/Documents/UltraNest/docs/using-ultranest.ipynb:453: WARNING: File not found: 'ultranest.html#ultranest.integrator.ReactiveNestedSampler'
/home/jacopo/Documents/UltraNest/docs/using-ultranest.ipynb:453: WARNING: File not found: 'ultranest.html#ultranest.integrator.ReactiveNestedSampler.run'
looking for now-outdated files... none found
pickling environment... done
checking consistency... /home/jacopo/Documents/UltraNest/docs/modules.rst: WARNING: document isn't included in any toctree

I figure that ultranest.html should probably be automatically generated, maybe by sphinx.ext.autodoc? I don't know why it wouldn't be generating it. For reference, this is the documentation that gets generated (I've removed the notebooks otherwise it'd be too heavy for github).

html.zip

It's a bit tricky to debug since I don't know where the ultranest.html file should be coming from...

[JohannesBuchner]

Those warnings are fine and you can ignore them

[JohannesBuchner]

yesterday I updated the docs to use the latest fastKDE

[jacopok]

I'm happy to ignore the warnings, the problem is that the part of the documentation that is being changed in this PR is not being built, therefore I'm not seeing how it will look. The changes should appear here, in the ultranest.html page, and since they include some nontrivial formatting (e.g. LaTeX equations) I'd like to preview them.

[JohannesBuchner]

The Makefile lists how you should build the documentation:

docs: ## generate Sphinx HTML documentation, including API docs
    rm -f docs/ultranest.rst
    rm -f docs/modules.rst
    rm -f docs/API.rst
    python3 setup.py build_ext --inplace
    sphinx-apidoc -H API -o docs/ ultranest
    cd docs; python3 modoverview.py
    $(MAKE) -C docs clean
    $(MAKE) -C docs html O=-jauto
    sed --in-place '/href="ultranest\/mlfriends.html"/d' docs/build/html/_modules/index.html
    sed --in-place '/href="ultranest\/stepfuncs.html"/d' docs/build/html/_modules/index.html
    $(BROWSER) docs/build/html/index.html

sphinx-apidoc builds ultranest.rst

[jacopok]

Got it, sorry about that, I was only looking in the docs folder assuming it was self-contained. Will fix the formatting now.

[JohannesBuchner]

Maybe this documentation part https://johannesbuchner.github.io/UltraNest/performance.html#output-files should then also point to the API docs to avoid duplication.

[jacopok]

Indeed! although there are some differences, I can draft a change.

Some other points that are not super clear to me (I've never delved this deep into nested sampling before):

In the combine_results function it seems like logz and logz_single are set to the same value, right? what is logz_single supposed to be? Also, what is the relevance of the logz_tail contribution?

I was not able to find information about the application of the MWW U-test in this context.

Also, I'm a bit confused about the difference between logw and weights: it seems like ultimately the "weights" are obtained as $\exp(\log w + \log L - \logZ)$, so does logw only account for the volume contribution?

[JohannesBuchner]

Indeed! although there are some differences, I can draft a change.

Thanks for your dedication here!

Some other points that are not super clear to me (I've never delved this deep into nested sampling before):

In the combine_results function it seems like logz and logz_single are set to the same value, right? what is logz_single supposed to be? Also, what is the relevance of the logz_tail contribution?

logz_single +- logzerr_tail is the classic estimator logz +- logzerr is the improved UltraNest estimator, including bootstrapping uncertainties.

I was not able to find information about the application of the MWW U-test in this context.

This was presented in section 4.5.2. in https://arxiv.org/pdf/2101.09675

Also, I'm a bit confused about the difference between logw and weights: it seems like ultimately the "weights" are obtained as exp⁡(log⁡w+log⁡L−\logZ), so does logw only account for the volume contribution?

IIRC they are the same (volume * loglikelihood) but one is in log.

[jacopok]

logz_single +- logzerr_tail is the classic estimator logz +- logzerr is the improved UltraNest estimator, including bootstrapping uncertainties.

I see! So, I've added a recommendation to users to use logz +- logzerr (if that was not obvious from the key name), I don't know if much more detail on the other estimates of the evidence error is needed

This was presented in section 4.5.2. in https://arxiv.org/pdf/2101.09675

Great, thanks! Added the reference.

Also, I'm a bit confused about the difference between logw and weights: it seems like ultimately the "weights" are obtained as exp⁡(log⁡w+log⁡L−\logZ), so does logw only account for the volume contribution?

IIRC they are the same (volume * loglikelihood) but one is in log.

I'm looking here and it seems like they're not: the weights come from saved_wt0, and the log-weights come from saved_logwt0, but the relation is not a simple log...

[JohannesBuchner]

Ah, yes. saved_wt0 is the (importance) weight for each dead point, saved_logwt0 are the log-volumes for each dead point.

[JohannesBuchner]

Recently I added some basic code format checks to reduce post-processing PRs on my side. You might have seen that the new code checks failed. I just pushed a commit to master which fixes these issues. If you update, it may pass.

[jacopok]

Ok, tomorrow I will rebase!

[jacopok]

I've rebased and fixed some whitespace, now flake8 passes locally on my integrator.py (though not on several other files in tests, as well as ultranest/dyhmc.py and ultranest/flatnuts.py - maybe it's a settings issue?)

Also, now I'm complying with pydocstyle and pycodestyle

[jacopok]

Maybe the commands pycodestyle and pydocstyle $(ls ultranest/*.py | grep -Ev '^ultranest/(flatnuts|dychmc|dyhmc|pathsampler).py') should be added to the Makefile under lint? otherwise I can see a clean lint locally but get errors in the CI.

[JohannesBuchner]

Great. I use a git pre-commit hook with those three lines, but probably make lint would be better

[JohannesBuchner]

Thank you for your contribution! I added your name to the contributor list in the README.