New classes for SampleBasedMetric, custom signatures for plots and summarizer dependence. Also adds automated unit tests

[CompiledAtBirth]

Hello, below is the list of changes in this PR:

• PosteriorCoverage and PosteriorSamples classes in ili.validation.metrics. The former combines Rank Statistics and TARP issue 56 and the latter allows to save posterior samples for ulterior use issue 50.
• Dataset class added in ili.dataloaders.loaders to remove, at least for now, ili-summarizer dependency from issue 58.
• ensemble_mode argument added in ili.validation.runner. The default behavior for a validation runner after loading an ensemble posterior (eg with a trained MAf and a trained MDN) is to compute the metrics for the NeuralPosteriorEnsemble sbi instance. In this case, the posteriors in the ensemble have their val_log_prob on validation data as weights. Alternatively, we can now compute the metrics for each trained posterior in the ensemble using ensemble_mode = False. This has been added in the example toy_sbi. I have not dealt with the pydelfi backend.
• added signatures and str_save in ili.inference.runner_sbi, and signatures for ili.validation.runner. For ease of usage, these signatures are passed as arguments to call functions in ili.validation.metrics instead of being attributes.
• In addition to the toy_sbi*.py examples, I also modified the tutorial notebook so that it works with the PosteriorCoverage class. I have not taken the time to modify nor rerun the third part of the tutorial using CAMELS data.

Please share your thoughts, especially about the ensemble_mode feature which I took the initiative to implement to deal with the case where we want to compare different models (MAF, MDN...) in the same sbi runner instance.

[maho3]

Overall, this PR is huge. Really great work. This cleans up many of the details that we needed fixed a while ago. Thanks Nicolas

The ensemble_mode thing is a really good idea. That will be really useful for checking multiple architectures at once. People may just end up using ensemble_mode=True most of the time, but the option will be very useful for trial-and-error.

I need to do a thorough review this week of all these changes. Here are some checklist for myself:

• [x] The ili_summarizer Dataset class might make more sense sitting in ili/utils, since its not a dataloader itself.
• [ ] Integrate or catch the ensemble_mode parameter in the pydelfi version
• [x] Test PosteriorCoverage and PosteriorSamples
• [x] Test signature functionality.
• [ ] Run full tutorial again
• [ ] Update INSTALL.md and related docs, removing ili-summarizer dependency.
• [ ] Add signature and ensemble_mode explanations to CONFIG.md

[CompiledAtBirth]

Thank you Matt for the comments and the checklist. @DeaglanBartlett, since issue #68 involves a lot of work for the test suite, do you want to link the issue here as well before thinking about merging? I hope the fact that the PosteriorCoverage class replaces the rank stats and TARP metrics, among other things, is not a pain to rewrite existing validation scripts.

[DeaglanBartlett]

@CompiledAtBirth yes it makes sense to wait for this PR to be approved before making a PR for #68 so that the changes made here can be added to the test suite.

[CompiledAtBirth]

Thank you Deaglan for all the changes. I just added the possibility to use existing sims at first round for multiround inference (for working example 2).

[maho3]

I think this PR is nearly good to go. I've gone around, tested things, and cleaned up code without changing much functionality. I've also rerun the tutorial.ipynb notebook and updated the documentation. Also, I've automatically added '"_"''s to every strsave and signature, so we don't need to add trailing 's in the config files.

However, there's two changes I'd like to recommend, but first want @CompiledAtBirth 's approval on before making them:

1) I think the meaning of str_save is a little obscure at first glance. I would recommend changing it to just name or something.

2) Instead of wrapping the NeuralPosteriorEnsemble in another layer of abstraction as in the PosteriorEnsemble class, I would recommend that we just directly assign signatures as an attribute of sbi's NeuralPosteriorEnsemble.

For example, in its usage for saving in this line, we can just do something like:

posterior = NeuralPosteriorEnsemble(
    posteriors=posteriors,
    weights=val_logprob
)
...
posterior.signatures = signatures

Then, when we pickle NeuralPosteriorEnsemble later, signatures should be saved with it.

This way, we don't have to do the awkward thing of e.g. posterior_ensemble.ensemble.sample to access the useful NeuralPosteriorEnsemble functions, as in the jupyter tutorial. This will also retain consistency with the pydelfi interface.

Just some recommendations, but I'm curious what you think. After these are addressed and changes are propogated through the code (including rerunning tutorial), I think I'm ready to approve the PR.

[CompiledAtBirth]

@maho3 super, thank you for the thoughtful review and cleaning. I agree on both points which make usage and future code modifications more straightforward. Special mention to 2), I did not know that one can add attributes dynamically to a class instance in Python :)