Adding a forward-only task

[psteinb]

Hey @jan-matthis -

this is a really great project. Thanks again for making it publicly available. I really like the publication that it comes with.

For me, it would be really helpful to test drive the benchmarks in order to see how much data I need to simulate for (S)NPE. However, I only have the forward simulation - no analytical posterior. I was wondering, if it'd be helpful if I contribute such an example to this repo. In this fashion, more people can profit from this exercise. Would that be OK?

if you can, feel free to provide me some pointers on where to start. The README.md currently says, that I should start from https://github.com/sbi-benchmark/sbibm/blob/main/sbibm/tasks/slcp/task.py but which functions are required?

Also (and likely later on) I'd love to know which environment on a cluster would be suitable to run the tests.

[jan-matthis]

Hi Peter, thanks so much for the kind words and for getting in touch!

I'd much welcome new tasks for the benchmark. The problem with tasks that do not have a reference posterior is that we will only be able to compute very few metrics. Could you describe your simulator in more detail, or perhaps even post some minimal code for it?

I am wondering whether we can figure out a tractable example with a similar structure -- in terms of benchmarking, we could get much more out of it if this was the case. If not, it can still be a useful exercise. However, we would be limited to diagnostics/calibration methods only, e.g. SBC in the case of NPE. With SNPE SBC can be expensive since the posterior is not amortized.

So far, I have only written little documentation about adding new tasks but would be very glad to help. The only methods we really need in task.py are get_prior, get_simulator, and _sample_reference_posterior. If we add tasks without tractable reference, we'd need to change the interface defined in sbibm/task.py such that _sample_reference_posterior is no longer required.

An environment adhering to the requirements defined in setup.py should suffice. If you install the package with pip install sbibm or pip install -e . during development, these should be resolved automatically. Ansible playbooks to recreate the exact setup I've used to run the benchmark at scale are also available -- but probably you will not need those.

[psteinb]

Hi, so for me I am mostly interested to reproduce your numbers with my task. :wink: At best, I'd love to compare SNPE, SNLE and SNRE approaches as my simulator provides me this freedom. c2st is totally enough. The reasoning behind this is, that I'd love exploit your somewhat tested setup of the methods before I dive in and implement this on my own for my simulator.

Moreover, I think this would be the hallmark of having an exampled forward-only task available. This way, people could just plug and play their simulator, submit jobs on their cluster and get a better bearing what to expect. So I am happy to help, but for my taste, a simple simulator like:

def simulator(parameter_set):
    return 1.0 + parameter_set + torch.randn(parameter_set.shape) * 0.1

works perfectly.

[psteinb]

Because you asked, my simulator at hand is a proxy for the real NPE I'd like to conduct. In the simplest case, it looks something like this in pseudo code:

def simulator(parameter_set):
    mu1, mu2 = parameter_set[0], parameter_set[1]
    cov = parameter_set[2:].reshape(2,2)

    # a multivariate gaussian discretized as a (100,100) pixel image
    mesh = torch.meshgrid(xx, yy)
    image = torch.exp(multi_variate_normal([mu1, mu2], cov).log_prob(mesh))

    # sum projections along each axis
    x_proj = image.sum(axis=0)
    y_proj = image.sum(axis=1)

    return cat([x_proj, y_proj], axis=-1) # one sample has shape (200,)

Reality is not that simple, but if the above synthetic simulator can be inverted - I am already half way there. ;-)

[psteinb]

And the task needs to be registered with tasks/__init__.py

[jan-matthis]

Hi Peter, thanks for the explanations!

If we want to be able to compute C2ST, we will need a simulator that allows computing a reference (so that we have to "bags" of samples that we can compare).

I looked at your simulator pseudo-code and the PR. Currently, the simulator is not stochastic. Most SBI algorithms are designed with stochastic simulators in mind, i.e. there is a probabilistic forward model p(x|theta) where theta are the parameters. How about using the probabilities (image) to draw from a Binomial distribution, e.g. Binomial(probs=image, total_count=1_000) or so? In this case, we would be able to calculate likelihoods and generate a reference posterior.

[psteinb]

I'll have a look at your proposal using the binomial.

In general, randomness is simple to add. I used the exp(log_prob) trick at this point to ease simulation. This can be replaced by a sample call without loss of generality.

Another inroads to stochastic effects is to add background noise, e.g. by adding Poisson noise to the 'image pixels'. I guess this is also what you had in mind with the Binomial.