Problems applying `sbi`: posteriors too wide.

[vivienr]

Short summary:

Problems applying sbi to my case:

• tired with posterior dimensionality of 1 and 2 parameters.
• input time-series (signal+noise) is 4096 points, which I compress using SVD of the signal-space to 200 or 100 elements.

The posteriors recovered are systematically too broad (as checked by an analytical likelihood function).

• variable number of simulations, from 100k to 10M. (convergence of the result seems to happen around 1M)
• used SNPE with all other setting defaults, and tested with increases by x2 and x4

Trying to increase settings further would benefit from GPU, but it seems to fail with TypeError at user_input_checks. IS there a working example available? Happy to file the GPU issue as a separate issues if helpful.

Longer version:

Thank you very much for making sbi available, and for helping others making use of it ! It was very easy to set-up on my problem (gravitational-wave astrophysics, I'm a lecturer at Cardiff University). My field is very well posed for simulation based inference: it is relatively easy to make simulations, but inverting can be challenging, and doing it at speed would present many advantages.

In general, the posterior dimension is variable: typically 15 but applications are possible from 1 to many hundreds. The input is a time-series or frequency-series (signal+noise), from ~1000 points to >2M points. It is usually compressed (SVD or others of the signal space) to hundreds elements. I'm trying first a simpler case limited to 1 or 2 parameters, on a narrow range of the parameter space. My issue is that the posterior recovered are too broad, as checked by independent traditional Bayesian sampling. It improves with more data somewhat, but the improvement seems to taper of at ~1M simulations. To increase that number further (if the issue is for the network to learn the likelihood better), I suspect running on GPU will be needed. However my attempts at doing so failed, TypeErrors, can't convert tensor. Do you have by any chance a working example on GPU I could start from?

Best regards, Vivien.

[janfb]

Hi Vivien,

and thanks for detailed issue.

It seems you do have access to the likelihood, right? And you obtain a ground truth posterior using MCMC to check the 2 dimensional posterior obtained by SNPE?

Is the MCMC method running on the SVD compressed time series as well? If not, it could be that the loss of information due to the compression results in the difference between the MCMC and the SNPE posterior.

If your simulations are cheap you could also quickly double-check the SNPE result in 1D using rejection ABC. If SNPE converges at 1M, then a 10M rejection ABC in 1D should be quite good as well. Just take the top 1k parameters according to some distance between simulated and actually observed data and compare it to 1k samples from the SNPE posterior.

Regarding the GPU problem, feel free to create a separate issue. However, we typically do not expect a speed up in NN training with the SNPE default settings, because the NN are relatively small and don't involve convolutions.

Best, Jan