search

sbi-dev / sbi

Simulation-based inference toolkit
https://sbi-dev.github.io/sbi/
Apache License 2.0
599 stars 152 forks source link

SNLE not working, always estimate gaussian on even on toy examples like Beta, Gamma distributions #1315

Closed rakaar closed 5 hours ago

rakaar commented 1 day ago

Describe the bug SNLE not working, always estimate gaussian on even on toy examples like Beta, Gamma distributions

To Reproduce Please add a minimal code example that reproduces the problem:

  1. Python version - 3.10.12 and SBI version: 0.23.2

  2. Minimal code example: Python Notebooks with SNLE tried on Gamma and Beta Distribution

  3. [Optional]: error message

    • Likelihood estimated is always gaussian like no matter what example image image

Expected behavior SNLE should be able to estimate the likelihood correctly after Neural network training has converged

Additional context Add any other context about the problem here.

janfb commented 1 day ago

Hi @rakaar thanks for reporting this. Are you using MAF as density estimator? When applied to one-dimensional parameter (data) spaces, MAF amounts to fitting a Gaussian.

If you are using MAF, can you please run it again with density_estimator="nsf" and show the plots?

Cheers, Jan

rakaar commented 1 day ago

@janfb

Thank you so much for the quick reply! image image

The likelihood is now much better with NSF Density Estimator!

Also, wouldn't it be nice to also "NSF" as default density estimator for SNLE?, because people come to SBI for complicated distributions, and a simple gaussian fit would most likely fail.

janfb commented 1 day ago

Glad that this worked!

The exact choice of density estimator really depends on the problem at hand. For example, for a 2-D example, MAF probably works similarly well.

In fact, you should have received a warning when using MAF with 1-D distributions. Did you receive this warning?

rakaar commented 1 day ago

Yes. I saw the warning now. Sorry, didn't notice it.

But wouldn't it be better to change density_estimator to NSF, if using SNLE with 1D automatically in the library and log(print) it? I can submit a Pull Request with including this in documentation if you are ok with the idea .

janfb commented 6 hours ago

As NSF also has its downside, e.g., it tends to be slower to train, I believe that keeping MAF as the default and raising a warning is a good solution here.

@michaeldeistler what do you think?

michaeldeistler commented 5 hours ago

yes I am also leaning towards sticking with MAF.

janfb commented 5 hours ago

OK, I will close to issue then. Thanks for offering help with implementing this @rakaar! We would be happy to get your help with other features in the future 🙏