search

sbi-benchmark / sbibm

Simulation-based inference benchmark
https://sbi-benchmark.github.io
MIT License
88 stars 34 forks source link

updating to sbi v0.18.0? #48

Closed psteinb closed 1 year ago

psteinb commented 2 years ago

sbi 0.18.0 brought in tons of changes. I was wondering if there are any plans to adopt those? If so, it might be useful to reflect performance changes in the rendered results.

For example, it might be worth considering to make the sbi version an additional field switch, e.g. like the Task currently.

jan-matthis commented 2 years ago

Currently, there are no plans to adopt sbibm to the new sbi API. A PR would definitely be welcome!

Maintaining backward compatibility across different versions of sbi and a field switch would be a neat feature. However, we are likely to end up with lots of duplicate code and in addition, we'd need to run tests against all different supported versions. To keep things light and more easily maintainable, my tendency would be to only support a single version of sbi. (So, if one wanted to use an older version of sbi, one would need to install an older version of sbibm as well.)

psteinb commented 2 years ago

I think there are 2 levels of abstracting the sbi version here:

I think fixing a singular sbi version to a singular sbibm is the way forward. But then, the benchmark data needs to honor this, I believe.

michaeldeistler commented 1 year ago

As a user, the pin to v0.17.2 is sometimes quite annoying since it always downgrades my sbi installation to v0.17.2 whenever I install the benchmark.

How about putting the entire algorithms folder into a separate repo? And then simply

from sbibm_algorithms import rej_abc  # instead of from sbibm.algorithms import rej_abc
michaeldeistler commented 1 year ago

Or one could simply dropping the pin and raise a warning in the run method if the detected sbi version is not 0.17.2

jan-matthis commented 1 year ago

How about installing with --no-deps as a workaround?

michaeldeistler commented 1 year ago

I did not know about this and it does the job, thanks!

It's still not ideal though because one has to install some deps manually (e.g. diffeqtorch). If anyone knows a way to ignore specific dependencies that would be great!