sbi-benchmark / sbibm

Simulation-based inference benchmark
https://sbi-benchmark.github.io
MIT License
88 stars 34 forks source link

gaussian_mixture true_theta / observation have shifted with version #45

Closed bkmi closed 2 years ago

bkmi commented 2 years ago

I was running the benchmark and found that no method was producing accurate posteriors (according to C2ST) for the gaussian_mixture task. I wondered if the simulator has somehow changed, thereby introducing a different ground truth posterior for each saved observation.

Indeed, this simple check shows that there has been some drift in the simulator

task = sbibm.get_task("gaussian_mixture")
num_observation = 5
true_theta = task.get_true_parameters(num_observation)
sbibm_obs = task.get_observation(num_observation)
new_obs = task.get_simulator()(true_theta)
obss = torch.concat([task.get_simulator()(true_theta) for _ in range(100)])
print(
    (torch.linalg.norm(sbibm_obs - obss)).mean(),
    (torch.linalg.norm(new_obs - obss)).mean(),
)

This typically returns tensor(115.6793) tensor(16.9946).


To fix the issue, either the simulator can be returned to its previous state or we could generate new ground truth parameters and observations; however, this runs the issue of not being backwards compatible with previous versions of sbibm.

bkmi commented 2 years ago

Hmm, now I've rerun things and I'm no longer sure that what I'm saying is correct.