tvwenger
commented
2 weeks ago Thanks for the feedback @bcappa ! It looks like this is a case of randomness-induced variation. As you noted, VI (approx=False) can give some poor results, especially if it is not tuned for a given model and data. For some models, VI just won't work.
When I run the optimization tutorial with VI, I get n_clouds=3 as the best model, but that must just be because VI was able to find that third component when I ran it. I suspect that we just need to tune the VI hyper-parameters a bit to get more consistent results. Or, we just need to stick to MCMC (slower but more reliable).
I've added a guide to the bayes_spec documentation to offer some advice about fitting models with bayes_spec. We'll need to spend a little time tuning the VI and MCMC hyper-parameters for this model and these data. Hopefully the advice in this guide will help with this effort!
https://bayes-spec.readthedocs.io/en/latest/tips.html
I will add the other examples/tutorials (SMC, approx=False) when I have some spare CPU time. :)
Hello,
I tried the 'Optimization Tutorial' (https://amoeba2.readthedocs.io/en/latest/notebooks/optimization.html) and when I copy and paste the code exactly to optimise the fit of the simulated data, I get "Best model has n_clouds = 2" instead of n_clouds = 3, like is stated in the tutorial.
And note that the 2 cloud model is not a good fit of the simulated data from the tutorial, one of the velocity components is missed:
In the tutorial, it uses the variational inference method to approximate the fit for the different models (null, 1 cloud, 2 clouds... up to 5 clouds).
When I looked at the bayes_spec optimization code in the github I found the option to make the optimization object sample all of the models with MCMC by putting in the keyword "approx=False" into opt.optimize(). This resulted in the best model being 3 clouds, which matches the plots in the tutorial:
So my suggestion/comment is to update the tutorial to reflect that you will get a best model of 2 clouds when following the exact code in the tutorial.
And, I would suggest adding in a section that demonstrates using full MCMC sampling in the optimisation and using the SMC option. That would help the user know about the different options with optimisation and help them decide the best method for their use case.