christophsax
commented
5 years ago move discussion from code
CHS
I would really like to have a formal test to tell me if convergence has reached or not!
If it is really necessary to look at this plots, we should have a function that makes all of the following functions
SETH
A few notes on this:
(1) Assessing convergence via a formal test isn't trivial: there is a nice overview here: http://www.mcmchandbook.net/HandbookChapter6.pdf. There is already an R package to do this --- the coda pacakge. We could introduce this as a dependency. However, I'm not inclined to make bdfm yet more complicated and draw out the project. Interested users should be albe to plug stored simulation values into coda functions to test convergence. However, looking at the trace plot works well, is simple, and obvious.
(2) There are a LOT of parameters that would need checking --- generating plots for all of them may not be very practical. For example, in a model with 20 observables, 2 factors and 3 lags we would have 40 loadings, 20 shocks to the observation equation (this matrix is diagonal), 12 transition equation parameters, and 4 shocks to the transition equation (this matrix is square) for a total of 76 parameters to check, and that's a pretty small model.
(3) A lot of convergence testing is used to update the mixing distribution in the Metropolis Hastings algorithm used to generate MCMC simulations. We just use Gibbs sampling, meaning the probability of accepting a draw is 1 and there is nothing to tweak. This is not necessarily the most efficient approach, but is simple, doesn’t require user input, and works well most of the time.
CHS
A lot of questions here: Why do things look good, how could I tell if they weren't, and mostly: how could I change the defaults to make them better
SETH
"Looking good" means sampling from a stationary distribution --- i.e. we're drawing values around a posterior mean. Not good would be something obviously non stationary, or with obvious breaks in the model jumping between 2 or more different mean parameter values. It's not always obvious to me how one might change defaults to make things look better --- good data is probably the biggest issue. Another time issues come up is when we use priors to force the model towards somewhere it doesn't want to be. Instead of gradually shifting towards the prior distribution, it is often the case that at a certain point the model will "jump" to a new distribution that satisfies the priors. If our priors put it right on edge, it might jump between models making our posterior estimates invalid. There's no obvious way to control for this that I'm aware of.. it's just guess and check with priors until you get somewhere where the model is "happy".
Initial discussion on how to check convergence is here: https://github.com/srlanalytics/bdfm/commit/9d9d65c3604cf625bb593af7f8855b3bf0c44eb1
A formal and automated convergence would hugely beneficial, perhaps with the coda package, or with a simpler ad test? To be investigated.