Poor mixing and low n_eff for intercept only models

@lizzieinvancouver, @legault, @FaithJones As we learned at our last mini-retreat, our intercept only models with two intercepts are producing low n_eff and poorly mixed chains. The post @lizzieinvancouver made on this issue is here.

We also tried to approach this problem with a simple dataset of scores of figure skating routines from the 1932 Olympics. The code can be found here and Stan code here. But the chains looked poor to us.

I also made a discourse post for this example. Interestingly both Lizzie and I were told that we should try ncp for the models. But we ran the models in rstanarm, which applies ncp when necessary and the chains still mixed poorly.

@lizzieinvancouver suggested emailing Andrew Gelman about it. He forwarded our problem to Jonah Sol Gabry, Ben Goodrich, and Lu Zhang. Andrew's first thought was that the issue with the example is that the sample size is small and the model might just be hard to fit with such weak priors. Andrew included Lu since he thought it might be an interesting model to run through Pathfinder.

Jonah replied and very generously wrote the Stan code for the Olympics example with ncp for us. This does improve the n_eff, producing values closer to 775, and good Rhat values very close to 1. But I was surprised since the chains still looked bad to me.

Both Andrew and Jonah did not seem to think this was concerning. Andrew's response was to look at the Rhat's and Jonah replied that he thought they looked decent enough given that they do not get "stuck" anywhere with no areas where chains flatline. But he also thinks the model would be improved by by using stronger priors on the sigmas.

I saw this recent post on Andrew's blog about using tighter priors as well!

lizzieinvancouver / ospree

Poor mixing and low n_eff for intercept only models #424