Default priors

[venpopov]

After the workshop discussion on "comparable priors", I thought more about the way we define priors for the models we have. For the SDM we have a fairly informative prior based on the recommendations of Oberauer (2023). For the mixture models and imm we have more informative priors than brms does for it's families, but they are still somewhat arbitrary. I had a chat with Klaus, and we both had the same idea.

Empirically-informed default priors

Since we have a lot of rich data for these tasks, we can actually fit the models in bmm to a large set of existing datasets, and then use the posterior estimates of those to define the default priors in a principled way. The results of this can also be described in an article on the website. The end resuslt would be not only a well motivated choice for priors, but a good resource and description of how these were established.

Implementation

I suggest we make a separate dataset R package repository, in which we gather and upload documented datasets. I will write a few functions to automated the process, and fit all models we have to all datasets on the cluster. Then analyze the results, write them up and suggest default priors based on them.

Since we get a posterior distribution from the fits, we'll have to figure out what's the closest appropriate analytic distribution that matches. Or maybe there is way to use posterior samples for priors? I can look into that, but the first approach would definitely be cleaner.

what do we need priors for?

• main levels of parameter values
• effects of continuous variables
• effects of categorical variables
• random intercepts and slopes sd

[venpopov]

What I am a bit unsure of, is how to determine the priors for the main parameters. Let's say kappa in the mixture models. Obviously, if we just fit a single model to all datasets with a single kappa level, well get a very narrow posterior because of the ton of data. Ok, so we estimate kappa separate for each dataset and condition. Then we would have a posterior distribution for the kappa for each dataset x condition combination. Do we just combine all the posterior samples into a single distribution? This could problematic because some datasets would be more informative than others, but we treat them equally. Or maybe that's not a problem, because the uncertainty will be reflected in the posterior - rich informative datasets would produce a narrow peaked posterior, less informative datasets would produce a wider, flatter posterior. The combination of both into a single posterior already takes into account that uncertainty. But let's say we have a dataset that has an outlier kappa, with a very precise narrow posterior (i.e. a huge prerception dataset with high kappa). Then we would end-up with a bimodal prior?

I will try first with a few datasets and one model before commiting to the whole approach.

[GidonFrischkorn]

I was thinking, if we should fit models only including an intercept with variations across subjects for all model parameters, and not include any of the conditions of the datasets. My reasoning was, that the priors should generalize across experimental manipulations and rather capture the parameter space for reasonable parameter values. If we include experimental effects, this would indeed likely result in very narrow priors and this would not be ideal.

Even if we do not include the condition variables, it might still result in narrow posteriors that are potentially not ideally suited as priors. Maybe it would be sufficient to determine the location of the prior by fitting previous data sets, and then look at the overall spread for each subject to determine the required width for the dispersion parameter.

Still, I generally like the idea of using existing data to specify informative priors. My tendency would still be to balance the information we get from previous data with a reasonable reflection of uncertainty. so that the models translate to new experimental design with new materials and manipulations.