gowerc
commented
1 month ago I think this can be closed now. There were a couple of findings though:
1) A major source of difference is that brms does loo at the observation level whereas we were doing it at the subject level. Having said that I'm not even sure loo is valid when done for multiple observations at a time 2) There appeared to be an issue caused by the fact that Stans internal functions drop the normalising constants for their distribution functions e.g. lpdf_normal(); though this was only an issue when doing loo on the subject level (multiple observations) and vanished when doing it on the observation level. 3) After a discussion with @mercifr1 we agreed that for the moment there isn't an obvious way to implement loo on the full joint model due to the conflicting levels they operate on and how they affect each other via the linking at an observation level.
A spin off issue from this is #402
Currently our loo estimates are showing pathologic issues which do not appear in equivalent brms implementations. Need to compare the loglik values to see if we can work out where these issues are coming from.