Closed seananderson closed 1 year ago
I'm glad you caught this - I was having some issues recently trying to interpret the difference in ELPD values between models w/simulated data and it didn't make sense.
Is it worth just removing the ELPD calculation entirely? Currently the sum of log likelihoods is being returned, so the ELPD calculation is redundant
Yes, it might break some people's code, but perhaps that's a good thing because it will make them stop, read the help file, and understand what happened rather than silently having the results for the elpd
element change.
I think ELPD is currently incorrect in sdmTMB. The averaging is happening across data points but the averaging is actually supposed to happen across MCMC samples, of which we have the equivalent of 1. So ELPD reduces to the sum of the log likelihoods, which is
fold_elpd
andsum_loglik
in thesdmTMB_cv()
output.Created on 2023-06-23 with reprex v2.0.2
That's not to say that what we're doing is completely wrong. I believe it should scale the same as the real ELPD and should render higher values for the 'better' model, but the scale is off. It's currently a log average likelihood. It's supposed to be a sum of log likelihoods.