Likelihood / posterior distributions from a single trajectory vs from the ensemble

[henryrscharf]

@henryrscharf @cboettig Likelihood / posterior distributions from a single trajectory vs from the ensemble

[cboettig]

Just a quick note that Henry & I got the matrix versions working for this in both nimble and greta.

In greta this doesn't actually change the model definition at all, x_t and x_t1 just become matrices with a column for each replicate instead of being a vector):

https://github.com/cboettig/nimbios-transients/blob/99bf6bd0eda98e8caac9234e2d7c77127643f4ff/code/greta.Rmd#L47-L49

In nimble this is basically the same, just with an explicit outer loop:

https://github.com/cboettig/nimbios-transients/blob/99bf6bd0eda98e8caac9234e2d7c77127643f4ff/code/nimble-ensemble.Rmd#L19-L37

As expected, this does improve the estimates, though it looks like we can in fact get reasonable estimates even on a single replicate.

[cboettig]

@henryrscharf have you tried running nimble_ghost.Rmd with different seeds? From what I'm seeing on greta my posteriors can look pretty different when running the same mcmc on different replicates (or ensembles of replicates). Curiously, I don't get the best estimates from including all 100 replicate timeseries from the ensemble...

[henryrscharf]

@cboettig I haven't tried that. I'm away from my office visiting family right now, so I may not be able to do this right away, but I'll follow up later.

[henryrscharf]

@cboettig Finally had a chance to take a look at this today. Even though the marginal likelihood profiles (p. 6) look similar for different seeds, I do see some different behavior in the posterior distributions. I suppose it might be that different realizations contain different amounts of information about particular parameter combinations, but that would be pretty surprising for simulated data of this length.

Your finding that including multiple replicates does not improve inference is really surprising. I can't think of a good explanation for that. Are you confident the model is specified correctly?

[cboettig]

@henryrscharf I needed to poke around a bit more on this, I think I was changing the prior at the same time. In general, looks like more replicates help. I'm keeping the seed now, and I get mostly reasonable results on the full data (I took a bit of a detour to get it running on the GPU, where I can now run 100 reps of 1000 time points each in ~ just over 2 min for 5 parameters).

Here's a summary of what I'm seeing. Links below should take you to the notebook with plots of chains and posteriors vs the true values:

• If I widen the prior on r, I find things don't converge well, even for 100 reps

• With a narrower prior, things look pretty good, though the true values for a and r are noticeably far from the (narrow) peaks of the posteriors (in this seed).

• with only 5 replicates & narrow prior, all posteriors look very good

• with only the first replicate, posteriors do not look so good. Most are just broad, but notably the r estimate does not seem to cover the true value, though admittedly the chains look like things aren't converging.

• that really depends on which replicate is used, heres the same setup but using a different single replicate where convergence looks better and estimates are better, though not as good as using the first 5 replicates.