search

CDCgov / cfa-viral-lineage-model

Apache License 2.0
9 stars 0 forks source link

Track MCMC convergence #30

Closed afmagee42 closed 3 weeks ago

afmagee42 commented 4 weeks ago

This PR (built off of #25) adds the capacity to store, per-model, some MCMC convergence diagnostics. Primarily ESS and PSRF, and by default I set it up to just log the worst of each (separately, not the joint worst parameter).

This reveals that we may need to experiment with the hierarchical model a bit, as I'm seeing a minimum ESS of ~10 and a maximum PSRF of ~1.8.

afmagee42 commented 3 weeks ago

I have expanded the functionality and amended behavior such that we now get more useful output.

In the config, we specify an ESS threshold, below which ESS is "bad" and a PSRF threshold, above which PSRF is "bad." (I instituted defaults of 625 for ESS and 1.005 for PSRF.) Any parameter which is bad in either sense now ends up in the report table.

Further, if the plot option is set to True, trace plots are produced for all failing parameters. They look like this: z_0 17,0

I propose that we treat the purpose of this PR as getting a diagnostic toolkit we're happy with, and relegate MCMC modifications to another.

thanasibakis commented 3 weeks ago

Woohoo!