search

kinto-b / elections

Predicting the Australian federal election by combining a bayesian poll-of-polls with multi-level regression and poststratification
1 stars 1 forks source link

Diagnostics #1

Open kinto-b opened 1 year ago

kinto-b commented 1 year ago

In c618bbe I increased the variance on the priors for what is effectively the division level swing and on the coefficients of the demographic covariates. This is good insofar as it lets the record-level data have more influence on the estimates. Prior to the change the estimates were anchored really strongly to the previous election result, which wasn't ideal.

However, this introduced pretty serious computational issues. The (cleaned up) stan output reads:

1: Divergent Transitions Warning: There were 1788 divergent transitions after warmup. To understand why this is an issue and how to address it, visit: https://mc-stan.org/misc/warnings.html#divergent-transitions-after-warmup (Package 'StanHeaders' was built under R version 4.2.3)

2: Sampling Problem Diagnosis: Examine the pairs() plot to identify potential sampling issues. (Package 'StanHeaders' was built under R version 4.2.3)

3: Low Bulk Effective Samples Size (ESS) Warning: Bulk Effective Samples Size (ESS) is too low, indicating potential unreliability of posterior means and medians. Running the chains for more iterations might help. (Package 'StanHeaders' was built under R version 4.2.3)

4: Low Tail Effective Samples Size (ESS) Warning: Tail Effective Samples Size (ESS) is too low, suggesting potential unreliability of posterior variances and tail quantiles. Running the chains for more iterations might help. (Package 'StanHeaders' was built under R version 4.2.3)

Divergences: 1788 out of 10000 iterations encountered divergences (17.88%). Consider increasing the 'adapt_delta' parameter to address the divergences.

Tree Depth: None of the 10000 iterations reached the maximum tree depth of 10.

Energy: E-BFMI indicates no pathological behavior.

The large number of divergent transitions is particularly concerning.

kinto-b commented 1 year ago

Have resolved the divergences, which were being caused by the regression component. Only 29/10000 now diverge and judging by the paris plots and divergent trajectories they are ignorable.

The low ESS relates to the poll aggregation component. Could try reducing the resolution on the random walk from weekly to monthly to see if that improves things