gbradburd / conStruct

method for modeling continuous and discrete population genetic structure
35 stars 14 forks source link

large difference across chains #65

Closed renhamm closed 11 months ago

renhamm commented 11 months ago

I am seeing major discrepancies across chains within a run, and I'm not sure what the cause is. I've attached the results from my K=3 spatial run (4 chains, 1000 iterations). The results from chain one show no discrete groups which is directly contrasted by the later 3 chains. How do I know which chain is correct?

I am also repeatedly getting Rhat and ESS errors (see below). I thought these fit the parameters for being accceptably ignored, but perhaps not with such wild divergence of ancestry estimation. " 1: There were 44 divergent transitions after warmup. See https://mc-stan.org/misc/warnings.html#divergent-transitions-after-warmup to find out why this is a problem and how to eliminate them. 2: There were 22 transitions after warmup that exceeded the maximum treedepth. Increase max_treedepth above 10. See https://mc-stan.org/misc/warnings.html#maximum-treedepth-exceeded 3: Examine the pairs() plot to diagnose sampling problems

4: The largest R-hat is 1.89, indicating chains have not mixed. Running the chains for more iterations may help. See https://mc-stan.org/misc/warnings.html#r-hat 5: Bulk Effective Samples Size (ESS) is too low, indicating posterior means and medians may be unreliable. Running the chains for more iterations may help. See https://mc-stan.org/misc/warnings.html#bulk-ess 6: Tail Effective Samples Size (ESS) is too low, indicating posterior variances and tail quantiles may be unreliable. Running the chains for more iterations may help. See https://mc-stan.org/misc/warnings.html#tail-ess

K3_layer.cov.curves.pdf K3_model.fit.CIs.pdf K3_structurePlots.pdf K3_trace.plots.pdf

gbradburd commented 11 months ago

The first run definitely seems different from the other 3, but it's difficult to visually compare the rest because it doesn't seem like you've dealt with label-switching between independent runs. Check out the visualize-results vignette for more information there.

Re: dealing with warnings - you can pass different stan control parameters (like max_treedepth or adapt_delta) to your conStruct run via the control argument (see here).

Re: choosing which run - looking at the trace plots of the posterior probability and the different parameter estimates for the different runs, it seems like runs 1 and 2 did not have good mixing (note how the posterior probability is still climbing for at least the first 20-40% of the post-burnin iterations). Runs 3&4 look much better (traceplots look like fuzzy caterpillars, etc.). If you want to pick one chain to report, you could pick the one with the highest mean posterior probability (which looks like run 3 in this case), but ideally you'd see that multiple independent runs were converging on the same stationary distribution.

renhamm commented 11 months ago

This was super helpful. Running the model longer made no difference in minimizing the discrepancy (actually the opposite), so maybe altering the stan control parameters will help!