Batched singletons have a few problems.

[karimn]

On the side, I ran with only two countries (with "--no-pooling"), PY and PT, and the model has some problems. Same run without "--no-pooling" -- so it's hierarchical over countries ---- worked well. I wonder if with no pooling we end up with the same problem we see when we have a completely uninformative prior on the SD param that was causing divergences before.

[karimn]

Latest run results:

64439045                 NA      40      0       1.004662        1020.445        718.5901        100     100     100     100     DONE    2240.899        multi
64439047                 NA      3       0       1.135677        19.6155         23.12698        100     100     100     100     DONE    5172.944        multi
64439048                 NA      0       0       1.005245        1381.892        1215.766        100     100     100     100     DONE    1922.682        multi
64439049                 NA      0       0       1.004802        1294.195        1157.78         100     100     100     100     DONE    3479.238        multi
64439051                 NA      0       0       1.00486         919.9874        760.6585        100     100     100     100     DONE    4490.774        multi
64439052                 NA      1       0       1.005137        1446.894        1582.326        100     100     100     100     DONE    2087.631        multi
64439053                 NA      1128    0       1.745784        6.108404        4.384233        100     100     100     100     DONE    1801.592        multi
64439056                 NA      0       0       1.004799        1264.387        1547.913        100     100     100     100     DONE    3323.802        multi
64439057                 NA      0       0       1.004986        1522.469        1526.205        100     100     100     100     DONE    5116.031        multi
64439058                 NA      3       0       1.005458        1226.369        1563.41         100     100     100     100     DONE    4774.41         multi

In particular, need to look into job 64439053 with countries

batch	country index	country code	country	job id
7	9	BA	Bosnia and Herzegovina	64439053
7	21	BY	Belarus	64439053
7	35	EC	Ecuador	64439053
7	48	HN	Honduras	64439053
7	61	KE	Kenya	64439053
7	74	LY	Libya	64439053
7	88	NI	Nicaragua	64439053
7	103	QA	Qatar	64439053
7	115	TG	Togo	64439053
7	128	YE	Yemen	64439053

[wwiecek]

@karimn not sure if there's an action here for me yet but let me know when there is one, I think this has pretty high ROI -- in the next 48 hrs I'm putting together a 1-page summary on heterogeneity in R0/Rt based on the results

[karimn]

No, I need to investigate this myself. I'm guessing it is no different than the problems we have with mult-region countries.

On Thu, Jul 16, 2020 at 10:25 AM Witold Wiecek notifications@github.com wrote:

@karimn https://github.com/karimn not sure if there's an action here for me yet but let me know when there is one, I think this has pretty high ROI -- in the next 48 hrs I'm putting together a 1-page summary on heterogeneity in R0/Rt based on the results

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/karimn/covid-19-transmission/issues/30#issuecomment-659447588, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABB553DSZPLNROFICB4POTR34EXFANCNFSM4O4H3Q2Q .

[karimn]

Looking like it might be another IFR problem. Also priors might be too tight for multi-country run. mobility_report.pdf

Rerun with fixed IFR and see what happens.

[karimn]

Runs well most of the time with batch sizes of 10 with fixed IFR and wider priors for national log R0 effects.

[wwiecek]

Runs well most of the time with batch sizes of 10 with fixed IFR and wider priors for national log R0 effects.

Great news. I think the national R0 hypersd prior should be very loose anyway, as we discussed.

[karimn]

Latest run:

64630455                 NA      0       0       1.005905        435.8951        496.5576        100     100     100     100     DONE    7952.973        multi
64630456                 NA      0       0       1.016424        388.8233        124.6698        100     100     100     100     DONE    12925.496       multi
64630457                 NA      0       0       1.007425        1005.575        1708.988        100     100     100     100     DONE    5479.352        multi
64630458                 NA      0       0       1.003787        1418.396        1698.136        100     100     100     100     DONE    7335.219        multi
64630459                 NA      162     0       1.689602        6.279237        9.979283        100     100     100     100     DONE    15382.078       multi
64630460                 NA      0       0       1.004618        877.9331        989.6823        100     100     100     100     DONE    6628.625        multi
64630461                 NA      0       0       1.005569        792.8631        714.7175        100     100     100     100     DONE    6637.729        multi
64630462                 NA      0       0       1.005679        1540.461        1655.015        100     100     100     100     DONE    7274.043        multi
64630463                 NA      0       0       1.005335        1164.583        1032.519        100     100     100     100     DONE    11640.736       multi
64630464                 NA      0       0       1.004944        1371.135        1484.963        100     100     100     100     DONE    19532.539       multi

One batch needs to be investigated. Report: mobility_report.pdf. Notice Austria, Greece, and Luxembourg.

[wwiecek]

The second to last column is seconds, I assume? I'm surprised by how variable it is.

Austria, Greece and Luxembourg seem to have the same problems as other fits before. Is it the problem where fixing IFR usually helps?

[wwiecek]

Basic learning here would be printing out hypermeans and hyper-SD's on hyperparameters. Do they change a lot across the batches? This could give us a bit of sense on how diffuse we should be making the corss-country hyperpriors.

[karimn]

Latest results. This is not consistent for the same batches. @wwiecek I'll get back to this on Monday and hopefully be able to answer some of your questions.

64658347                 NA      0       0       1.009928        357.3179        414.2274        100     100     100     100     DONE    14601.696       multi
64658348                 NA      0       0       1.01    1072.933        1299.68         100     100     100     100     DONE    11688.573       multi
64658349                 NA      0       0       1.005477        778.6421        1081.133        100     100     100     100     DONE    6788.095        multi
64658350                 NA      0       0       1.005348        1401.838        1747.692        100     100     100     100     DONE    6174.96         multi
64658351                 NA      0       0       1.005727        1257.703        1177.077        100     100     100     100     DONE    9162.912        multi
64658352                 NA      0       0       1.007109        705.2451        832.4779        100     100     100     100     DONE    5349.094        multi
64658353                 NA      0       0       1.007354        782.4214        622.8227        100     100     100     100     DONE    5562.366        multi
64658354                 NA      226     0       1.948024        5.537161        6.416773        100     100     100     100     DONE    20050.084       multi
64658355                 NA      1000    0       1.664336        6.406624        13.25955        100     100     100     100     DONE    9565.342        multi
64658356                 NA      0       0       1.005207        1218.213        1319.012        100     100     100     100     DONE    14780.155       multi

[karimn]

I'll do prior prediction to get a better sense of how it looks and to compare it to the posterior. Tomorrow; cluster server is down today.

[karimn]

mobility_report.pdf

[wwiecek]

mobility_report.pdf

Is it the same problem we've seen elsewhere? I think SK is a bit special because they had very early cases and very strong suppression (which was v successful), so there might be a different pattern there to other countries that didn't converge.

[karimn]

I'm seeing all kinds of problems on trying out different things like

• widening the priors for R0 and betas
• Turning on/off the trend
• Turning off IFR noise
• Introducing separate overdispersion parameters for each country

It seems one of 10 batches is always having problems. I'm seeing funnels sometimes and bimodal splits sometimes.

I have not yet been able to figure out the cause of the problem and I'm starting to think there are different issues here. The model shouldn't be so sensitive to unexpected data that it trips up. I suspect the trend fixes things for some countries while breaking it for others. My approach in these situations is to start from the simplest possible model with no problems and iteratively add more complexity. This doesn't work well here since the model is very complex to start with and it is not clear how these parameters play off each other.

[wwiecek]

As I just commented elsewhere, the problem is that this is a bad model. Even if we give it a lot of flexibility, it's seemingly still a bad model. It will fit much better to "real" epidemics, which have a big peak. If there are multiple peaks or no definite peak, the model will have more trouble.

But I also wouldn't overplay the fact that one of the batches has problems. I think our priority is still to work toward the model that's closer to the real one. Let's discuss tomorrow.

[karimn]

It's not that one of the batches has problems, several of them fail in response to different changes. We're alway getting at least two batches failing. This makes it very difficult to run all 99 singletons countries together. I just ran that scenario and got 2000 divergent transitions (the good news is that it took just short of two days to run).

I agree that this is a bad model, but that doesn't mean it isn't useful. But instead of a post hoc approach of declaring countries incompatible with the model, can we figure out what characteristics of the data a priori make them unfit? This might be harder said than actually done because there are too many potential qualitative differences between the countries. I also wonder how useful a model is whenever you have to exclude some of the data to make it work. Are we biting the bullet and saying this a model that only says something useful about single peaked epidemics? Or is just missing further complexity/flexibility in order to nest weak or multi-peaked epidemics?

There are a lot of knobs that provide flexibility in adjusting the model, but it seems like every time we adjust one we get another problem somewhere else and the more complicated we make the model, the harder it would be to figure where things went wrong. This is why I want to start with the simplest (definitely underfit model) to generate synthetic data to see if the model is even able to fit its own prior generated data, and then iteratively increase the complexity of the model guided by what we think the model is missing (hierarchical structure, contact rate non-linear trend, adding new new time-varying and time-invariant covariates).

On Thu, Jul 23, 2020 at 5:28 PM Witold Wiecek notifications@github.com wrote:

As I just commented elsewhere, the problem is that this is a bad model. Even if we give it a lot of flexibility, it's seemingly still a bad model. It will fit much better to "real" epidemics, which have a big peak. If there are multiple peaks or no definite peak, the model will have more trouble.

But I also wouldn't overplay the fact that one of the batches has problems. I think our priority is still to work toward the model that's closer to the real one. Let's discuss tomorrow.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/karimn/covid-19-transmission/issues/30#issuecomment-663241296, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABB552HQPBNG5THWTDWLETR5CTPHANCNFSM4O4H3Q2Q .