Open karimn opened 4 years ago
Latest run results:
64439045 NA 40 0 1.004662 1020.445 718.5901 100 100 100 100 DONE 2240.899 multi
64439047 NA 3 0 1.135677 19.6155 23.12698 100 100 100 100 DONE 5172.944 multi
64439048 NA 0 0 1.005245 1381.892 1215.766 100 100 100 100 DONE 1922.682 multi
64439049 NA 0 0 1.004802 1294.195 1157.78 100 100 100 100 DONE 3479.238 multi
64439051 NA 0 0 1.00486 919.9874 760.6585 100 100 100 100 DONE 4490.774 multi
64439052 NA 1 0 1.005137 1446.894 1582.326 100 100 100 100 DONE 2087.631 multi
64439053 NA 1128 0 1.745784 6.108404 4.384233 100 100 100 100 DONE 1801.592 multi
64439056 NA 0 0 1.004799 1264.387 1547.913 100 100 100 100 DONE 3323.802 multi
64439057 NA 0 0 1.004986 1522.469 1526.205 100 100 100 100 DONE 5116.031 multi
64439058 NA 3 0 1.005458 1226.369 1563.41 100 100 100 100 DONE 4774.41 multi
In particular, need to look into job 64439053 with countries
batch | country index | country code | country | job id |
---|---|---|---|---|
7 | 9 | BA | Bosnia and Herzegovina | 64439053 |
7 | 21 | BY | Belarus | 64439053 |
7 | 35 | EC | Ecuador | 64439053 |
7 | 48 | HN | Honduras | 64439053 |
7 | 61 | KE | Kenya | 64439053 |
7 | 74 | LY | Libya | 64439053 |
7 | 88 | NI | Nicaragua | 64439053 |
7 | 103 | QA | Qatar | 64439053 |
7 | 115 | TG | Togo | 64439053 |
7 | 128 | YE | Yemen | 64439053 |
@karimn not sure if there's an action here for me yet but let me know when there is one, I think this has pretty high ROI -- in the next 48 hrs I'm putting together a 1-page summary on heterogeneity in R0/Rt based on the results
No, I need to investigate this myself. I'm guessing it is no different than the problems we have with mult-region countries.
On Thu, Jul 16, 2020 at 10:25 AM Witold Wiecek notifications@github.com wrote:
@karimn https://github.com/karimn not sure if there's an action here for me yet but let me know when there is one, I think this has pretty high ROI -- in the next 48 hrs I'm putting together a 1-page summary on heterogeneity in R0/Rt based on the results
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/karimn/covid-19-transmission/issues/30#issuecomment-659447588, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABB553DSZPLNROFICB4POTR34EXFANCNFSM4O4H3Q2Q .
Looking like it might be another IFR problem. Also priors might be too tight for multi-country run. mobility_report.pdf
Rerun with fixed IFR and see what happens.
Runs well most of the time with batch sizes of 10 with fixed IFR and wider priors for national log R0 effects.
Runs well most of the time with batch sizes of 10 with fixed IFR and wider priors for national log R0 effects.
Great news. I think the national R0 hypersd prior should be very loose anyway, as we discussed.
Latest run:
64630455 NA 0 0 1.005905 435.8951 496.5576 100 100 100 100 DONE 7952.973 multi
64630456 NA 0 0 1.016424 388.8233 124.6698 100 100 100 100 DONE 12925.496 multi
64630457 NA 0 0 1.007425 1005.575 1708.988 100 100 100 100 DONE 5479.352 multi
64630458 NA 0 0 1.003787 1418.396 1698.136 100 100 100 100 DONE 7335.219 multi
64630459 NA 162 0 1.689602 6.279237 9.979283 100 100 100 100 DONE 15382.078 multi
64630460 NA 0 0 1.004618 877.9331 989.6823 100 100 100 100 DONE 6628.625 multi
64630461 NA 0 0 1.005569 792.8631 714.7175 100 100 100 100 DONE 6637.729 multi
64630462 NA 0 0 1.005679 1540.461 1655.015 100 100 100 100 DONE 7274.043 multi
64630463 NA 0 0 1.005335 1164.583 1032.519 100 100 100 100 DONE 11640.736 multi
64630464 NA 0 0 1.004944 1371.135 1484.963 100 100 100 100 DONE 19532.539 multi
One batch needs to be investigated. Report: mobility_report.pdf. Notice Austria, Greece, and Luxembourg.
The second to last column is seconds, I assume? I'm surprised by how variable it is.
Austria, Greece and Luxembourg seem to have the same problems as other fits before. Is it the problem where fixing IFR usually helps?
Basic learning here would be printing out hypermeans and hyper-SD's on hyperparameters. Do they change a lot across the batches? This could give us a bit of sense on how diffuse we should be making the corss-country hyperpriors.
Latest results. This is not consistent for the same batches. @wwiecek I'll get back to this on Monday and hopefully be able to answer some of your questions.
64658347 NA 0 0 1.009928 357.3179 414.2274 100 100 100 100 DONE 14601.696 multi
64658348 NA 0 0 1.01 1072.933 1299.68 100 100 100 100 DONE 11688.573 multi
64658349 NA 0 0 1.005477 778.6421 1081.133 100 100 100 100 DONE 6788.095 multi
64658350 NA 0 0 1.005348 1401.838 1747.692 100 100 100 100 DONE 6174.96 multi
64658351 NA 0 0 1.005727 1257.703 1177.077 100 100 100 100 DONE 9162.912 multi
64658352 NA 0 0 1.007109 705.2451 832.4779 100 100 100 100 DONE 5349.094 multi
64658353 NA 0 0 1.007354 782.4214 622.8227 100 100 100 100 DONE 5562.366 multi
64658354 NA 226 0 1.948024 5.537161 6.416773 100 100 100 100 DONE 20050.084 multi
64658355 NA 1000 0 1.664336 6.406624 13.25955 100 100 100 100 DONE 9565.342 multi
64658356 NA 0 0 1.005207 1218.213 1319.012 100 100 100 100 DONE 14780.155 multi
I'll do prior prediction to get a better sense of how it looks and to compare it to the posterior. Tomorrow; cluster server is down today.
Is it the same problem we've seen elsewhere? I think SK is a bit special because they had very early cases and very strong suppression (which was v successful), so there might be a different pattern there to other countries that didn't converge.
I'm seeing all kinds of problems on trying out different things like
It seems one of 10 batches is always having problems. I'm seeing funnels sometimes and bimodal splits sometimes.
I have not yet been able to figure out the cause of the problem and I'm starting to think there are different issues here. The model shouldn't be so sensitive to unexpected data that it trips up. I suspect the trend fixes things for some countries while breaking it for others. My approach in these situations is to start from the simplest possible model with no problems and iteratively add more complexity. This doesn't work well here since the model is very complex to start with and it is not clear how these parameters play off each other.
As I just commented elsewhere, the problem is that this is a bad model. Even if we give it a lot of flexibility, it's seemingly still a bad model. It will fit much better to "real" epidemics, which have a big peak. If there are multiple peaks or no definite peak, the model will have more trouble.
But I also wouldn't overplay the fact that one of the batches has problems. I think our priority is still to work toward the model that's closer to the real one. Let's discuss tomorrow.
It's not that one of the batches has problems, several of them fail in response to different changes. We're alway getting at least two batches failing. This makes it very difficult to run all 99 singletons countries together. I just ran that scenario and got 2000 divergent transitions (the good news is that it took just short of two days to run).
I agree that this is a bad model, but that doesn't mean it isn't useful. But instead of a post hoc approach of declaring countries incompatible with the model, can we figure out what characteristics of the data a priori make them unfit? This might be harder said than actually done because there are too many potential qualitative differences between the countries. I also wonder how useful a model is whenever you have to exclude some of the data to make it work. Are we biting the bullet and saying this a model that only says something useful about single peaked epidemics? Or is just missing further complexity/flexibility in order to nest weak or multi-peaked epidemics?
There are a lot of knobs that provide flexibility in adjusting the model, but it seems like every time we adjust one we get another problem somewhere else and the more complicated we make the model, the harder it would be to figure where things went wrong. This is why I want to start with the simplest (definitely underfit model) to generate synthetic data to see if the model is even able to fit its own prior generated data, and then iteratively increase the complexity of the model guided by what we think the model is missing (hierarchical structure, contact rate non-linear trend, adding new new time-varying and time-invariant covariates).
On Thu, Jul 23, 2020 at 5:28 PM Witold Wiecek notifications@github.com wrote:
As I just commented elsewhere, the problem is that this is a bad model. Even if we give it a lot of flexibility, it's seemingly still a bad model. It will fit much better to "real" epidemics, which have a big peak. If there are multiple peaks or no definite peak, the model will have more trouble.
But I also wouldn't overplay the fact that one of the batches has problems. I think our priority is still to work toward the model that's closer to the real one. Let's discuss tomorrow.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/karimn/covid-19-transmission/issues/30#issuecomment-663241296, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABB552HQPBNG5THWTDWLETR5CTPHANCNFSM4O4H3Q2Q .
On the side, I ran with only two countries (with "--no-pooling"), PY and PT, and the model has some problems. Same run without "--no-pooling" -- so it's hierarchical over countries ---- worked well. I wonder if with no pooling we end up with the same problem we see when we have a completely uninformative prior on the SD param that was causing divergences before.