Completing Scenario 3

[ChrisRackauckas]

• [ ] do independent parts https://github.com/ChrisRackauckas/ASKEM_Evaluation_Staging/issues/3
• [x] Get case data (from TA1)
• [x] Get hospitalization and deaths data (from TA1)
• [x] Get any fixed parameters (from TA1)
• [x] Get vaccination data (including vaccination percentage for deaths and hospitalizations) (from TA1)
• [ ] Get age-stratified vaccination, hospitalization, and deaths data (from TA1)
• [x] Get reinfection data (from TA1)
• [x] Get longer time period data (from TA1)
• [x] Swap models to AlgebraicJulia (from TA2)

[ChrisRackauckas]

TA3 data requests for TA1. For June 1st 2021 – June 1st, 2022, we need the following:

• Fixed or suggested parameters for any of the models
• Case data (susceptible, infected and recovered)
• Hospitalizations and deaths. Total hospitalization or deaths plus a value on the % of cases which required hospitalizations and mortality rate would be fine if a full time series of both is not possible.
• % difference in deaths due to hospitalization, i.e. the factor for the difference in the mortality rate in the population that is infected and not hospitalized vs the mortality rate in the population that is infected and hospitalized.
• Vaccination rate (% of individuals vaccinated), % difference in hospitalizations due to vaccination, % difference in deaths due to vaccination
• Age-stratified versions of the data, specifically age differences in % of cases which require hospitalization , % of cases which result in mortality, case data (particularly, number or percentage of infected by age), and in % of population vaccinated.

[ChrisRackauckas]

TA2 model JSONs can be found at https://github.com/DARPA-ASKEM/program-milestones/issues/9

[ChrisRackauckas]

Vaccination data: https://github.com/DARPA-ASKEM/program-milestones/pull/11

[liunelson]

@ChrisRackauckas Where are the results associated with the "real" training/test dataset? I only see placeholder synthetic results. https://chrisrackauckas.github.io/ASKEM_Evaluation_Staging/dev/Scenario3/Evaluation_Scenario_3/#Question-1

[ChrisRackauckas]

It just merged 8 minutes ago: https://github.com/ChrisRackauckas/ASKEM_Evaluation_Staging/pull/67. You can start running the script if you want to see it, takes just about 5 minutes. The full bound found that the TA2 model swap didn't pass a unit test due to a tolerance that was a bit too low (https://github.com/ChrisRackauckas/ASKEM_Evaluation_Staging/actions/runs/4067984576) and that needs to be fixed for the full builder to run again (already triggered https://github.com/ChrisRackauckas/ASKEM_Evaluation_Staging/actions/runs/4067975452/jobs/7005966223)

[liunelson]

The MITRE scenario specified "explore the space of closely related models that incorporate either deaths, hospitalizations, or both."

TA2 provided SIR, SIRD, SIRH, SIRDH variants and (as I understand) MITRE expects calibration and testing on all of them. Your notebook only considered SIR vs. SIRDH. Can you provide the test performance metric (L2 norm) for all these variants against the test dataset?

[ChrisRackauckas]

Yes, that's in the files already if you look at the code.

[liunelson]

Where is that? I'm looking at here: https://chrisrackauckas.github.io/ASKEM_Evaluation_Staging/dev/Scenario3/Evaluation_Scenario_3/#Question-2:-Add-Hospitalizations-and-Deaths

I only see

norm(solve(_prob, saveat = t_test)[S] - data_test[1][2]) + norm(solve(_prob, saveat = t_test)[I] - data_test[2][2]) + norm(solve(_prob, saveat = t_test)[R] - data_test[3][2])

norm(solve(_prob2, saveat = t_test)[S] - data_test[1][2]) + norm(solve(_prob2, saveat = t_test)[I] - data_test[2][2]) + norm(solve(_prob2, saveat = t_test)[R] - data_test[3][2]) + norm(solve(_prob2, saveat = t_test)[H] - data_test[4][2]) + norm(solve(_prob2, saveat = t_test)[D] - data_test[5][2])

I was expecting to see the same as above but for _probN where N is for each of SIR, SIRH, SIRD, SIRDH.

[ChrisRackauckas]

That's the later questions. Question 2 only has those two models if I'm reading the text correctly. Are you asking about Question 6?

[liunelson]

Nope, these models are meant for the Question 2 as I understand. Otherwise, it'd be a rather poor model-space selection process to just select between 2 things.