Unable to replicate the results

[wayne315315]

Dear COVIDAnalytics Team, First, thanks for you excellent job on this project.

We are trying to replicate the results on provinces in Canada from DELPHI v3.0 (the latest commit on master branch). We follow the instruction and execute the follow command on our terminal: (since it's the first run on Oct 22, it didn't use yesterday parameters) python3 DELPHI_model_V3.py -u wayne -o tnc -ci 0 -s100 1 -w 0 python3 DELPHI_model_V3.py -u wayne -o trust-constr -ci 0 -s100 1 -w 0 python3 DELPHI_model_V3.py -u wayne -o annealing -ci 0 -s100 1 -w 0

All 3 results got similar output, but all the results we got have a huge 15-day MAPE error, and totally different from the prediction downloaded from your website https://www.covidanalytics.io/projections In-Sample MAPE Last 15 Days ('Canada', 'Alberta'): 24.18 % In-Sample MAPE Last 15 Days ('Canada', 'Manitoba'): 52.33 % In-Sample MAPE Last 15 Days ('Canada', 'British Columbia'): 29.2 %

Can you give us some guidances on how to correctly execute the model ? (maybe using different branch other than master, or ) The DELPHI_output we got is on this delphi_output (including logs message & danger_map folders & diagrams showing discrepancy between prediction & true value we ran on)

We are looking forward to your reply. Thank you.

p.s. we also notice that only cases_cnt, death_cnt will really affect parameter estimation, maybe it's because the loss function for ODE parameter updating only consider these two fields ?

[MichaelLLi]

Hi Wayne, thanks for this. I believe the current difficulty is caused by the default initial bounds - they were decided a few months ago and intended to warm start the optimization at the start of the pandemic. It is not a good starting point at this stage anymore, and you should utilize a recent parameter file that we have. For reference, I have attached the latest file we have estimated yesterday.

Parameters_Global_V2_20201022.zip

[MichaelLLi]

You are correct in saying that only cases_cnt and death_cnt affect parameter estimation - we are closely monitoring the other data but we have still deemed recovery and hospitalization data to be unreliable at this stage to be utilized.

[wayne315315]

Hi Michael, Really appreciate your feedback ~ We'll run the code again with the updated file tonight. We'll let you know if the problem is solved or not ASAP. Thanks again for your help

[MichaelLLi]

Hi Wayne, is the issue resolved? Thanks!

[wayne315315]

Hi michael,

sorry I didn't reply. The issue is resolved. But the results on Hospitalization & ICU seems to be way different than the actual numbers Cases & Deaths have the similar trends as the ground truth, but a bit deviation on the offset has been notice.

Best, Wayne

Michael Lingzhi Li notifications@github.com 於 2020年11月9日 週一 上午6:26寫道：

Hi Wayne, is the issue resolved? Thanks!

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/COVIDAnalytics/DELPHI/issues/66#issuecomment-724046272, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGLPZHVW7D5FUQ74XNG2KUDSO73ZBANCNFSM4S3VEUDA .

[MichaelLLi]

The hospitalization and ICU numbers (along with recovery numbers) are not fitted to the data due to the fact that most countries have unreliable data. In particular, in most countries, where there is not a fully centralized hospitalized system we expect the hospitalization data to deviate from the ground truth up to 500% (we have first hand data on this).

[MichaelLLi]

That would explain why hospitalization/icu data numbers are different - we are utilizing a global mean average prediction. And for where have you seen deviation in cases/deaths on ground truth? We are happy to investigate.