search

epiforecasts / covid-rt-estimates

National and subnational estimates of the time-varying reproduction number for Covid-19
https://epiforecasts.io/covid/
MIT License
34 stars 17 forks source link

UK UTLA level estimates not always updating #206

Closed seabbs closed 2 years ago

seabbs commented 3 years ago

I see that the UTLA level estimates haven't updated for the last two days: https://github.com/epiforecasts/covid-rt-estimates/tree/master/subnational/united-kingdom-local/cases/summary

Not clear what may be driving this.

joeHickson commented 3 years ago

last nights is still running. It looks like no new data for the previous two days:

2021-08-04 01:43:10 INFO Data has not been updated since last run. If wanting to run again then remove /home/rstudio/covid-rt-estimates/last-update/united-kingdom-local.rds

2021-08-03 07:04:19 INFO Data has not been updated since last run. If wanting to run again then remove /home/rstudio/covid-rt-estimates/last-update/united-kingdom-local.rds 2021-08-03 07:04:19 DEBUG Max date in data - 2021-07-29, last run date from file - 2021-07-29

seabbs commented 3 years ago

Are there still two pools of jobs running (hence the run at 1.43 and at 7?) If yes is the 1:43 run from the 3rd causing an issue?

It is now 3 days since the last update so that means at least two days without data which seems a bit surprising given the PHE dashboard (the source) has been updating...

It is good todays run is still going but am slightly concerned this is pointing to a deeper issue. It seems runs fail at critical points only. This may be because these are harder for the model to fit and hence take longer leading to time outs etc but not seen evidence of that.

joeHickson commented 3 years ago

The log files show two days with no update - extracted above. Note the two different dates - it's just that one got allocated a node much quicker (both queued at midnight).

The covid-r-job is processing no tasks - you can see there are no modified tasks in that pool in the past week. I will disable it to avoid any confusion.

seabbs commented 3 years ago

Great!

seabbs commented 3 years ago

Closing as updated elsewhere. Setting up external monitoring to check the last data update to flag if this is driven by something else.