epiforecasts / covid-rt-estimates

National and subnational estimates of the time-varying reproduction number for Covid-19
https://epiforecasts.io/covid/
MIT License
34 stars 17 forks source link

I116 summary dataset definitions #119

Closed joeHickson closed 4 years ago

joeHickson commented 4 years ago

closes #116

"Collated Derivative" = Summary dataset produced by stitching together multiple other datasets results.

Note UK summary changes to more standardised format - currently this outputs a dated file in the old archival style. With this change it will just output united-kingdom/collated/rt.csv

This includes publishing the collated dataset to DV - https://demo.dataverse.org/dataset.xhtml?persistentId=doi:10.70122/FK2/6GNVT2

This task also brings a lot of tidy up in areas that needed expanding with the collated derivatives - entities are broken out to individual files, lists are in their own subfolder too.

Also switched to using

if (!exists("add_uk", mode = "function")) source(here::here("R", "utils.R")) 

at the top of files to list required includes - this should ensure they are only pulled in once but guarantees they are available in that file. By starting to do this in each file it's possible to pick and choose which bits you use when cobbling random test scripts together and ensures each piece can be used in more isolation. e.g. source("R/collate-derivative.R") doesn't require to run all the source(x) from the top of run-region-updates before calling it

This is handy for testing / debugging as you can call the script much like it would be in the bigger suite

source("R/collate-derivative.R") # get the script to collate derivatives
source("data/runtime/config.R") # load the dataverse config (optional, allows testing of publication)
source("R/lists/collated-derivative-list.R") # load the list of defined collated derivatives
futile.logger::flog.threshold(futile.logger::TRACE) # set your log level for debugging
collate_derivative(COLLATED_DERIVATIVES[[1]]) # call the function the same way it is from run-region-updates.R and check it works
joeHickson commented 4 years ago

Todo: Reimplement old uk summary temporarily

joeHickson commented 4 years ago

reintroduced uk method as a deprecated block for the time being

joeHickson commented 4 years ago

@seabbs I think this is good to merge

seabbs commented 4 years ago

Nice stuff - all looks good to me.