FAIR DATA: Pipeline for processing archived datasets into a common format

[jeanpaulrsoucy]

Once a common output data format is established (#10), a huge number of workflows will need to be developed, one per dataset, in order to transform raw, archived data into FAIR data.

The exact nature of these data workflows has not yet been decided, but will likely include one or more of: SQL, Python, R and related tools.

These workflows require a few different features:

• Ability to link several datasets together if they represent a continuous data series (e.g., if the name/URL of the dataset changed over time) (see also comment in #7)
• Ability to incorporate manual data entry to fill gaps, errors and other entries requiring correction (these should be clearly indicated in metadata with a reason)
• Easy to establish data provenance (#12)

Since May 2021, automation has been used to maintain the Covid19Canada and CovidTimelineCanada datasets. This process involved writing and maintaining a significant amount of R code to process dozens of existing datasets, see the R packages Covid19CanadaETL, Covid19CanadaData and Covid19CanadaDataProcess to view this existing and ongoing effort.

[colliand]

Establishing the "common output data format" might involve ongoing discussions with a variety of stakeholders. An open discussion on data formats with all these stakeholders will likely be difficult to convene and it will be challenging to arrive at technical standards.

FAIR provides some scaffolding for how data should be structured to accelerate collaboration and discovery. Our project, ideally in collaboration with other CANMOD personnel, should strive to finesse these obstacles to standardization. CCODWG data streams are being used already by policymakers. I suggest that our project should make opinionated choices on the best paths forward toward FAIRification while trying to avoid coding ourselves into a corner. Technical debt in a project that delivers value is better than spinning our wheels in governance and standardization meetings...

[jeanpaulrsoucy]

Agreed @colliand. I think we can use a simple provisional standard that looks something like the below in the mean time. Always time to revise in response to feedback down the road...

• name: The name of the metric (e.g., cases, testing)
• country: The three-letter country code (e.g., CAN)
• region: The two-letter code for the province or territory (e.g., ON, AB)
• sub_region_1 [optional]: The sub-provincial administrative unit (e.g., health regions)
• sub_region_2 [optional]: The smaller sub-provincial administrative unit (e.g., municipalities within health regions)
• date: The date in YYYY-MM-DD format
• group_type [optional]: The type of the (optional) grouping variable (e.g., age group)
• group_value [optional]: The value of the (optional) grouping variable (e.g., 0-17, 65+, etc.)
• value: The cumulative value (e.g., cumulative number of cases, cumulative number of tests)
• value_daily: The daily value (e.g., daily number of cases, daily number of tests)

This is adapted from the existing format for CovidTimelineCanada, which itself is inspired by various sources including the Google community mobility reports. It's also similar to the existing format for Covid19Canada, which as you mentioned is already in wide use.