Metadata Schema

[awm33]

Proposed schema for tracking datasets / metadata. Top level are tables/objects, second level are fields, third level are notes or enum values. If something has a ?, it means uncertainty and looking for feedback.

• Source (raw data source)
  • Name - Text - Required
  • Description - Text - Required
  • Status - Enum - Required
    • Published
    • In Development
    • Proposed
  • Link(s) - Array of Text - Required
  • Format - Enum - Required
    • Delimited Text (CSV, TSV, etc)
    • Excel (XSL, XLSX)
    • Website Table
    • Website Unstructured
    • PDF Table
    • PDF Unstructured
    • API
    • RSS
    • Other
  • Format Notes - Text
  • Geographic Region(s) (if applicable) - Links to Geographic regions
  • Update Frequency - Text
  • Date Start- Date
  • Date End - Date
  • Organization / Authority - Text - Required (We may want to create an organization table)
  • Organization / Authority Link - Text - Required
  • License - Text
  • Import Process - Text - Description of import process
  • Github Link(s) - Array of Text (Link(s) to data/scripts)
• Category
  • Parent - Link to Parent Category
  • Name - Text - Required
• Table
  • Name - Text - Required
  • Description - Text - Required
  • Input Source(s) - Links to input Sources - Required
  • Input Table(s) - Links to Tables (if data is derived from other tables / non-raw sources)
  • Category - Link to category - Required
  • Table Name / Slug - Text (something to reference the file path or API name) - Required
  • Geographic Region(s) (if applicable) - Links to Geographic regions
  • Start Date (if applicable) - Date
  • End Date (if applicable) - Date
• Fields
  • Table - Link to table - Required
  • Name - Text - Required
  • Description - Text - Required
  • Type - Enum - Required
    • Text
    • Integer
    • Decimal
    • Date
    • Datetime
    • Boolean
    • Enum
    • Possible Values - Array of Text (only for Enum type)
• Geographic Region
  • Parent - Link to parent Geographic Region (can be tricky with overlaps)
  • Name - Text - Required
  • Keywords - Array of Text
  • Type - Enum - Required
    • Country (based on ISO-3166)
    • State / Province / Subdivision (based on ISO-3166)
    • Administrative (ex counties)
    • City
    • Postal Code
  • Code - Text

[d1gits]

Proposed schema for tracking datasets / metadata. Top level are tables/objects, second level are fields, third level are notes or enum values. If something has a ?, it means uncertainty and looking for feedback.

• Source (raw data source)
  • Name - Text
  • Description - Text
  • Status - Enum
    • Published
    • In Development
    • Proposed
  • Link(s) - Array of Text
  • Format - Enum
    • Delimited Text (CSV, TSV, etc)
    • Excel (XSL, XLSX)
    • Website Table
    • Website Unstructured
    • PDF Table
    • PDF Unstructured
    • API
    • RSS
    • Other
  • Format Notes - Text
  • Geographic Region(s) (if applicable) - Links to Geographic regions
  • Update Frequency - Text
  • Record Date (when the data was recorded)
  • Date Start- Date
  • Date End - Date
  • Organization / Authority - Text (optional is fine)
  • Organization / Authority Link - Text
  • License - Text (optional is fine)
  • Import Process - Text - Description of import process. (optional is fine)
  • Github Link(s) - Array of Text (Link(s) to data/scripts)
• Category
  • Parent - Link to Parent Category
  • Name - Text
• Table
  • Name - Text
  • Description - Text
  • Input Source(s) - Links to input Sources
  • Input Table(s) - Links to Tables (if data is derived from other tables / non-raw sources)
  • Category - Link to category
  • Table Name / Slug - Text (something to reference the file path or API name)
  • Geographic Region(s) (if applicable) - Links to Geographic regions
  • Start Date (if applicable) - Date (optional is fine)
  • End Date (if applicable) - Date (optional is fine)
• Fields
  • Table - Link to table
  • Name - Text
  • Description - Text
  • Type - Enum
    • Text
    • Integer
    • Decimal
    • Date
    • Datetime
    • Boolean
    • Enum? (not sure @nikovanmeurs )
• Geographic Region
  • Parent - Link to parent Geographic Region (can be tricky with overlaps)
  • Name - Text
  • Type - Enum
    • Country (based on ISO-3166)
    • State / Province / Subdivision (based on ISO-3166)
    • County (US health departments are often county level) (not sure if this works well globally)
    • City
    • Postal Code (make it optional if available)
  • Code - Text

[nikovanmeurs]

Regarding the record date: are we treating datasets as if they are immutable? If not it'd be desirable to allow for a recordDates property which accepts a list of dates instead.

Regarding The Enum typed Field type: When recording enums like this, a data structure to define the possible enum values seems to be desirable. Otherwise an enum value does not differ from a plain text value.

@d1gits Postal Code is a type of Geographic Region. Ie. if a data set is tied to a postal code, you'd add a Geographic Region of type Postal Code, with a parent which can be of type City, Administrative Region, State or Country.

I'd like to suggest adding a Geographic Regioin of type Global or World as well, which can be useful to tie global reports to.

[awm33]

@nikovanmeurs No, hence the Update Frequency.

An enum would be a string, but to distinguish it from free text, as in there are a set number of values. Even if we just use a string type, we should list the possible values in the description.

[nikovanmeurs]

@awm33 clear on both points. Let's add the enum type in there to emphasize the fact that it's value should be one of a predefined set and keep the Record Date as is.

[awm33]

@nikovanmeurs Added "Possible Values - Array of Text (only for Enum type)" so that list can be structured.

I think I need clarity on what "Record Date" means?

[nikovanmeurs]

@awm33 I was referring to this field in @d1gits's proposal:

• Record Date (when the data was recorded)

I get the impression that the two fields you proposed on Slack (Publication Date and Scrape Date) are a better fit here though.

[nikovanmeurs]

It might be desirable to store relations between datasets as well. Say we produce a dataset of our own which is based on 3 other datasets, the organisation would be COVID-19-Data, however for other people to determine the reliability of this dataset we should be able to point them to the source data sets.

[awm33]

@nikovanmeurs That's already covered in Tables

Input Source(s) - Links to input Sources - Required
Input Table(s) - Links to Tables (if data is derived from other tables / non-raw sources)

[awm33]

Could even build a lineage graph using those fields

[nikovanmeurs]

Check. Missed that one 👍