Update/Combine meta-data

SPI-Birds / pipelines

Pipelines for generating a standard data format for bird data

2 stars 6 forks source link

Update/Combine meta-data #131

Open LiamDBailey opened 3 years ago

LiamDBailey commented 3 years ago

Some thoughts about meta-data:

Is there any information that we are missing with the current meta-data?
How can we ensure the meta-data are up to do (i.e. when we get a new year of data)?
Should our meta-data excel spreadsheet and version control meta-data be combined?
Should we include column 'translations' in the meta-data (i.e. explanation of the column names in the primary data)?

LiamDBailey commented 3 years ago

@ChloeRN suggested that we should include column 'translation' in the meta-data, but it seems infeasible to require data owners to provide this (some primary data has 100s of columns!). As an alternative, developers can keep track of the columns they use in the pipeline (and verify with the data owners during the pipeline verification process) and we can include this subset of column descriptions in the meta-data.

LiamDBailey commented 3 years ago

If we plan to do this it should be part of the standard workflow document so devs remember to keep track of the columns they use and describe them.