Establish standard nomenclature for data

isudatateam / datateam

ISU Data Team Effort

MIT License

5 stars 3 forks source link

Establish standard nomenclature for data #16

Open giorgichi opened 7 years ago

giorgichi commented 7 years ago

Several variation of "N/A"s are used in the TD and CSCAP databases to denote not-applicable or missing values. Need to standardized non-numeric data entries across all datasets.

akrherz commented 7 years ago

This has been a long term "problem" for the technology stack used. In general, the ISU database stores everything as a string type, so logic is made for each materialization of the data requiring numeric datatypes. This includes the accounting of

values that are blank or null on the upstream source
values that contain < > lod
values that are . did not collect n/a

Sadly, the solution to this may be more strict type enforcement with the upstream datasheets, but that is difficult too.