We generate reports, and downstream consumers are affected by data format changes. (e.g., new fields, see Slack note). Changing the schema impacts them, which may reduce traction for us as well!
If we have a schema and versioning, we can validate, and can report. This could be an automatic script, shouldn't require too much handholding.
I don't believe that this would be tricky to do. We can use ajv to potentially generate a schema, and save in schemas. Reports could include a "version" field.
I believe @shaperilio was doing some work around this as well.
Original issue https://github.com/covidatlas/coronadatascraper/issues/767, transferred here on Sunday Apr 12, 2020 at 16:39 GMT
Description
We generate reports, and downstream consumers are affected by data format changes. (e.g., new fields, see Slack note). Changing the schema impacts them, which may reduce traction for us as well!
If we have a schema and versioning, we can validate, and can report. This could be an automatic script, shouldn't require too much handholding.
Why do you need this feature or component?
Additional context/notes
ajvto potentially generate a schema, and save in schemas. Reports could include a "version" field.