search

openaq / project-metadata-format

This repo houses project discussions for building a metadata format for the metadata editor.
2 stars 0 forks source link

How to handle versioning? #5

Open jflasher opened 5 years ago

jflasher commented 5 years ago

I am wondering how best to handle versioning? If the metadata is the same for 1 year, then the instrument changes, how is that handled? Is that somehow tied to the measurements within OpenAQ?

olafveerman commented 5 years ago

We had a discussion about this the other day. The difficulty is knowing whether an edit reflects a change on the ground, or if it simply complements or improves existing metadata.

The initial idea was for the API to always return the latest version of the metadata. On the backend we'd have some way to store edits. Eg through a log or versions.

Curious to know more about the use case. How often does a change to an instrument occur? And how often would this impact the metadata?

cc @RocketD0g @jflasher @sethvincent

jflasher commented 5 years ago

Not to put a solution out there before we fully understand the problem, but it may be an interesting use case for https://aws.amazon.com/qldb as it has a transparent update chain/path. I presume there is a way to point to older versions of the data/record. This could also be achieved with S3 versioning or a million other ways, but just wanted to throw it out there!

RocketD0g commented 5 years ago

On how often an instrument changes for a given pollutant or suite of pollutants, in some cases (e.g. PM2.5 and PM10 are often measured from the same instrument, while, say, CO2 and PM2.5 are not), I'd say typically it'd change every few years, when older equipment wears out. Realistically, we'll likely have a lag with when anyone reports that change to us (if ever).

At the station level (which is often composed of multiple instruments monitoring different pollutants), I'd expect every these changes to occur in reality every 2-5 years. Often times, these changes can occur in bulk, instead of instrument-by-instrument, when a given station gets funding to update all of their equipment. There can be cases where for one reason or another, a single instrument is replaced though, while the others stay the same.