Erroneous measurements

[tjohnson-scottlogic]

How should we handle erroneous values? We currently filter out -1s before they get to the database, but we don't filter out 9999. This ticket is for documenting the different values and proposing how we handle them,

Acceptance Criteria

• The common high values have been researched and a wiki page created to describe the values we see and any information we've been able to determine (we could contact OpenAQ and ask them about it)
• We have a proposed way of handling erroneous values

[tjohnson-scottlogic]

Updated the Decision Records to say that we are not going to let 9999 values into the database. Going to delete the OpenAQ Invalid Readings page as it's no longer needed, original text below just in case:

Invalid Readings There are two readings that we are currently unsure about.

-1 9999 Currently during ETL we ignore any readings of -1 so this doesn't make it to the database.

For readings of 9999 this get's through to the database and presented on the UI. As 9999 is significantly higher than a normal reading, this can malform the display of a graph.

Should we also be ommiting the 9999 value? Generally readings don't go above several hundred, and very rarely breaches 1000. What should the cut off be?