Convert PPM measurements from OpenAQ data based on CAMS air density

[amehta-scottlogic]

Description

From the last call with the mentors, they are keen to include PPM values from OpenAQ. We want to store the data in µg/m³ so all units are the same. The problem is we need a value for air density from that location to do the conversion.

The approach to take for now will be to query CAMS and use their surface temperature + pressure values (as we do already for forecast data) and when units are PPM we convert. The cams data should be queried with the lat/longs of the measurement stations themselves (rather than the city lat/long) for the best approximation.

While this is not likely to be 100% accurate, it should mean that more data can be used.

Formula: ppb = ppm * 1000

Weights no2 = 46.01 g/mol so2 = 64.07 g/mol o3 = 48 g/mol

https://www.apis.ac.uk/unit-conversion https://www.breeze-technologies.de/blog/air-pollution-how-to-convert-between-mgm3-%C2%B5gm3-ppm-ppb/

Acceptance Criteria

• PPM measurements from OpenAQ are converted to ug/m3 using data from CAMS
  • Make calls in parallel for both
  • CAMS data fetched is cached similarly to forecast etl process
• The structure of each pollutant in each document changes from numeric value to nested document with fields "value" and new field 'original_unit' value of µg/m³ or ppm EDIT: fields "value", "unit", and new field 'original_unit' value of µg/m³ or ppm, and new field "original_value"

EDIT - only ppm measurements for so2, no2 and o3 are converted. We now filter out any instance of pm10 or pm2_5 and do not store in the database

Test Checklist:

• [x] Test Analysis & discussion with developer
• [x] Write Test Cases / Charters
• [x] Test Cases & Charters Reviewed
• [x] Test Cases Executed
• [x] Test Charters Executed
• [x] Regression tests executed
• [x] Automation Tests added to project

[mwalker-scottlogic]

forecast_api_seeded_data_tests.py, forecast_api_validation_tests.py, cams_known_grib_test.py, forecast_calibration_tests.py & open_aq_etl_tests.py tests all passed.

Executed manual test cases, had to make some changes to test assertions, but expected behaviour was observed

[mwalker-scottlogic]

Test Analysis:

• Manual test required to by hand get ppm data from a grib and check that the data in the database has the correct original value and converted value
• Automated integration tests using cached data that assert that the process doesn't get broken and known values are stored
• Exploratory testing not needed

Elements to verify:

• check that we are not converting values that are already in ug/m3 already tested elsewhere
• check that we are converting values that are ppm
• check that conversion is correct
• check the document structure in database

[mnyamunda-scottlogic]

Looking good, we already have some code snippets that may expedite the process too! 😎

[mwalker-scottlogic]

Finding: pm10 and pm2.5 original_values are not being converted - RESOLVED ✅

Steps to reproduce:

1. Mock the API response from OpenAQ, with data that has units of ppm, and all pollutants
2. Observe the data in the database

Expected:

• all ppm measurements converted to µg/m³

Actual:

• Only no2, so2 and o3 measurements are converted
• pm10 and pm2_5 unit and value is identical to original_unit and original_value = not converted

I am concerned this data could make it through to the UI.

The document structure is slightly different than described, with fields value, unit, original_value & original_unit

[mwalker-scottlogic]

Test Conclusion

• We have an automated test that verifies the values µg/m³ generated from ppm values using a specified estimated_surface_pressure_pa and estimated_temperature_k
• A manual verification is needed to ensure that the estimated_surface_pressure_pa and estimated_temperature_k are correct. They are generated by interpolating data from a given GRIB file, for a given latitude, longitude and time. See Manual testing of interpolating surface temperature and pressure values from CAMS data to use in converting OpenAQ ppm to µg m³ for details - test passed
• Implemented change to document structure in database: