Create an API to query in situ data for single city page

[amehta-scottlogic]

Acceptance Criteria

• An API exists with details: GET /air-pollutant/measurements
• Required query parameters should be date_from, date_to, location_type
• Optional query parameters should be location_names (this can support multiple parameters comma separated), api_source
• Parameters are validated and if not valid we'll return 422
• Success will return 200

• Response should be JSON array with following fields in each document:	Field	Value from database
  measurement_date	measurement_date
  location_type	location_type
  location_name	name
  no2	no2 value
  o3	o3 value
  pm2_5	pm2_5 value
  pm10	pm10 value
  so2	so2 value
  entity	metadata.entity
  sensor_type	metadata.entity
  site_name	location_name

• Mongo Query performance is analysed and index is confirmed to be being used

For this ticket to be done it must include automation

Test Checklist:

• [x] Test Analysis & discussion with developer
• [x] Write Test Cases / Charters
• [x] Test Cases & Charters Reviewed
• [x] Test Cases Executed
• [x] Test Charters Executed
• [x] Regression tests executed
• [x] Automation Tests added to project

[mnyamunda-scottlogic]

Test Analysis

Functionality

Having spoken to Ajay (Mike):

• date_from (datetime), date_to (datetime), location_type 'city'? - this call returns every site for every city?
• date_from (datetime), date_to (datetime), location_type 'city'?, location_names(string matching db name?), api_source (string matching db api_source?) - as above, but filters to specified city and api_source sites

User should be able to enter required parameters which will essentially filter/query the DB and return matches in JSON format. Charter 11: Explore GET air‐pollutant measurements In Situ API with creative parameters and headers to discover surprises

Parameters.

• Exploration charter: Explore responses and error messages upon entering various unwanted characters.
  • Verify that our parameters only allow certain values. e.g. string, date, int formats. ✅ Charter 11
  • What is the limit for location_names ✅ Charter 11
• Verify that optional parameters can be omitted. ✅ (Automation)
• Verify that multiple options can be entered via comma separation. e.g. London, Glasgow / OpenAQ, SomeAPI. ✅ (Automation)

Output

• Verify that on happy path, valid required params with/without optional params returns a 200 OK ✅ (Automation)
• Verify the format of the JSON response, expect it to match the criteria shown in above AC✅ (Automation) - not JSON schema validation, but asserted whole response
• Verify that measurement_date accurately represents the date_from and date_to values that were previously entered in the api query. ✅ (Automation)

Error

• Verify that invalid parameters are met with a 422 status. (What counts as invalid? Case sensitivity?) ✅ Charter 11
• Verify backend error logging. User must be led to the happy path when entering wrong query ✅ Charter 11 e.g. entering only one 1/3 and 2/3 required parameters. ✅ Charter 11 -> only date_from and measurement_date_to ✅ Charter 11 -> only date_from and location_type ✅ Charter 11 -> only measurement date_to and location type ✅ Charter 11 User must be made aware of the missing parameter each time. ✅ Charter 11
• Given that our queries only allow certain characters. We should expect appropriate errors to notify users of incorrect data type. (Bad input Error code+) ✅ Charter 11
• Verify correct error display given that the documents are unavailable. e.g. "There is no data for Texas"

Data

• Verify single city data displayed matches that of the database. ✅ (Automation)

  Side by side scan with mongoDB Investigate regularity of new document upsertion/modification How often do we expect in-situ database data to change?

• Possible JSON schema enforcement. Seed mongoDB data with various data types to discover surprises upon API call.

• Monitor query response time with postman? Exploratory charter here? Monitor performance with different data set sizes, e.g. 10, 20 , 30 seeded results

Must be regressed, important query function.

[mwalker-scottlogic]

forecast_api_seeded_data_tests.py, forecast_api_validation_tests.py, cams_known_grib_test.py, forecast_calibration_tests.py & open_aq_etl_tests.py tests all passed.

Executed manual test cases, had to make some changes to test assertions, but expected behaviour was observed

[mwalker-scottlogic]

Finding - api_source is in response unexpectedly

Description:

• Based on AC, api_source is not an anticipated field in response

Steps to Reproduce:

1. Make a Postman call that will return data e.g. in the form /air-pollutant/measurements?date_from=2024-06-21T14%3A00%3A00.000%2B00%3A00&date_to=2024-06-21T16%3A00%3A00.000%2B00%3A00&location_type=city
2. Observe response and compare to AC

Expected:

• measurement_date
• location_type
• location_name
• no2
• o3
• pm2_5
• pm10
• so2
• entity
• sensor_type
• site_name

Actual:

• measurement_date"
• location_type"
• location_name"
• api_source"
• no2"
• o3"
• pm2_5
• pm10
• so2"
• entity
• sensor_type
• site_name

[mwalker-scottlogic]

Finding - location_names param spelled incorrectly

Expected:

• needs to be location_names

Actual:

• needs to be location_name

[mwalker-scottlogic]

Exploratory testing Findings

Charter 11: Explore GET air‐pollutant measurements In Situ API with creative parameters and headers to discover surprises ✅

• date_from and date_to
  • 500 internal server error if you set the year to 0000
  • Can send a number as a date (+/-)
• location_names and api_sources don't seem to have any validation

[mwalker-scottlogic]

Finding - api_source 500

Steps to reproduce:

• Make a GET call to /air-pollutant/measurements endpoint with date_from (valid Datetime), date_to (valid Datetime), location_type (city) parameters and api_source set to 'OpenAQ'
• Make a GET call to /air-pollutant/measurements endpoint with date_from (valid Datetime), date_to (valid Datetime), location_type (city) parameters and api_source set to 'openaq'
• e.g http://127.0.0.1:8000/air-pollutant/measurements?date_from=2024-06-21T14%3A00%3A00.000%2B00%3A00&date_to=2024-06-21T16%3A00%3A00.000%2B00%3A00&location_type=city&api_source=OpenAQ

Expected:

• data returned, status 200

Actual:

for OpenAQ: 422 status code

for openaq: 500 status code

[mwalker-scottlogic]

We discovered that if you send a request to the air-pollutant/measurements endpoint and have seeded in the in_situ database data which has an api_source that also isn't "OpenAQ" you get a 500 response. This has been determined to be expected behaviour so tests have been added for this to make sure this continues. We need to prevent incorrect data entering the database using schema validation which has a ticket in the backlog.

All automated API tests are now passing