Forecast ETL performance improvement - parallel interpolation

[amehta-scottlogic]

Seb's findings:

While looking at https://github.com/ECMWFCode4Earth/vAirify/issues/27 I found a potential way to reduce the processing time of the > forecast "transformation". Currently, this is done by using ThreadPoolExecutor to loop over all cities and therefore doing a separate > > interpolation for each city. But xarray does provide a way to interpolate all cities in parallel in a single call (see https://docs.xarray.dev/en/stable/user-guide/interpolation.html#advanced-interpolation). Implementing this would require quite some code changes so I made a small proof-of-concept script first comparing both methods. > > On my computer, this reduces processing times from 19 to 0.6 seconds. Would probably be good to have a quick chat about how this > could be best implemented, but I think it is worth it

PoC here: https://github.com/ECMWFCode4Earth/vAirify/blob/parallel-int/air-quality-backend/scripts/test-parallel-interp.py

Acceptance Criteria

• bilinear interpolation should be done for all cities in parallel

Test Checklist:

• [x] Test Analysis & discussion with developer
• [x] Write Test Cases / Charters
• [x] Test Cases & Charters Reviewed - [ ] Test Cases Executed
• [x] Test Charters Executed
• [x] Regression tests executed - [ ] Write Automation tests for regression

[mwalker-scottlogic]

Test Analysis

• Regression testing and re-running exploratory testing on this part of the system to establish if everything still works as expected, and is more performant

Test Case required:

• N/A

Test Charter:

• Charter 2: Explore backend performance to discover surprises

Regression candidate:

• No

[mwalker-scottlogic]

Previous exploratory testing notes from 28/05

Test notes from 04/06