Closed amehta-scottlogic closed 1 month ago
current status: β
Overall Coverage
Lines Covered Coverage Threshold Status 266 200 75% 0% π’ New Files
File Coverage Status air-quality-backend/src/database/location.py 100% π’ TOTAL 100% π’ Modified Files
File Coverage Status air-quality-backend/src/database/air_quality_dashboard_dao.py 0% π’ air-quality-backend/src/etl/forecast/forecast_adapter.py 100% π’ air-quality-backend/src/etl/forecast/forecast_dao.py 100% π’ air-quality-backend/src/etl/forecast/forecast_data.py 93% π’ TOTAL 73% π’ updated for commit:
69e77c2
by actionπ
Description
Performance was particularly slow because xarray lazily loads data from a dataset. We then try to access each dataset for each city which without eagerly loading is quite slow.
By explicitly calling load, we use more memory but performance improves. Introducing threads also significantly improves the runtime.
We now process all 153 in around 2 minutes instead of 10.
Default
15 cities per minutes
With eager load
25 cities per minute
With eager load and thread pool
76 cities per minute
Output