Closed amehta-scottlogic closed 1 month ago
current status: β
Overall Coverage
Lines Covered Coverage Threshold Status 305 218 71% 0% π’ New Files
No new covered files...
Modified Files
File Coverage Status air-quality-backend/scripts/run_forecast_etl.py 0% π’ air-quality-backend/scripts/run_in_situ_etl.py 0% π’ air-quality-backend/scripts/run_purge_old_data.py 0% π’ TOTAL 0% π’ updated for commit:
6cbd93c
by actionπ
Description
This is quite a big PR. The purpose for it was to make it possible to run our ETL processes via the command line as well as through PyCharm. This now does work in both ways
Changes
Explanation
The issue with running via the command line was because of our project structure. Our source code was placed in the 'src' directory under several packages (database, api and etl). Our application scripts that needed to reference these python modules had to import each other with 'src' prepended to the statement, because the 'src' directory was not included in the PYTHONPATH (and actually this is the whole point of src layout).
The solution was to restructure the repo and bring in poetry as is advised here. This has the added advantage of making dependency management much simpler and more like npm.
It should also be possible to leverage poetry for build/packaging purposes when we need to