search

ECMWFCode4Earth / vAirify

code repository for 2024 Code for Earth project #16
MIT License
1 stars 0 forks source link

As a system user the latest version of the CAMs job is deployed automatically to the linux machine so that data is automatically collected #59

Closed rstrange-scottlogic closed 3 weeks ago

rstrange-scottlogic commented 1 month ago

Requires the output of this story : https://github.com/orgs/ECMWFCode4Earth/projects/12/views/6?pane=issue&itemId=65384063

Once we know whether or not we are going to use docker we will add a github action to deploy the CAMs ETL into the linux machine.

Acceptance Criteria:

amehta-scottlogic commented 3 weeks ago

Testing

Was able to connect to Linux box and see logs from process sudo docker logs 7d54d092b3a8

Snippet:

time="2024-06-14T00:30:01Z" level=info msg="2024-06-14 00:30:01,882 - INFO - Finding data for 153 cities" channel=stdout iteration=2 job.command="conda run --no-capture-output -n air-quality-backend python \"/usr/src/app/scripts/run_forecast_etl.py\"" job.position=0 job.schedule="30 0,12 * * *"
time="2024-06-14T00:30:01Z" level=info msg="2024-06-14 00:30:01,883 - INFO - Extracting pollutant forecast data" channel=stdout iteration=2 job.command="conda run --no-capture-output -n air-quality-backend python \"/usr/src/app/scripts/run_forecast_etl.py\"" job.position=0 job.schedule="30 0,12 * * *"
time="2024-06-14T00:30:01Z" level=info msg="2024-06-14 00:30:01,884 - INFO - Loading data from CAMS to file single_level_41_from_2024-06-13_12.grib" channel=stdout iteration=2 job.command="conda run --no-capture-output -n air-quality-backend python \"/usr/src/app/scripts/run_forecast_etl.py\"" job.position=0 job.schedule="30 0,12 * * *"
time="2024-06-14T00:30:33Z" level=info msg="2024-06-14 00:30:33,932 - INFO - Loading data from CAMS to file multi_level_41_from_2024-06-13_12.grib" channel=stdout iteration=2 job.command="conda run --no-capture-output -n air-quality-backend python \"/usr/src/app/scripts/run_forecast_etl.py\"" job.position=0 job.schedule="30 0,12 * * *"
time="2024-06-14T00:30:52Z" level=info msg="2024-06-14 00:30:52,361 - INFO - Transforming forecast data" channel=stdout iteration=2 job.command="conda run --no-capture-output -n air-quality-backend python \"/usr/src/app/scripts/run_forecast_etl.py\"" job.position=0 job.schedule="30 0,12 * * *"
time="2024-06-14T00:30:53Z" level=info msg="2024-06-14 00:30:53,588 - INFO - Persisting forecast data" channel=stdout iteration=2 job.command="conda run --no-capture-output -n air-quality-backend python \"/usr/src/app/scripts/run_forecast_etl.py\"" job.position=0 job.schedule="30 0,12 * * *"
time="2024-06-14T00:30:53Z" level=info msg="2024-06-14 00:30:53,588 - INFO - Persisting 6273 documents" channel=stdout iteration=2 job.command="conda run --no-capture-output -n air-quality-backend python \"/usr/src/app/scripts/run_forecast_etl.py\"" job.position=0 job.schedule="30 0,12 * * *"
time="2024-06-14T00:31:17Z" level=info msg="2024-06-14 00:31:17,651 - INFO - 6273 documents upserted, 0 modified" channel=stdout iteration=2 job.command="conda run --no-capture-output -n air-quality-backend python \"/usr/src/app/scripts/run_forecast_etl.py\"" job.position=0 job.schedule="30 0,12 * * *"

Confirmed job running twice daily (previous run at 12:30)

Confirmed data in database: image