As a system user the latest version of the In-Situ job is deployed automatically to the linux machine so that data is automatically collected

[amehta-scottlogic]

Requires the output of this story : https://github.com/orgs/ECMWFCode4Earth/projects/12/views/6?pane=issue&itemId=65384063

This includes containerising the in-situ job if that is what we've done with the CAMs job.

Acceptance Criteria:

• Containerise the In-situ job, and update all docs as relevant
• Whenever a change is made to the In-Situ ETL pipeline and checked into main the CI pipeline automatically deploys this latest version into the linux machine. This includes setting up a relevant API key for Open AQ, and a relevant connection string for the Mongo DB. These are likely to be GitHub secrets and configurable in the pipeline.
• Job set to run every hour

[amehta-scottlogic]

Testing

Was able to connect to Linux box and see logs from process sudo docker logs d718631000d6

Snippet:

time="2024-06-14T11:00:01Z" level=info msg="2024-06-14 11:00:01,610 - INFO - Retrieving Open AQ in situ data" channel=stdout iteration=24 job.command="conda run --no-capture-output -n air-quality-backend python \"/usr/src/app/scripts/run_in_situ_etl.py\"" job.position=0 job.schedule="0 */1 * * *"
time="2024-06-14T11:00:01Z" level=info msg="2024-06-14 11:00:01,614 - INFO - Extracting in situ pollutant data between 2024-06-13_11:00 and 2024-06-14_11:00" channel=stdout iteration=24 job.command="conda run --no-capture-output -n air-quality-backend python \"/usr/src/app/scripts/run_in_situ_etl.py\"" job.position=0 job.schedule="0 */1 * * *"
  time="2024-06-14T11:01:44Z" level=info msg="2024-06-14 11:01:44,441 - INFO - Transforming in situ data complete" channel=stdout iteration=24 job.command="conda run --no-capture-output -n air-quality-backend python \"/usr/src/app/scripts/run_in_situ_etl.py\"" job.position=0 job.schedule="0 */1 * * *"
time="2024-06-14T11:01:44Z" level=info msg="2024-06-14 11:01:44,479 - INFO - Persisting open AQ in situ data" channel=stdout iteration=24 job.command="conda run --no-capture-output -n air-quality-backend python \"/usr/src/app/scripts/run_in_situ_etl.py\"" job.position=0 job.schedule="0 */1 * * *"
time="2024-06-14T11:01:44Z" level=info msg="2024-06-14 11:01:44,479 - INFO - Persisting 33343 documents" channel=stdout iteration=24 job.command="conda run --no-capture-output -n air-quality-backend python \"/usr/src/app/scripts/run_in_situ_etl.py\"" job.position=0 job.schedule="0 */1 * * *"
time="2024-06-14T11:02:50Z" level=info msg="2024-06-14 11:02:50,004 - INFO - 1367 documents upserted, 31976 modified" channel=stdout iteration=24 job.command="conda run --no-capture-output -n air-quality-backend python \"/usr/src/app/scripts/run_in_situ_etl.py\"" job.position=0 job.schedule="0 */1 * * *"
time="2024-06-14T11:02:50Z" level=info msg="job succeeded" iteration=24 job.command="conda run --no-capture-output -n air-quality-backend python \"/usr/src/app/scripts/run_in_situ_etl.py\"" job.position=0 job.schedule="0 */1 * * *"

Confirmed running hourly as last run was at 10 UTC before that.

Confirmed data in database at correct time: