Automate Geoglam & NO2 dataset ingestion

smohiudd commented 4 months ago

Description

NO2 (#89) and Geoglam (#167, #173) datasets requires monthly ingestion as new assets are created. This is currently a manual process however should be automated. veda-data-airflow has a feature that allows scheduled ingestion by creating dataset specific DAGs. The file must still be transferred to the collection s3 bucket. A json file must be uploaded to the airflow event bucket. Here is an example json:

{
    "collection": "emit-ch4plume-v1",
    "bucket": "lp-prod-protected",
    "prefix": "EMITL2BCH4PLM.001/",
    "filename_regex": ".*.tif$",
    "schedule": "00 05 * * *",
    "assets": {
        "ch4-plume-emissions": {
            "title": "EMIT Methane Point Source Plume Complexes",
            "description": "Methane plume complexes from point source emitters.",
            "regex": ".*.tif$"
        }
    }
}

Acceptance Criteria

[x] transfer no2-monthly and no2-monthly-diff from s3://covid-eo-data bucket to s3://veda-data-store-staging and s3://veda-data-store using MWAA transfer dag
[x] scheduled ingestion (bi weekly) json files are created and uploaded for NO2 (no2-monthly, no2-monthly-diff) in mwaa event bucket for staging (UAH) and production (MCP)
[ ] scheduled ingestion (bi weekly) json files are created and uploaded for Geoglam (geoglam) in mwaa event bucket for staging (UAH) and production (MCP)
[ ] Validate that new ingestions are initiated in staging and production. Add new geoglam files from #167, #173 into veda-data-store-staging bucket.
[ ] transfer, automated ingestion json configs are in veda-data