NASA-IMPACT / veda-data-airflow

Airflow implementation of ingest pipeline for VEDA STAC data
Other
10 stars 4 forks source link

Branch single assets to correct path #187

Closed slesaad closed 4 months ago

slesaad commented 4 months ago

Description

Issue

Even for collections with single item assets, if we provide the "assets" input with just a single asset, it still goes through the grouping path where the id_regex is required and the name is changed based on the id_template - making it impossible for the item_id to be the filename.

Solution

If the provided assets only contains one asset, follow the same process as when no assets is provided and a default asset is created.

Testing

Deployed to ghgc-smce-dev environment using veda-deploy. Tested with the following configs

emit-ch4plume-v1.json ```json { "assets": { "ch4-plume-emissions": { "description": "Methane plume complexes from point source emitters.", "regex": ".*.tif$", "title": "EMIT Methane Point Source Plume Complexes" } }, "bucket": "lp-prod-protected", "collection": "emit-ch4plume-v1", "filename_regex": ".*.tif$", "prefix": "EMITL2BCH4PLM.001/" } ```

Worked as expected with the correct item_id as seen here - https://dev.ghg.center/api/stac/collections/emit-ch4plume-v1/items.

Not passing any assets created a cog_default asset.