Add/generic vector ingests

NASA-IMPACT / veda-data-airflow

Airflow implementation of ingest pipeline for VEDA STAC data

Other

10 stars 4 forks source link

Add/generic vector ingests #220

Closed paridhi-parajuli closed 3 months ago

paridhi-parajuli commented 3 months ago

Summary: Refactor code to support vector data ingest Addresses Issue

Changes

Generalized code for vector data ingest and removed eis-related functionalies.
Change default value for EXTERNAL_ROLE_ARN passed to the vector ingest ECS
Add configs for loading data into features database
Add function to assume AWS IAM role and return temporary credentials

PR Checklist

[ ] Update CHANGELOG
[ ] Unit tests
[ ] Ad-hoc testing - Deploy changes and test manually
[ ] Integration tests

smohiudd commented 3 months ago

I just added some general comments on documentation. The way the original EIS vector dag is labelled it's hard to know the difference between that and the generic. So in the docs would be good to be specific that the ingest_vector DAG has logic that is tailored to EIS Fire datasets only