search

NASA-IMPACT / veda-data-airflow

Airflow implementation of ingest pipeline for VEDA STAC data
Other
10 stars 4 forks source link

Add transfer task definition #121

Closed smohiudd closed 8 months ago

smohiudd commented 8 months ago

Add task definition for transfer DAG.

slesaad commented 8 months ago

looks like there was a PR already that did this - https://github.com/NASA-IMPACT/veda-data-airflow/pull/95/files? can that be closed once this is merged? or are there other changes we wanna bring in from there?

anayeaye commented 8 months ago

@slesaad it looks like that pr you linked also has some cog transformation work so maybe tackle it second? It also shuffles the task and ecr names (underscores for hyphens) so we'll have to watch for that. I'd be in favor of seeing the transfer task work first though for our first production backfill