Support processing data coming from multiple staging buckets with one Airflow environment

NASA-PDS / nucleus

Nucleus is a software platform used to create workflows for the Planetary Data (PDS).

https://nasa-pds.github.io/nucleus

Apache License 2.0

0 stars 0 forks source link

Support processing data coming from multiple staging buckets with one Airflow environment #99

Closed ramesh-maddegoda closed 2 weeks ago

ramesh-maddegoda commented 1 month ago

💡 Description

At the moment, Nucleus is processing data received on a single staging s3 bucket. However, we should have dedicated staging bucket for each PDS Node. Each PDS Node may upload their date to the dedicated staging buckets allocated the Node. It is required to support processing data coming from multiple staging buckets with one Airflow environment.

⚔️ Parent Epic / Related Tickets

No response