search

NASA-IMPACT / csdap-cumulus

SmallSat Cumulus Deployment
Other
1 stars 0 forks source link

Replace loop w/distributed map in DiscoverAndQueueGranules #278

Closed chuckwondo closed 8 months ago

chuckwondo commented 9 months ago

This restructures the DiscoverAndQueueGranules workflow to avoid the event transition quota on Step Functions, thus allowing us to ingest an entire collection with a single rule, rather than 1 rule per year.

Note: this will be ideally suited for migrating to CBA Prod, but might require changes in coordination with the CMR JSON file generation Airflow process, as that process currently only restores from glacier and generates metadata files 1 year at a time.

chuckwondo commented 9 months ago

@krisstanton and @jsrikish, when you're reviewing this PR, it's best to review the changes to docs/DISCOVERY.md rendered as HTML so you can see the diagram I added: https://github.com/NASA-IMPACT/csdap-cumulus/blob/full-scale-discovery/docs/DISCOVERY.md#improved-solution-unlimited-scalability

Fixes #274