search

NASA-PDS / nucleus

Nucleus is a software platform used to create workflows for the Planetary Data (PDS).
https://nasa-pds.github.io/nucleus
Apache License 2.0
0 stars 0 forks source link

Nucleus: Track the DAG processing status of each PDS product for monitoring and troubleshooting #88

Open ramesh-maddegoda opened 4 months ago

ramesh-maddegoda commented 4 months ago

💡 Description

At the moments, multiple product labels are processed in Nucleus DAGs as batches. The Airflow UI shows the completion status of each batch and it is also possible to identify individual product failures in a given batch by checking task logs on Airflow UI.

However, when we process thousands of products within a short duration, it is more efficient to keep track of the processing status of each PDS product in a central place such as a database. Also, this will allow us to easily query for failures and troubleshoot.

This task is to design and implement an approach to keep track of processing status of individual PDS products.

⚔️ Parent Epic / Related Tickets

No response

tloubrieu-jpl commented 5 days ago

Ramesh is adding a status monitoring in the nucleus RDS database, as a new table to avoid conflict with the product completion tracking.