the transfer of a deep archive package from the node to EN, including the opening a new GitHub issue, and
the posting of the deep archive package in EN’s online manifest repository.
💪 Motivation
For PDS deliveries to the NSSDCA, the steps on the PDS side -- Node open EN GitHub issue then EN checks and posts Deep Archive package online for NSSDCA to auto pull -- is mostly manual and may quickly become a burden for the weekly deliveries such as the Catalina Sky Survey (CSS) from PSI node. For example every week after producing a CSS deep archive package (~350k new products/week to start), Jesse Stone at PSI must manually open a new issue in GitHub and Catherine Suh at EN must promptly review and post the package online for the NSSDCA to automatically pull by the next day for processing and ingest, which the NSSDCA wants to complete before the next weekly deliver arrives.
Would it be possible to automate the transfer and posting of these weekly CSS deep archive packages, particularly after the we’ve successfully ingested a few deliveries (via the current process) and demonstrated the weekly SIPs are stable? Also Jesse noted the Catalina Sky Survey will not be the last continuous pipeline where NSSDCA deliveries will be made on a very frequent basis. This is another reason to consider automation.
📖 Additional Details
From private email Jordan on 2021-11-18:
Our ultimate goal is to have the registry drive the generation of SIPs/AIPs, where EN will generate all the products for the nodes using the central registry. For most, this will still be manually triggered when a node deems a bundle “ready”, but for this use case, we can have a simple cron that executes regularly and re-generates the SIPs/AIPs.
That being said, feel free to create a ticket for this as well, and we can consider it. But I would prefer EN and the nodes focus on ensuring our registry system is robust and accurate, which will dramatically streamline many of the manual processes currently instantiated by the PDS.
At 2021-12-02 telecon with Jordan, PSI Node, and NSSDCA, it was agreed that Jesse Stone and Jordan should coordinate on the online location/URL of production CSS Deep Archive packages -- for EN to automatically pull those packages and drop them in the special online manifest repository for CSS (prescribed by issue https://github.com/NASA-PDS/operations/issues/200). Here is a sketch by Jesse of his ideas on this topic:
For the PDS4>NSSDCA process, automate:
💪 Motivation
For PDS deliveries to the NSSDCA, the steps on the PDS side -- Node open EN GitHub issue then EN checks and posts Deep Archive package online for NSSDCA to auto pull -- is mostly manual and may quickly become a burden for the weekly deliveries such as the Catalina Sky Survey (CSS) from PSI node. For example every week after producing a CSS deep archive package (~350k new products/week to start), Jesse Stone at PSI must manually open a new issue in GitHub and Catherine Suh at EN must promptly review and post the package online for the NSSDCA to automatically pull by the next day for processing and ingest, which the NSSDCA wants to complete before the next weekly deliver arrives.
Would it be possible to automate the transfer and posting of these weekly CSS deep archive packages, particularly after the we’ve successfully ingested a few deliveries (via the current process) and demonstrated the weekly SIPs are stable? Also Jesse noted the Catalina Sky Survey will not be the last continuous pipeline where NSSDCA deliveries will be made on a very frequent basis. This is another reason to consider automation.
📖 Additional Details
From private email Jordan on 2021-11-18: Our ultimate goal is to have the registry drive the generation of SIPs/AIPs, where EN will generate all the products for the nodes using the central registry. For most, this will still be manually triggered when a node deems a bundle “ready”, but for this use case, we can have a simple cron that executes regularly and re-generates the SIPs/AIPs.
That being said, feel free to create a ticket for this as well, and we can consider it. But I would prefer EN and the nodes focus on ensuring our registry system is robust and accurate, which will dramatically streamline many of the manual processes currently instantiated by the PDS.
At 2021-12-02 telecon with Jordan, PSI Node, and NSSDCA, it was agreed that Jesse Stone and Jordan should coordinate on the online location/URL of production CSS Deep Archive packages -- for EN to automatically pull those packages and drop them in the special online manifest repository for CSS (prescribed by issue https://github.com/NASA-PDS/operations/issues/200). Here is a sketch by Jesse of his ideas on this topic:
⚖️ Acceptance Criteria
tbd
⚙️ Engineering Details