Due to the extremely large number products accumulating in the Catalina Sky Survey (CSS) bundle (~315K new products/week for ~1.7 TB), both the NSSDCA and PSI Node concluded that 'delta' Submission Information Package (SIP) products are necessary for producing the Deep Archive package in timely manner and for the NSSDCA to ingest the SIP delivery before that next weekly CSS SIP hits their queue. Therefore Jesse Stone augmented the pds-deep-archive software to produce a 'delta' SIP product where the manifest file contains only:
new basic products added to the bundle since the last delivery,
all collection products that contain those new basic products, and
the bundle product.
💪 Motivation
Ideally the enhancements Jesse Stone made to the pds-deep-archive software to produce a 'delta' SIP product shoud be incorporated into the production verison of the software that all nodes use. It is very likely that PDS will need to archive other large PDS4 bundles like the Catalina Sky Survey, e.g. other all-sky surveys or long-duration missions, where generating a 'delta' SIP product is the most effecient means to deliver data to to the NSSDCA.
📖 Additional Detail
Jesse offered to share his augmented code with EN and provide insight on 'delta' SIPs. Here is a sketch by Jesse Stone of his ideas on this topic:
Due to the extremely large number products accumulating in the Catalina Sky Survey (CSS) bundle (~315K new products/week for ~1.7 TB), both the NSSDCA and PSI Node concluded that 'delta' Submission Information Package (SIP) products are necessary for producing the Deep Archive package in timely manner and for the NSSDCA to ingest the SIP delivery before that next weekly CSS SIP hits their queue. Therefore Jesse Stone augmented the pds-deep-archive software to produce a 'delta' SIP product where the manifest file contains only:
💪 Motivation
Ideally the enhancements Jesse Stone made to the pds-deep-archive software to produce a 'delta' SIP product shoud be incorporated into the production verison of the software that all nodes use. It is very likely that PDS will need to archive other large PDS4 bundles like the Catalina Sky Survey, e.g. other all-sky surveys or long-duration missions, where generating a 'delta' SIP product is the most effecient means to deliver data to to the NSSDCA.
📖 Additional Detail
Jesse offered to share his augmented code with EN and provide insight on 'delta' SIPs. Here is a sketch by Jesse Stone of his ideas on this topic:
⚖️ Acceptance Criteria
tbd
⚙️ Engineering Details