NASA-PDS / deep-archive

PDS Open Archival Information System (OAIS) utilities, including Submission Information Package (SIP) and Archive Information Package (AIP) generators
https://nasa-pds.github.io/deep-archive/
Other
7 stars 4 forks source link

Create new pds-deep-archive program and improve performance #26

Closed nutjob4life closed 4 years ago

nutjob4life commented 4 years ago

With these changes, running sipgen on my Mac¹ can process a 272GiB insight_cameras export in 1:03. On pdsimg-int1, it handles the 1.5TiBinsight_cameras dataset in under 4 hours.

Footnotes:

nutjob4life commented 4 years ago

Again, you can ignore the Packaging step of the check failing; it's because it ran earlier today, the file is date-stamped, and the test PyPI doesn't allow duplicate files to be uploaded.

jordanpadams commented 4 years ago

@nutjob4life could we change this to datetime? or too much of a pain?