search

NASA-PDS / deep-archive

PDS Open Archival Information System (OAIS) utilities, including Submission Information Package (SIP) and Archive Information Package (AIP) generators
https://nasa-pds.github.io/deep-archive/
Other
7 stars 4 forks source link

As a user, I want deep archive to support LBLX label extensions #157

Open tbarnes4 opened 6 months ago

tbarnes4 commented 6 months ago

Checked for duplicates

No - I haven't checked

🐛 Describe the bug

When I run the software on a bundle that has only lblx files, the software runs without errors, but produces empty SIP, checksum, and manifest *.tab files.

🕵️ Expected behavior

I expect the software to create the tables as if the bundle contained xml labels.

📜 To Reproduce

  1. Run pds-deep-archive
  2. Ex: pds-deep-archive -s PDS_SBN -b https://pdssbn.astro.umd.edu/holdings/ --include-latest-collection-only pds4-nh_documents-v1.0/bundle.lblx

  3. ...

🖥 Environment Info

📚 Version of Software Used

pds-deep-archive 1.1.4

🩺 Test Data / Additional context

https://pdssbn.astro.umd.edu/holdings/pds4-nh_documents-v1.0

🦄 Related requirements

🦄 #xyz

⚙️ Engineering Details

No response

tbarnes4 commented 6 months ago

@jordanpadams Since I've seen no further comment on this, just pinging again.

jordanpadams commented 6 months ago

@tbarnes4 thanks for the ping. I updated this ticket to be a new requirement since this is an overall new requirement on the system that was not implemented yet.

That being said, we added this to the backlog, but it is not on the top of our radar since there is a workaround.

If you load this data into the Registry/API (which it looks like it is), then you can run pds-deep-registry-archive to generate the SIPs much easier (and faster), not to mention it is a good forcing factor to load the data into the registry.

$ pds-deep-registry-archive --site PDS_SBN urn:nasa:pds:nh_documents::1.0
tbarnes4 commented 6 months ago

@jordanpadams Thanks! I was thinking of trying that work around as well. It would be the first we've done from SBN-UMD thru the registry, so that will be fun.

jordanpadams commented 6 months ago

@tbarnes4 actually... standby on running deep-archive. I forgot we have a bug right now. A PR is up and we will hopefully have a new version tagged by end of day (or first thing Monday).

jordanpadams commented 6 months ago

@tbarnes4 you should be able to upgrade your PDS Deep Archive software and run the utility as described above. I just tested it and it appears to work great.

tbarnes4 commented 6 months ago

@jordanpadams Upgraded to v1.1.15 on two different systems (same as above) and I am still getting the same result with the same bundle (as above) where the 3 *tab files are all empty.

% source ~/.virtualenvs/pds-deep-archive/bin/activate.csh % pds-deep-archive -s PDS_SBN -b https://pdssbn.astro.umd.edu/holdings/ pds4-nh_documents-v1.0/bundle.lblx INFO 👟 PDS Deep Archive, version 1.1.5 INFO 🏃‍♀️ Starting AIP generation for pds4-nh_documents-v1.0/bundle.lblx INFO 🎉 Success! AIP done, files generated: INFO 📄 Checksum manifest: nh_documents_v1.0_20240227_checksum_manifest_v1.0.tab INFO 📄 Transfer manifest: nh_documents_v1.0_20240227_transfer_manifest_v1.0.tab INFO 📄 XML label for them both: nh_documents_v1.0_20240227_aip_v1.0.xml INFO 🏃‍♀️ Starting SIP generation for pds4-nh_documents-v1.0/bundle.lblx INFO 🎉 Success! From /Volumes/echeclus02/TEMP/nssdc_test/pds4-nh_documents-v1.0/bundle.lblx, generated these output files: INFO 📄 SIP Manifest: nh_documents_v1.0_20240227_sip_v1.0.tab INFO 📄 XML label for the SIP: nh_documents_v1.0_20240227_sip_v1.0.xml INFO 👋 That's it for now. Bye.

jordanpadams commented 6 months ago

@tbarnes4 sorry for the confusion, I meant you are good to use the pds-deep-registry-archive utility to generate the SIPs using the next-gen Registry. This requirement has still not been implemented, and will unfortunately not be implemented for a bit since we have a known workaround.

tbarnes4 commented 6 months ago

@jordanpadams That makes much more sense. Tracking down other issues and I felt I was going crazy. I expect to try the registry-archive later.

jordanpadams commented 6 months ago

@tbarnes4 copy that. it should take ~30 seconds to try it :-) . and you can run it from anywhere (your own laptop works).

tbarnes4 commented 6 months ago

@jordanpadams That it does, even less. For kicks and giggles tried a collection LIDVID and it errored out with unexpected errors. But can't expect it to do things it wasn't designed to do. ;-)

smclaughlin7 commented 6 months ago

Subscribed to monitor progress. @tbarnes4 Do you expect to submit this bundle shortly after deep archive is enhanced?

smclaughlin7 commented 5 months ago

@jordanpadams Should NSSDCA expect Deep Archive tool continue to use '.xml' as the extension for SIP products, at least until IM 2.0? I assume '.lblx' is not presently an option for that tool, correct? Thanks!

jordanpadams commented 5 months ago

@smclaughlin7 correct. And I am not even sure we will be upgrading with IM 2.0 since all of these SIPs belong to the same collection. We don't plan on going back and reprocessing those SIPs to use lblx.

smclaughlin7 commented 5 months ago

@jordanpadams Thanks! We/NSSDCA will continue to expect only '.xml' for SIPs.

tbarnes4 commented 4 months ago

Subscribed to monitor progress. @tbarnes4 Do you expect to submit this bundle shortly after deep archive is enhanced?

@smclaughlin7 Sorry for the delay. I have just submitted the two bundles. Hopefully they go thru without problem. Thanks!