Unexpected fatal error when running pds-deep-archive against bundle

jordanpadams commented 2 years ago

🐛 Describe the bug

user encountered fatal error when trying to run pds-deep-archive but not sure what the issue. was able to reproduce with data downloaded on pds-dev3.

pds-deep-archive) [tgueth@atmos ~]$ .virtualenvs/pds-deep-archive/bin/pds-deep-archive -s PDS_ATM -b https://pds-atmospheres.nmsu.edu/PDS/data/PDS4/ /PDS/data/anonymous/PDS/data/PDS4/lab.hydrocarbon_spectra/bundle.lab.hydrocarbon_spectra.xml
INFO 👟 PDS Deep Archive, version 1.0.0
CRITICAL 🛑 Cannot proceed as a critical problem has occurred; re-run with --debug for more info.
INFO 👋 That's it for now. Bye.

I thought initially that it was something with the program (updates in python or such) so I ran several other data volumes to see if I get the same error message, but nope. Those all created the 5 output files as usual, regardless which one I ran (I tried 7 different ones just to be thorough).
Therefore, I ran the same command as before but used --debug as mentioned in the return. I get the following, but it doesn't make much sense to me. Is it a permission problem that Lyle has to fix?

(pds-deep-archive) [tgueth@atmos ~]$ .virtualenvs/pds-deep-archive/bin/pds-deep-archive -s PDS_ATM -b https://pds-atmospheres.nmsu.edu/PDS/data/PDS4/ /PDS/data/anonymous/PDS/data/PDS4/lab.hydrocarbon_spectra/bundle.lab.hydrocarbon_spectra.xml --debug
INFO 👟 PDS Deep Archive, version 1.0.0
DEBUG ⚙️ command line args = Namespace(bundle=<_io.BufferedReader name='/PDS/data/anonymous/PDS/data/PDS4/lab.hydrocarbon_spectra/bundle.lab.hydrocarbon_spectra.xml'>, bundle_base_url='https://pds-atmospheres.nmsu.edu/PDS/data/PDS4/', disable_url_validation=False, include_latest_collection_only=False, loglevel=10, site='PDS_ATM')
DEBUG ⚙️ Creating potentially future-mulitprocessing–capable DB in /tmp/deeptn2nxugt.dir/pds-deep-archive.sqlite3
DEBUG 📄 Deconstructing /PDS/data/anonymous/PDS/data/PDS4/lab.hydrocarbon_spectra/bundle.lab.hydrocarbon_spectra.xml
DEBUG 📄 Deconstructing /PDS/data/anonymous/PDS/data/PDS4/lab.hydrocarbon_spectra/context/collection_lab.hydrocarbon_spectra_context.xml
DEBUG 📄 Deconstructing /PDS/data/anonymous/PDS/data/PDS4/lab.hydrocarbon_spectra/document/collection_lab.hydrocarbon_spectra_document.xml
DEBUG 📄 Deconstructing /PDS/data/anonymous/PDS/data/PDS4/lab.hydrocarbon_spectra/document/isobutane/n2h2202k295k.xml
DEBUG 📄 Deconstructing /PDS/data/anonymous/PDS/data/PDS4/lab.hydrocarbon_spectra/document/isobutane/n2h2210k295k.xml
DEBUG 📄 Deconstructing /PDS/data/anonymous/PDS/data/PDS4/lab.hydrocarbon_spectra/document/propane/h2he200k298k.xml
DEBUG 📄 Deconstructing /PDS/data/anonymous/PDS/data/PDS4/lab.hydrocarbon_spectra/document/propane/h2he150k210k.xml
DEBUG 📄 Deconstructing /PDS/data/anonymous/PDS/data/PDS4/lab.hydrocarbon_spectra/document/propane/h2205k292k.xml
DEBUG 📄 Deconstructing /PDS/data/anonymous/PDS/data/PDS4/lab.hydrocarbon_spectra/document/propane/he202k292k.xml
DEBUG 📄 Deconstructing /PDS/data/anonymous/PDS/data/PDS4/lab.hydrocarbon_spectra/document/propene/n2he202k295k.xml
DEBUG 📄 Deconstructing /PDS/data/anonymous/PDS/data/PDS4/lab.hydrocarbon_spectra/document/neopentane/n2202k297k.xml
DEBUG 📄 Deconstructing /PDS/data/anonymous/PDS/data/PDS4/lab.hydrocarbon_spectra/document/ethane/h2n2195k295k.xml
DEBUG 📄 Deconstructing /PDS/data/anonymous/PDS/data/PDS4/lab.hydrocarbon_spectra/document/ethane/h2he202k295k.xml
DEBUG 📄 Deconstructing /PDS/data/anonymous/PDS/data/PDS4/lab.hydrocarbon_spectra/xml_schema/collection_lab.hydrocarbon_spectra_xml_schema.xml
DEBUG 📄 Deconstructing /PDS/data/anonymous/PDS/data/PDS4/lab.hydrocarbon_spectra/data/collection_lab.hydrocarbon_spectra_data.xml
CRITICAL 🛑 Cannot proceed as a critical problem has occurred; re-run with --debug for more info.
DEBUG 🖥 Here is the exception: IntegrityError('columns lid, vid, to_lid, to_vid are not unique',)
Traceback (most recent call last):
  File "/home/tgueth/.virtualenvs/pds-deep-archive/lib64/python3.6/site-packages/pds/aipgen/main.py", line 108, in main
    comprehendDirectory(os.path.dirname(os.path.abspath(args.bundle.name)), con)
  File "/home/tgueth/.virtualenvs/pds-deep-archive/lib64/python3.6/site-packages/pds/aipgen/utils.py", line 181, in comprehendDirectory
    _addInterLabelReferencesFromTabFile(lid, vid, filepath, con)
  File "/home/tgueth/.virtualenvs/pds-deep-archive/lib64/python3.6/site-packages/pds/aipgen/utils.py", line 114, in _addInterLabelReferencesFromTabFile
    (lid, vid, match.group(1), match.group(2))
sqlite3.IntegrityError: columns lid, vid, to_lid, to_vid are not unique
INFO 👋 That's it for now. Bye.

📜 To Reproduce

Steps to reproduce the behavior:

pds-deep-archive -s PDS_ATM -b https://pds-atmospheres.nmsu.edu/PDS/data/PDS4/ \
    /PDS/data/anonymous/PDS/data/PDS4/lab.hydrocarbon_spectra/bundle.lab.hydrocarbon_spectra.xml

🕵️ Expected behavior

Executes successfully.

📚 Version of Software Used

1.0.0

🩺 Test Data / Additional context

pds-dev3:$TEST_DATA_HOME/registry/lab.hydrocarbon_spectra

🦄 Related requirements

⚙️ Engineering Details

jordanpadams commented 2 years ago

@nutjob4life after PLAID work, this is next

jordanpadams commented 2 years ago

@nutjob4life now that PLAID is mostly done... ☝️

nutjob4life commented 2 years ago

@jordanpadams 😁

nutjob4life commented 2 years ago

Hi @jordanpadams … I'm logged into pds-dev3. Where can I find this file /PDS/data/anonymous/PDS/data/PDS4/lab.hydrocarbon_spectra/bundle.lab.hydrocarbon_spectra.xml?

nutjob4life commented 2 years ago

Note to self: see encrypted notes (1Password) for location of $TEST_DATA_HOME

jordanpadams commented 2 years ago

@nutjob4life also put this on our internal wiki: https://wiki.jpl.nasa.gov/display/PDSEN/Test+Data

NASA-PDS / deep-archive