search

NASA-PDS / deep-archive

PDS Open Archival Information System (OAIS) utilities, including Submission Information Package (SIP) and Archive Information Package (AIP) generators
https://nasa-pds.github.io/deep-archive/
Other
7 stars 4 forks source link

Upgrade to latest PDS API Client #128

Closed jordanpadams closed 2 years ago

jordanpadams commented 2 years ago

💡 Description

https://github.com/NASA-PDS/pds-api-client/releases/latest

Blocked by

Tagging new pds-api-client release per https://github.com/NASA-PDS/software-issues-repo/issues/30

jordanpadams commented 2 years ago

@nutjob4life can you confirm the latest dev version works with deep archive?

nutjob4life commented 2 years ago

@jordanpadams, sorry, having a little confusion:

In any case, the Deep Archive is using 0.8.0 already, so does that mean we're good? 🤷‍♀️

jordanpadams commented 2 years ago

@nutjob4life this is blocked by https://github.com/NASA-PDS/software-issues-repo/issues/30 for the time being. @tloubrieu-jpl mentioned this needed to be upgraded, but I think he still needs to tag a release?

nutjob4life commented 2 years ago

Oh, I completely misinterpreted the "directionality" of "blocked by" 🤦

Clearly I need a second cup of Groundwork™ brand coffee!

I'll await a release of pds.api-client (hopefully it has a full changelog so I'll know what I need to fix as a user of it!).

tloubrieu-jpl commented 2 years ago

Hi @jordanpadams @nutjob4life , actually I was thinking to use deep-archive as a test for the pds-api-client before I do the release of the pds-api-client. If @nutjob4life can work on it before I do, the pre-release is on test pypi version 1.1.0 I think.

nutjob4life commented 2 years ago

Hi @tloubrieu-jpl! Thanks for being willing to use the Deep Archive as a test.

I tried out the pds.api-client from test.pypi.org and it failed to run due to a missing module pds.api_client.model.product.

But as a control, I also tried out the stable version of Deep Archive from pypi.org and it fails in all three places I tried it!

Here's a summary table:

Deep Archive Version Gamma Expo Production
main branch with pds.api-client 1.1.0 ╳ ╳ ╳
PyPI 1.1.1 with pds.api-client 0.8.0 ╳ ╳ ╳

For testing with the pds.api.client on test.pypi.org, here are the steps I took:

$ : Make a virtual environment
$ python3.9 -m venv venv
$ : Go into it
$ cd venv
$ : Prepare it
$ bin/pip install --upgrade --quiet setuptools wheel pip build urllib3==1.25.3
$ : Add the latest API client from test.pypi.org
$ bin/pip install --index-url https://test.pypi.org/simple pds.api-client==1.1.0
$ : Clone the Deep Archive repository
$ git clone https://github.com/NASA-PDS/deep-archive.git
$ : Remove the version pin from the API client in its setup.cfg
$ sed -e 's/pds\.api-client == 0.8.0/pds.api-client/g' -i.orig deep-archive/setup.cfg
$ : Install it into the venv
$ bin/pip install --editable deep-archive
$ : Try it on Gamma
$ bin/pds-deep-registry-archive --url https://pds-gamma.jpl.nasa.gov/api --site PDS_ATM urn:nasa:pds:insight_rad::2.1
Traceback (most recent call last):
  File "/private/tmp/venv/deep-archive/src/pds2/aipgen/registry.py", line 60, in <module>
    from pds.api_client.model.product import Product  # type: ignore
ModuleNotFoundError: No module named 'pds.api_client.model.product'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/private/tmp/venv/bin/pds-deep-registry-archive", line 33, in <module>
    sys.exit(load_entry_point('pds.deeparchive', 'console_scripts', 'pds-deep-registry-archive')())
  File "/private/tmp/venv/bin/pds-deep-registry-archive", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/metadata.py", line 77, in load
    module = import_module(match.group('module'))
  File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/private/tmp/venv/deep-archive/src/pds2/aipgen/registry.py", line 62, in <module>
    from pds.api_client.models.product import Product  # type: ignore
ModuleNotFoundError: No module named 'pds.api_client.models.product'
$ : Try it on Expo
$ bin/pds-deep-registry-archive --url https://pds-dev.jpl.nasa.gov/expo --site PDS_ATM urn:nasa:pds:insight_rad::2.1
Traceback (most recent call last):
  File "/private/tmp/venv/deep-archive/src/pds2/aipgen/registry.py", line 60, in <module>
    from pds.api_client.model.product import Product  # type: ignore
ModuleNotFoundError: No module named 'pds.api_client.model.product'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/private/tmp/venv/bin/pds-deep-registry-archive", line 33, in <module>
    sys.exit(load_entry_point('pds.deeparchive', 'console_scripts', 'pds-deep-registry-archive')())
  File "/private/tmp/venv/bin/pds-deep-registry-archive", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/metadata.py", line 77, in load
    module = import_module(match.group('module'))
  File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/private/tmp/venv/deep-archive/src/pds2/aipgen/registry.py", line 62, in <module>
    from pds.api_client.models.product import Product  # type: ignore
ModuleNotFoundError: No module named 'pds.api_client.models.product'
$ : Try it on Production
$ bin/pds-deep-registry-archive --url https://pds.nasa.gov/api --site PDS_ATM urn:nasa:pds:insight_rad::2.1
Traceback (most recent call last):
  File "/private/tmp/venv/deep-archive/src/pds2/aipgen/registry.py", line 60, in <module>
    from pds.api_client.model.product import Product  # type: ignore
ModuleNotFoundError: No module named 'pds.api_client.model.product'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/private/tmp/venv/bin/pds-deep-registry-archive", line 33, in <module>
    sys.exit(load_entry_point('pds.deeparchive', 'console_scripts', 'pds-deep-registry-archive')())
  File "/private/tmp/venv/bin/pds-deep-registry-archive", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/metadata.py", line 77, in load
    module = import_module(match.group('module'))
  File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/private/tmp/venv/deep-archive/src/pds2/aipgen/registry.py", line 62, in <module>
    from pds.api_client.models.product import Product  # type: ignore
ModuleNotFoundError: No module named 'pds.api_client.models.product'

For the stable (PyPI 1.1.1) version of Deep Archive, here are the steps I took to test it:

$ python3.9 -m venv venv
$ cd venv
$ bin/pip install --upgrade --quiet setuptools wheel pip build
$ bin/pip install pds.deeparchive
$ bin/pds-deep-registry-archive --url https://pds.nasa.gov/api --site PDS_ATM urn:nasa:pds:insight_rad::2.1
…
pds.api_client.exceptions.ServiceException: (503)
Reason: Service Unavailable
HTTP response headers: HTTPHeaderDict({'Content-Type': 'text/html; charset=iso-8859-1', 'Content-Length': '299', 'Connection': 'keep-alive', 'Date': 'Wed, 27 Apr 2022 16:37:48 GMT', 'Set-Cookie': 'AWSALB=lhk19yV8XyXqSwM0sDMAQvYYQtq5r4/PeS6GWVP4e2xKuF9A1QCMdmQ3rvP2VDTv342ioo6Mg+K+oOiJ73YhPqdLbvzsPxNMhAvHX30nGG8x+JXxNJo8OfgJ/u/3; Expires=Wed, 04 May 2022 16:37:48 GMT; Path=/, AWSALBCORS=lhk19yV8XyXqSwM0sDMAQvYYQtq5r4/PeS6GWVP4e2xKuF9A1QCMdmQ3rvP2VDTv342ioo6Mg+K+oOiJ73YhPqdLbvzsPxNMhAvHX30nGG8x+JXxNJo8OfgJ/u/3; Expires=Wed, 04 May 2022 16:37:48 GMT; Path=/; SameSite=None; Secure', 'Server': 'Apache', 'X-Frame-Options': 'SAMEORIGIN', 'X-XSS-Protection': '1; mode=block', 'Strict-Transport-Security': 'max-age=31536000; preload', 'X-Cache': 'Error from cloudfront', 'Via': '1.1 66651bffa749255769c1c4956851dc96.cloudfront.net (CloudFront)', 'X-Amz-Cf-Pop': 'LAX50-C2', 'X-Amz-Cf-Id': '0wtKafbJNd_ugbq9kTC5hD8uxJWH4CovXPtGEjmzuUttSTxM1w1Mfw=='})
HTTP response body: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>503 Service Unavailable</title>
</head><body>
<h1>Service Unavailable</h1>
<p>The server is temporarily unable to service your
request due to maintenance downtime or capacity
problems. Please try again later.</p>
</body></html>

INFO 👋 Thanks for using this program! Bye!
$ bin/pds-deep-registry-archive --url https://pds-gamma.jpl.nasa.gov/api --site PDS_ATM urn:nasa:pds:insight_rad::2.1
…
pds.api_client.exceptions.ServiceException: (503)
Reason: Service Unavailable
HTTP response headers: HTTPHeaderDict({'Date': 'Wed, 27 Apr 2022 16:39:00 GMT', 'Content-Type': 'text/html; charset=iso-8859-1', 'Content-Length': '299', 'Connection': 'keep-alive', 'Server': 'Apache', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Frame-Options': 'SAMEORIGIN'})
HTTP response body: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>503 Service Unavailable</title>
</head><body>
<h1>Service Unavailable</h1>
<p>The server is temporarily unable to service your
request due to maintenance downtime or capacity
problems. Please try again later.</p>
</body></html>

INFO 👋 Thanks for using this program! Bye!
$ bin/pds-deep-registry-archive --url https://pds-dev.jpl.nasa.gov/expo --site PDS_ATM urn:nasa:pds:insight_rad::2.1
INFO 👟 PDS Deep Registry-based Archive, version 1.1.1
INFO 👋 Thanks for using this program! Bye!
Traceback (most recent call last):
  File "/private/tmp/venv/bin/pds-deep-registry-archive", line 8, in <module>
    sys.exit(main())
  File "/private/tmp/venv/lib/python3.9/site-packages/pds2/aipgen/registry.py", line 476, in main
    generatedeeparchive(args.url, args.bundle, args.site, not args.include_latest_collection_only)
  File "/private/tmp/venv/lib/python3.9/site-packages/pds2/aipgen/registry.py", line 451, in generatedeeparchive
    prefixlen, bac, title = _comprehendregistry(url, bundlelidvid, allcollections)
  File "/private/tmp/venv/lib/python3.9/site-packages/pds2/aipgen/registry.py", line 319, in _comprehendregistry
    _addfiles(bundle, bac)
  File "/private/tmp/venv/lib/python3.9/site-packages/pds2/aipgen/registry.py", line 245, in _addfiles
    lidvid, props = product['id'], product['properties']
KeyError: 'properties'

I hope you can make some sense of the issue with ModuleNotFoundError: No module named 'pds.api_client.models.product' 🙏

jordanpadams commented 2 years ago

@al-niessner any idea what is going on here? did we change something in the Swagger API that would remove this module?

tloubrieu-jpl commented 2 years ago

@jordanpadams @al-niessner @nutjob4life I am working on that ticket, I have a fix for that in deep-archive. I will create a pull request for that. The change is because the renaming of response objects in the swagger API.