Closed jordanpadams closed 2 years ago
@nutjob4life can you confirm the latest dev version works with deep archive?
@jordanpadams, sorry, having a little confusion:
https://github.com/NASA-PDS/pds-api-client/releases/latest
which is version 0.7.1 (2021-10-08)https://pypi.org/project/pds.api-client/
says the latest is 0.8.0 (2021-10-14)In any case, the Deep Archive is using 0.8.0 already, so does that mean we're good? π€·ββοΈ
@nutjob4life this is blocked by https://github.com/NASA-PDS/software-issues-repo/issues/30 for the time being. @tloubrieu-jpl mentioned this needed to be upgraded, but I think he still needs to tag a release?
Oh, I completely misinterpreted the "directionality" of "blocked by" π€¦
Clearly I need a second cup of Groundworkβ’ brand coffee!
I'll await a release of pds.api-client (hopefully it has a full changelog so I'll know what I need to fix as a user of it!).
Hi @jordanpadams @nutjob4life , actually I was thinking to use deep-archive as a test for the pds-api-client before I do the release of the pds-api-client. If @nutjob4life can work on it before I do, the pre-release is on test pypi version 1.1.0 I think.
Hi @tloubrieu-jpl! Thanks for being willing to use the Deep Archive as a test.
I tried out the pds.api-client from test.pypi.org
and it failed to run due to a missing module pds.api_client.model.product
.
But as a control, I also tried out the stable version of Deep Archive from pypi.org
and it fails in all three places I tried it!
Here's a summary table:
Deep Archive Version | Gamma | Expo | Production |
---|---|---|---|
main branch with pds.api-client 1.1.0 |
β³ | β³ | β³ |
PyPI 1.1.1 with pds.api-client 0.8.0 | β³ | β³ | β³ |
For testing with the pds.api.client on test.pypi.org
, here are the steps I took:
$ : Make a virtual environment
$ python3.9 -m venv venv
$ : Go into it
$ cd venv
$ : Prepare it
$ bin/pip install --upgrade --quiet setuptools wheel pip build urllib3==1.25.3
$ : Add the latest API client from test.pypi.org
$ bin/pip install --index-url https://test.pypi.org/simple pds.api-client==1.1.0
$ : Clone the Deep Archive repository
$ git clone https://github.com/NASA-PDS/deep-archive.git
$ : Remove the version pin from the API client in its setup.cfg
$ sed -e 's/pds\.api-client == 0.8.0/pds.api-client/g' -i.orig deep-archive/setup.cfg
$ : Install it into the venv
$ bin/pip install --editable deep-archive
$ : Try it on Gamma
$ bin/pds-deep-registry-archive --url https://pds-gamma.jpl.nasa.gov/api --site PDS_ATM urn:nasa:pds:insight_rad::2.1
Traceback (most recent call last):
File "/private/tmp/venv/deep-archive/src/pds2/aipgen/registry.py", line 60, in <module>
from pds.api_client.model.product import Product # type: ignore
ModuleNotFoundError: No module named 'pds.api_client.model.product'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/tmp/venv/bin/pds-deep-registry-archive", line 33, in <module>
sys.exit(load_entry_point('pds.deeparchive', 'console_scripts', 'pds-deep-registry-archive')())
File "/private/tmp/venv/bin/pds-deep-registry-archive", line 25, in importlib_load_entry_point
return next(matches).load()
File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/metadata.py", line 77, in load
module = import_module(match.group('module'))
File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 850, in exec_module
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "/private/tmp/venv/deep-archive/src/pds2/aipgen/registry.py", line 62, in <module>
from pds.api_client.models.product import Product # type: ignore
ModuleNotFoundError: No module named 'pds.api_client.models.product'
$ : Try it on Expo
$ bin/pds-deep-registry-archive --url https://pds-dev.jpl.nasa.gov/expo --site PDS_ATM urn:nasa:pds:insight_rad::2.1
Traceback (most recent call last):
File "/private/tmp/venv/deep-archive/src/pds2/aipgen/registry.py", line 60, in <module>
from pds.api_client.model.product import Product # type: ignore
ModuleNotFoundError: No module named 'pds.api_client.model.product'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/tmp/venv/bin/pds-deep-registry-archive", line 33, in <module>
sys.exit(load_entry_point('pds.deeparchive', 'console_scripts', 'pds-deep-registry-archive')())
File "/private/tmp/venv/bin/pds-deep-registry-archive", line 25, in importlib_load_entry_point
return next(matches).load()
File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/metadata.py", line 77, in load
module = import_module(match.group('module'))
File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 850, in exec_module
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "/private/tmp/venv/deep-archive/src/pds2/aipgen/registry.py", line 62, in <module>
from pds.api_client.models.product import Product # type: ignore
ModuleNotFoundError: No module named 'pds.api_client.models.product'
$ : Try it on Production
$ bin/pds-deep-registry-archive --url https://pds.nasa.gov/api --site PDS_ATM urn:nasa:pds:insight_rad::2.1
Traceback (most recent call last):
File "/private/tmp/venv/deep-archive/src/pds2/aipgen/registry.py", line 60, in <module>
from pds.api_client.model.product import Product # type: ignore
ModuleNotFoundError: No module named 'pds.api_client.model.product'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/tmp/venv/bin/pds-deep-registry-archive", line 33, in <module>
sys.exit(load_entry_point('pds.deeparchive', 'console_scripts', 'pds-deep-registry-archive')())
File "/private/tmp/venv/bin/pds-deep-registry-archive", line 25, in importlib_load_entry_point
return next(matches).load()
File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/metadata.py", line 77, in load
module = import_module(match.group('module'))
File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 850, in exec_module
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "/private/tmp/venv/deep-archive/src/pds2/aipgen/registry.py", line 62, in <module>
from pds.api_client.models.product import Product # type: ignore
ModuleNotFoundError: No module named 'pds.api_client.models.product'
For the stable (PyPI 1.1.1) version of Deep Archive, here are the steps I took to test it:
$ python3.9 -m venv venv
$ cd venv
$ bin/pip install --upgrade --quiet setuptools wheel pip build
$ bin/pip install pds.deeparchive
$ bin/pds-deep-registry-archive --url https://pds.nasa.gov/api --site PDS_ATM urn:nasa:pds:insight_rad::2.1
β¦
pds.api_client.exceptions.ServiceException: (503)
Reason: Service Unavailable
HTTP response headers: HTTPHeaderDict({'Content-Type': 'text/html; charset=iso-8859-1', 'Content-Length': '299', 'Connection': 'keep-alive', 'Date': 'Wed, 27 Apr 2022 16:37:48 GMT', 'Set-Cookie': 'AWSALB=lhk19yV8XyXqSwM0sDMAQvYYQtq5r4/PeS6GWVP4e2xKuF9A1QCMdmQ3rvP2VDTv342ioo6Mg+K+oOiJ73YhPqdLbvzsPxNMhAvHX30nGG8x+JXxNJo8OfgJ/u/3; Expires=Wed, 04 May 2022 16:37:48 GMT; Path=/, AWSALBCORS=lhk19yV8XyXqSwM0sDMAQvYYQtq5r4/PeS6GWVP4e2xKuF9A1QCMdmQ3rvP2VDTv342ioo6Mg+K+oOiJ73YhPqdLbvzsPxNMhAvHX30nGG8x+JXxNJo8OfgJ/u/3; Expires=Wed, 04 May 2022 16:37:48 GMT; Path=/; SameSite=None; Secure', 'Server': 'Apache', 'X-Frame-Options': 'SAMEORIGIN', 'X-XSS-Protection': '1; mode=block', 'Strict-Transport-Security': 'max-age=31536000; preload', 'X-Cache': 'Error from cloudfront', 'Via': '1.1 66651bffa749255769c1c4956851dc96.cloudfront.net (CloudFront)', 'X-Amz-Cf-Pop': 'LAX50-C2', 'X-Amz-Cf-Id': '0wtKafbJNd_ugbq9kTC5hD8uxJWH4CovXPtGEjmzuUttSTxM1w1Mfw=='})
HTTP response body: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>503 Service Unavailable</title>
</head><body>
<h1>Service Unavailable</h1>
<p>The server is temporarily unable to service your
request due to maintenance downtime or capacity
problems. Please try again later.</p>
</body></html>
INFO π Thanks for using this program! Bye!
$ bin/pds-deep-registry-archive --url https://pds-gamma.jpl.nasa.gov/api --site PDS_ATM urn:nasa:pds:insight_rad::2.1
β¦
pds.api_client.exceptions.ServiceException: (503)
Reason: Service Unavailable
HTTP response headers: HTTPHeaderDict({'Date': 'Wed, 27 Apr 2022 16:39:00 GMT', 'Content-Type': 'text/html; charset=iso-8859-1', 'Content-Length': '299', 'Connection': 'keep-alive', 'Server': 'Apache', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Frame-Options': 'SAMEORIGIN'})
HTTP response body: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>503 Service Unavailable</title>
</head><body>
<h1>Service Unavailable</h1>
<p>The server is temporarily unable to service your
request due to maintenance downtime or capacity
problems. Please try again later.</p>
</body></html>
INFO π Thanks for using this program! Bye!
$ bin/pds-deep-registry-archive --url https://pds-dev.jpl.nasa.gov/expo --site PDS_ATM urn:nasa:pds:insight_rad::2.1
INFO π PDS Deep Registry-based Archive, version 1.1.1
INFO π Thanks for using this program! Bye!
Traceback (most recent call last):
File "/private/tmp/venv/bin/pds-deep-registry-archive", line 8, in <module>
sys.exit(main())
File "/private/tmp/venv/lib/python3.9/site-packages/pds2/aipgen/registry.py", line 476, in main
generatedeeparchive(args.url, args.bundle, args.site, not args.include_latest_collection_only)
File "/private/tmp/venv/lib/python3.9/site-packages/pds2/aipgen/registry.py", line 451, in generatedeeparchive
prefixlen, bac, title = _comprehendregistry(url, bundlelidvid, allcollections)
File "/private/tmp/venv/lib/python3.9/site-packages/pds2/aipgen/registry.py", line 319, in _comprehendregistry
_addfiles(bundle, bac)
File "/private/tmp/venv/lib/python3.9/site-packages/pds2/aipgen/registry.py", line 245, in _addfiles
lidvid, props = product['id'], product['properties']
KeyError: 'properties'
I hope you can make some sense of the issue with ModuleNotFoundError: No module named 'pds.api_client.models.product'
π
@al-niessner any idea what is going on here? did we change something in the Swagger API that would remove this module?
@jordanpadams @al-niessner @nutjob4life I am working on that ticket, I have a fix for that in deep-archive. I will create a pull request for that. The change is because the renaming of response objects in the swagger API.
π‘ Description
https://github.com/NASA-PDS/pds-api-client/releases/latest
Blocked by
Tagging new pds-api-client release per https://github.com/NASA-PDS/software-issues-repo/issues/30