KeyError: 'feed' on .Query call

[joerobinson20]

Running into an issue when querying ICESat-2 data using icepyx (version 1.3.0). I have tried different ATL products and different queries such as date ranges, bounding boxes, etc.

Running any of the examples for acessing/subsetting from the webpage returns the attached image. I believe this is different than previous issues (like #166)? The issue here is in is2ref.py and not granules.py. Thanks for the help!

[mfisher87]

Can you please share a minimal & fully reproducible example of how you got this error? Thanks!

[joerobinson20]

Sorry! My attached image occurred after running just the first example given under the "Accessing ICESat-2 Data" header on the webpage (below).

import icepyx as ipx #v1.3.0

short_name = 'ATL06'
spatial_extent = [-55, 68, -48, 71]
date_range = ['2019-02-20','2019-02-28']

region_a = ipx.Query(short_name, spatial_extent, date_range);

I haven't downloaded ICESat-2 data with icepyx in a month or so, so it's possible something on my end changed. I downgraded icepyx to version 0.8 (which I had used last time to download data) and was also receiving a similar 'feed' error.

[mfisher87]

Thank you for providing that detail! I attempted your example as originally written, and couldn't reproduce. I made some slight edits to make your example more minimal and still couldn't reproduce! I could use more help understanding what's going on. Can you try this in a fresh Python interpreter like so and see if it reproduces? Can you also please try in a freshly-created environment?

$ python
Python 3.12.7 | packaged by conda-forge | (main, Oct  4 2024, 16:05:46) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import icepyx as ipx
<stdin>:1: FutureWarning: icepyx v1.x is being deprecated; the back-end systems on which it relies
will be shut down as of late 2024. At that time, upgrade to icepyx v2.x, which uses the
new NASA Harmony back-end, will be required. Please see
<https://icepyx.readthedocs.io/en/latest/user_guide/changelog/v1.3.0.html> for more
information!

>>> ipx.__version__
'1.3.0'
>>> qry = ipx.Query(product='ATL06', spatial_extent=[-55, 68, -48, 71], date_range=['2019-02-20','2019-02-
>>> qry.product_summary_info()
title :  ATLAS/ICESat-2 L3A Land Ice Height V006
short_name :  ATL06
version_id :  006
time_start :  2018-10-14T00:00:00.000Z
coordinate_system :  CARTESIAN
summary :  This data set (ATL06) provides geolocated, land-ice surface heights (above the WGS 84 ellipsoid, ITRF2014 reference frame), plus ancillary parameters that can be used to interpret and assess the quality of the height estimates. The data were acquired by the Advanced Topographic Laser Altimeter System (ATLAS) instrument on board the Ice, Cloud and land Elevation Satellite-2 (ICESat-2) observatory.
orbit_parameters :  {}

[joerobinson20]

Thanks for the continued help. That still produces the 'feed' error. My remote machine is on python 3.9.2 and GCC 10.2.1

6 eclair% python
Python 3.9.2 (default, Feb 28 2021, 17:03:44) 
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import icepyx as ipx
<stdin>:1: FutureWarning: icepyx v1.x is being deprecated; the back-end systems on which it relies
will be shut down as of late 2024. At that time, upgrade to icepyx v2.x, which uses the
new NASA Harmony back-end, will be required. Please see
<https://icepyx.readthedocs.io/en/latest/user_guide/changelog/v1.3.0.html> for more
information!

>>> ipx.__version__
'1.3.0'
>>> qry = ipx.Query(product='ATL06', spatial_extent=[-55, 68, -48, 71], date_range=['2019-02-20','2019-02-28'])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/disk/p/jrobin15/.local/lib/python3.9/site-packages/icepyx/core/query.py", line 425, in __init__
    self._version = val.prod_version(is2ref.latest_version(self._prod), version)
  File "/home/disk/p/jrobin15/.local/lib/python3.9/site-packages/icepyx/core/is2ref.py", line 350, in latest_version
    return max([entry["version_id"] for entry in _about_product["feed"]["entry"]])
KeyError: 'feed'
>>> 

[mfisher87]

Thanks for following up so quickly! I'm still unable to reproduce with Python 3.9.2:

$ python
Python 3.9.2 | packaged by conda-forge | (default, Feb 21 2021, 05:02:46)                                                                                                                      
[GCC 9.3.0] on linux                                                                                                                                                                           
Type "help", "copyright", "credits" or "license" for more information.                                                                                                                         
>>> import icepyx as ipx                                                                                                                                                                       
<stdin>:1: FutureWarning: icepyx v1.x is being deprecated; the back-end systems on which it relies                                                                                             
will be shut down as of late 2024. At that time, upgrade to icepyx v2.x, which uses the                                                                                                        
new NASA Harmony back-end, will be required. Please see                                                                                                                                        
<https://icepyx.readthedocs.io/en/latest/user_guide/changelog/v1.3.0.html> for more                                                                                                            
information!                                                                                                                                                                                   

/home/robatt/.local/share/miniforge3/envs/icepyx-1.3/lib/python3.9/site-packages/dask/dataframe/_pyarrow_compat.py:21: UserWarning: You are using pyarrow version 10.0.1 which is known to be insecure. See https://www.cve.org/CVERecord?id=CVE-2023-47248 for further details. Please upgrade to pyarrow>=14.0.1 or install pyarrow-hotfix to patch your current version.                   
  warnings.warn(                                                                                                                                                                               
>>> ipx.__version__                                                                                                                                                                            
'1.3.0'                                                                                                                                                                                        
>>> qry = ipx.Query(product='ATL06', spatial_extent=[-55, 68, -48, 71], date_range=['2019-02-20','2019-02-28'])                                                                                
>>> qry.product_summary_info()
title :  ATLAS/ICESat-2 L3A Land Ice Height V006                                                                                                                                               
short_name :  ATL06                                                                                                                                                                            
version_id :  006                                                                                                                                                                              
time_start :  2018-10-14T00:00:00.000Z                                                                                                                                                         
coordinate_system :  CARTESIAN                                                                                                                                                                 
summary :  This data set (ATL06) provides geolocated, land-ice surface heights (above the WGS 84 ellipsoid, ITRF2014 reference frame), plus ancillary parameters that can be used to interpret and assess the quality of the height estimates. The data were acquired by the Advanced Topographic Laser Altimeter System (ATLAS) instrument on board the Ice, Cloud and land Elevation Satellite-2 (ICESat-2) observatory.                                                                                                                                                                   
orbit_parameters :  {}                                                                                                                                                                         

Are you using conda or some other tool for managing your environment? I'd like to try installing icepyx into a fresh environment.

[joerobinson20]

I activated a virtual environment and freshly installed icepyx (1.3.0) and all of its dependencies using pip. Running the command qry = ipx.Query(...) still produces a KeyError: 'feed' error.

Successfully installed Jinja2-3.1.4 MarkupSafe-3.0.2 aiobotocore-2.15.2 aiohappyeyeballs-2.4.3 aiohttp-3.10.10 aioitertools-0.12.0 aiosignal-1.3.1 async-timeout-4.0.3 attrs-24.2.0 backoff-2.2.1 bleach-6.1.0 bokeh-3.4.3 botocore-1.35.36 bounded-pool-executor-0.0.3 certifi-2024.8.30 charset-normalizer-3.4.0 click-8.1.7 click-plugins-1.1.1 cligj-0.7.2 cloudpickle-3.1.0 colorcet-3.1.0 contourpy-1.3.0 cycler-0.12.1 dask-2024.8.0 dask-expr-1.1.10 datashader-0.16.3 earthaccess-0.11.0 fiona-1.10.1 fonttools-4.54.1 frozenlist-1.4.1 fsspec-2024.9.0 geopandas-1.0.1 h5netcdf-1.4.0 h5py-3.12.1 holoviews-1.19.1 hvplot-0.11.1 icepyx-1.3.0 idna-3.10 importlib-metadata-8.5.0 importlib-resources-6.4.5 jmespath-1.0.1 kiwisolver-1.4.7 linkify-it-py-2.0.3 llvmlite-0.43.0 locket-1.0.0 markdown-3.7 markdown-it-py-3.0.0 matplotlib-3.9.2 mdit-py-plugins-0.4.2 mdurl-0.1.2 multidict-6.1.0 multimethod-1.12 multipledispatch-1.0.0 numba-0.60.0 numpy-2.0.2 packaging-24.1 pandas-2.2.3 panel-1.4.5 param-2.1.1 partd-1.4.2 pillow-11.0.0 pqdm-0.2.0 propcache-0.2.0 pyarrow-17.0.0 pyct-0.5.0 pyogrio-0.10.0 pyparsing-3.2.0 pyproj-3.6.1 python-cmr-0.13.0 python-dateutil-2.9.0.post0 pytz-2024.2 pyviz-comms-3.0.3 pyyaml-6.0.2 requests-2.32.3 s3fs-2024.9.0 scipy-1.13.1 shapely-2.0.6 six-1.16.0 tinynetrc-1.3.1 toolz-1.0.0 tornado-6.4.1 tqdm-4.66.5 typing-extensions-4.12.2 tzdata-2024.2 uc-micro-py-1.0.3 urllib3-1.26.20 webencodings-0.5.1 wrapt-1.16.0 xarray-2024.7.0 xyzservices-2024.9.0 yarl-1.15.5 zipp-3.20.2

(myenv) 40 eclair% python
Python 3.9.2 (default, Feb 28 2021, 17:03:44) 
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import icepyx as ipx
<stdin>:1: FutureWarning: icepyx v1.x is being deprecated; the back-end systems on which it relies
will be shut down as of late 2024. At that time, upgrade to icepyx v2.x, which uses the
new NASA Harmony back-end, will be required. Please see
<https://icepyx.readthedocs.io/en/latest/user_guide/changelog/v1.3.0.html> for more
information!

>>> ipx.__version__
'1.3.0'
>>> qry = ipx.Query(product='ATL06', spatial_extent=[-55, 68, -48, 71], date_range=['2019-02-20','2019-02-28'])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/disk/cupcake/jrobin15/atlas_blowing_snow/myenv/lib/python3.9/site-packages/icepyx/core/query.py", line 425, in __init__
    self._version = val.prod_version(is2ref.latest_version(self._prod), version)
  File "/home/disk/cupcake/jrobin15/atlas_blowing_snow/myenv/lib/python3.9/site-packages/icepyx/core/is2ref.py", line 350, in latest_version
    return max([entry["version_id"] for entry in _about_product["feed"]["entry"]])
KeyError: 'feed'

[mfisher87]

I activated a virtual environment

Was this a new empty environment? What did you use to create it and to install icepyx into it? :thinking:

[joerobinson20]

Sorry for the confusion. I used virtualenv to activate the new environment (which should be empty by default?) and then pip to install icepyx and its dependencies.

Is there something else I should try? I've not used conda to manage environments but could give it a go.

[mfisher87]

Sorry for the confusion. I used virtualenv to activate the new environment (which should be empty by default?) and then pip to install icepyx and its dependencies.

Great, thank you! I tried this as well, and had no issues:

$ virtualenv .venv
created virtual environment CPython3.10.14.final.0-64 in 216ms
  creator CPython3Posix(dest=/tmp/foo/.venv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/robatt/.local/share/virtualenv)
    added seed packages: pip==24.2, setuptools==73.0.1, wheel==0.44.0
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

$ . .venv/bin/activate

$ pip install icepyx
<... snip ...>

$ python
Python 3.10.14 | packaged by conda-forge | (main, Mar 20 2024, 12:45:18) [GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import icepyx as ipx
<stdin>:1: FutureWarning: icepyx v1.x is being deprecated; the back-end systems on which it relies
will be shut down as of late 2024. At that time, upgrade to icepyx v2.x, which uses the
new NASA Harmony back-end, will be required. Please see
<https://icepyx.readthedocs.io/en/latest/user_guide/changelog/v1.3.0.html> for more
information!

>>> ipx.__version__
'1.3.0'
>>> qry = ipx.Query(product='ATL06', spatial_extent=[-55, 68, -48, 71], date_range=['2019-02-20','2019-02-28'])
>>> qry
<icepyx.core.query.Query object at 0x78e28c42ffd0>

Can you share how you installed Python itself?

Is there something else I should try? I've not used conda to manage environments but could give it a go.

You could try with conda, I'd be interested to see if that helps!

[joerobinson20]

Hey there, I got around to trying this using a fresh python install on my local Mac machine (not my department's computing cluster where I'm not exactly sure how python was installed/is managed). I had no issues running the examples we were trying. So it looks like something is going wrong on the cluster? I'll touch base with our computing folks to check if anything changed recently that could have impacted how icepyx works and update that here. Are there any hypotheses I could send their way about why the 'feed' error might be happening?

I can use this as a work around for a little bit as I figure out the issue on the remote machine.

Python 3.9.6 (default, Aug  9 2024, 14:24:13) 
[Clang 16.0.0 (clang-1600.0.26.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import icepyx as ipx #v1.3.0
<stdin>:1: FutureWarning: icepyx v1.x is being deprecated; the back-end systems on which it relies
will be shut down as of late 2024. At that time, upgrade to icepyx v2.x, which uses the
new NASA Harmony back-end, will be required. Please see
<https://icepyx.readthedocs.io/en/latest/user_guide/changelog/v1.3.0.html> for more
information!

>>> ipx.__version__
'1.3.0'
>>> qry = ipx.Query(product='ATL06', spatial_extent=[-55, 68, -48, 71], date_range=['2019-02-20','2019-02-28'])
>>> qry.product_summary_info()
title :  ATLAS/ICESat-2 L3A Land Ice Height V006
short_name :  ATL06
version_id :  006
time_start :  2018-10-14T00:00:00.000Z
coordinate_system :  CARTESIAN
summary :  This data set (ATL06) provides geolocated, land-ice surface heights (above the WGS 84 ellipsoid, ITRF2014 reference frame), plus ancillary parameters that can be used to interpret and assess the quality of the height estimates. The data were acquired by the Advanced Topographic Laser Altimeter System (ATLAS) instrument on board the Ice, Cloud and land Elevation Satellite-2 (ICESat-2) observatory.
orbit_parameters :  {}
>>> 

Thanks again for the quick replies and help!

[mfisher87]

Are there any hypotheses I could send their way about why the 'feed' error might be happening?

This error is related to parsing the JSON response from NASA CMR. The failing query is https://cmr.earthdata.nasa.gov/search/collections.json?short_name=ATL06. "feed" is the first key in a valid JSON response, and icepyx is trying to access it, it's not finding it. Perhaps a proxy is set up for the cluster?

Thanks again for the quick replies and help!

Happy to help!

[mfisher87]

I'm going to mark this closed, but commenting is still OK in a closed issue, and we can always re-open it if we learn something new. I'd love to hear about what you hear back from the folks managing your departmental cluster!