Granule Generation Variable Feed Error

[sgmurphy216]

Describe the bug A clear and concise description of what the bug is.

Hello again! Thank you for your previous response- I left the code running overnight, but returned to a strangely variable error this morning that I haven't been able to debug. When querying granules using the same gdf as in issue #10 , I inevitably run into a key error 'feed'. I am using the list method mentioned previously, and when re-running granule_client.query, the length of the list changes pretty significantly (anywhere from 0-1, 9, to 407 and 479 granules long). Wondering if this is an issue with my dataset, or if there is an argument I'm missing to pass over queries which produce this error?

To Reproduce Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Load .json into gdf, dropna geometries and daterange fields

for col, row in gdf.iterrows():
    erikson_bbox = row['geometry']
    row_granules = granule_client.query(start_date=row['Daterange_'], end_date=row['Daterang_1'], bounding_box=erikson_bbox,limit=20000)
    #GranuleHandler.download_from_granules(erikson_granules, modis_session=session, threads=4)
    erikson_granules.extend(list(row_granules))

# print(erikson_granules)
# GranuleHandler.download_from_granules(erikson_granules, modis_session=session)

Error message:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
File /opt/conda/lib/python3.9/site-packages/modis_tools/resources.py:123, in GranuleApi.query(self, start_date, end_date, time_delta, spatial, bounding_box, limit, **kwargs)
    122 resp = self.no_auth.get(params=params, auth=None)
--> 123 feed = resp.json()["feed"]
    124 granule_feed = GranuleFeed(**feed)

KeyError: 'feed'

The above exception was the direct cause of the following exception:

Exception                                 Traceback (most recent call last)
Input In [24], in <module>
      3 row_granules = granule_client.query(start_date=row['Daterange_'], end_date=row['Daterang_1'], bounding_box=erikson_bbox,limit=20000)
      4 #GranuleHandler.download_from_granules(erikson_granules, modis_session=session, threads=4)
----> 5 erikson_granules.extend(list(row_granules))

File /opt/conda/lib/python3.9/site-packages/modis_tools/resources.py:126, in GranuleApi.query(self, start_date, end_date, time_delta, spatial, bounding_box, limit, **kwargs)
    124     granule_feed = GranuleFeed(**feed)
    125 except (json.JSONDecodeError, KeyError, IndexError) as err:
--> 126     raise Exception("Can't read response") from err
    127 granules = granule_feed.entry
    128 if limit:

Exception: Can't read response

Here is a link on how to make a good reproducible example.

Expected behavior and actual behavior Please tell us what results you expected, as well as the output you got instead. If providing an error message, please make sure it's readable by formatting it with a code block.

For example: I expected to download Nigeria MOD13A2 tiles. However, the following exception occurred:

Expected response: ~5500 granules in list Actual response: 0-479 granules in list, changing each time code is run

Screenshots If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: Win 10 Pro
• Browser: Firefox
• Version 98

Your Environment:

• modis_tools version used: 1.0.0
• Other dependencies and packages installed in environment you are using (e.g. output of pip freeze or conda list or poetry show):

absl-py @ file:///home/conda/feedstock_root/build_artifacts/absl-py_1634676905105/work aiohttp @ file:///home/conda/feedstock_root/build_artifacts/aiohttp_1637087032947/work aiosignal @ file:///home/conda/feedstock_root/build_artifacts/aiosignal_1636093929600/work alembic @ file:///home/conda/feedstock_root/build_artifacts/alembic_1636656371607/work altair @ file:///home/conda/feedstock_root/build_artifacts/altair_1640799865332/work anyio @ file:///home/conda/feedstock_root/build_artifacts/anyio_1641898043316/work/dist appdirs @ file:///home/conda/feedstock_root/build_artifacts/appdirs_1603108395799/work argon2-cffi @ file:///home/conda/feedstock_root/build_artifacts/argon2-cffi_1640817743617/work argon2-cffi-bindings @ file:///home/conda/feedstock_root/build_artifacts/argon2-cffi-bindings_1640855140139/work asttokens @ file:///home/conda/feedstock_root/build_artifacts/asttokens_1618968359944/work astunparse @ file:///home/conda/feedstock_root/build_artifacts/astunparse_1610696312422/work async-generator==1.10 async-timeout @ file:///home/conda/feedstock_root/build_artifacts/async-timeout_1640026696943/work attrs @ file:///home/conda/feedstock_root/build_artifacts/attrs_1640799537051/work Babel @ file:///home/conda/feedstock_root/build_artifacts/babel_1619719576210/work backcall @ file:///home/conda/feedstock_root/build_artifacts/backcall_1592338393461/work backports.functools-lru-cache @ file:///home/conda/feedstock_root/build_artifacts/backports.functools_lru_cache_1618230623929/work beautifulsoup4 @ file:///home/conda/feedstock_root/build_artifacts/beautifulsoup4_1631087867185/work black @ file:///home/conda/feedstock_root/build_artifacts/black-recipe_1626468239465/work bleach @ file:///home/conda/feedstock_root/build_artifacts/bleach_1629908509068/work blinker==1.4 bokeh @ file:///home/conda/feedstock_root/build_artifacts/bokeh_1625756941579/work Bottleneck @ file:///home/conda/feedstock_root/build_artifacts/bottleneck_1636024229570/work brotlipy @ file:///home/conda/feedstock_root/build_artifacts/brotlipy_1636012194889/work cached-property @ file:///home/conda/feedstock_root/build_artifacts/cached_property_1615209429212/work cachetools @ file:///home/conda/feedstock_root/build_artifacts/cachetools_1633010882559/work certifi==2021.10.8 certipy==0.1.3 cffi @ file:///home/conda/feedstock_root/build_artifacts/cffi_1636046055389/work charset-normalizer @ file:///home/conda/feedstock_root/build_artifacts/charset-normalizer_1641584610847/work click @ file:///home/conda/feedstock_root/build_artifacts/click_1635822573388/work click-plugins==1.1.1 cligj==0.7.2 cloudpickle @ file:///home/conda/feedstock_root/build_artifacts/cloudpickle_1631273254894/work colorama @ file:///home/conda/feedstock_root/build_artifacts/colorama_1602866480661/work conda==4.11.0 conda-package-handling @ file:///home/conda/feedstock_root/build_artifacts/conda-package-handling_1636021710069/work cryptography @ file:///home/conda/feedstock_root/build_artifacts/cryptography_1639699280527/work cycler @ file:///home/conda/feedstock_root/build_artifacts/cycler_1635519461629/work Cython @ file:///home/conda/feedstock_root/build_artifacts/cython_1639702830128/work cytoolz==0.11.2 dask @ file:///home/conda/feedstock_root/build_artifacts/dask-core_1642200691755/work dataclasses @ file:///home/conda/feedstock_root/build_artifacts/dataclasses_1628958434797/work debugpy @ file:///home/conda/feedstock_root/build_artifacts/debugpy_1636043249011/work decorator @ file:///home/conda/feedstock_root/build_artifacts/decorator_1641555617451/work defusedxml @ file:///home/conda/feedstock_root/build_artifacts/defusedxml_1615232257335/work dill @ file:///home/conda/feedstock_root/build_artifacts/dill_1623610058511/work distributed @ file:///home/conda/feedstock_root/build_artifacts/distributed_1642210134503/work entrypoints @ file:///home/conda/feedstock_root/build_artifacts/entrypoints_1605121927639/work/dist/entrypoints-0.3-py2.py3-none-any.whl executing @ file:///home/conda/feedstock_root/build_artifacts/executing_1633213722787/work Fiona==1.8.21 flatbuffers @ file:///home/conda/feedstock_root/build_artifacts/python-flatbuffers_1617723079010/work flit_core @ file:///home/conda/feedstock_root/build_artifacts/flit-core_1640697294996/work/source/flit_core fonttools @ file:///home/conda/feedstock_root/build_artifacts/fonttools_1643090801278/work frozenlist @ file:///home/conda/feedstock_root/build_artifacts/frozenlist_1643222560572/work fsspec @ file:///home/conda/feedstock_root/build_artifacts/fsspec_1641933807587/work gast @ file:///home/conda/feedstock_root/build_artifacts/gast_1596839682936/work geopandas==0.10.2 gmpy2 @ file:///home/conda/feedstock_root/build_artifacts/gmpy2_1641732907770/work google-auth @ file:///home/conda/feedstock_root/build_artifacts/google-auth_1629296548061/work google-auth-oauthlib @ file:///home/conda/feedstock_root/build_artifacts/google-auth-oauthlib_1630497468950/work google-pasta==0.2.0 greenlet @ file:///home/conda/feedstock_root/build_artifacts/greenlet_1635836790477/work grpcio @ file:///home/conda/feedstock_root/build_artifacts/grpcio_1635853264746/work h5py @ file:///home/conda/feedstock_root/build_artifacts/h5py_1604753637292/work HeapDict==1.0.1 idna @ file:///home/conda/feedstock_root/build_artifacts/idna_1642433548627/work imagecodecs @ file:///home/conda/feedstock_root/build_artifacts/imagecodecs_1637750400866/work imageio @ file:///home/conda/feedstock_root/build_artifacts/imageio_1643214107019/work importlib-metadata @ file:///home/conda/feedstock_root/build_artifacts/importlib-metadata_1642513903188/work importlib-resources @ file:///home/conda/feedstock_root/build_artifacts/importlib_resources_1635615662634/work ipykernel @ file:///home/conda/feedstock_root/build_artifacts/ipykernel_1642098101968/work/dist/ipykernel-6.7.0-py3-none-any.whl ipympl @ file:///home/conda/feedstock_root/build_artifacts/ipympl_1642683202097/work ipython @ file:///home/conda/feedstock_root/build_artifacts/ipython_1642613619922/work ipython-genutils==0.2.0 ipywidgets @ file:///home/conda/feedstock_root/build_artifacts/ipywidgets_1631590360471/work jedi @ file:///home/conda/feedstock_root/build_artifacts/jedi_1637175083648/work Jinja2 @ file:///home/conda/feedstock_root/build_artifacts/jinja2_1636510082894/work joblib @ file:///home/conda/feedstock_root/build_artifacts/joblib_1633637554808/work json5 @ file:///home/conda/feedstock_root/build_artifacts/json5_1600692310011/work jsonschema @ file:///home/conda/feedstock_root/build_artifacts/jsonschema-meta_1642000296051/work jupyter-client @ file:///home/conda/feedstock_root/build_artifacts/jupyter_client_1642858610849/work jupyter-core @ file:///home/conda/feedstock_root/build_artifacts/jupyter_core_1636814260563/work jupyter-server @ file:///home/conda/feedstock_root/build_artifacts/jupyter_server_1642861369800/work jupyter-telemetry @ file:///home/conda/feedstock_root/build_artifacts/jupyter_telemetry_1605173804246/work jupyterhub @ file:///home/conda/feedstock_root/build_artifacts/jupyterhub-feedstock_1643124509075/work jupyterlab @ file:///home/conda/feedstock_root/build_artifacts/jupyterlab_1642103664616/work jupyterlab-pygments @ file:///home/conda/feedstock_root/build_artifacts/jupyterlab_pygments_1601375948261/work jupyterlab-server @ file:///home/conda/feedstock_root/build_artifacts/jupyterlab_server_1641592475363/work jupyterlab-widgets @ file:///home/conda/feedstock_root/build_artifacts/jupyterlab_widgets_1631590465624/work keras @ file:///home/conda/feedstock_root/build_artifacts/keras_1637159014053/work/keras-2.6.0-py2.py3-none-any.whl Keras-Preprocessing @ file:///home/conda/feedstock_root/build_artifacts/keras-preprocessing_1610713559828/work kiwisolver @ file:///home/conda/feedstock_root/build_artifacts/kiwisolver_1635836719639/work libmambapy @ file:///home/conda/feedstock_root/build_artifacts/mamba-split_1643117251182/work/libmambapy llvmlite==0.38.0 locket==0.2.0 Mako @ file:///home/conda/feedstock_root/build_artifacts/mako_1637170539581/work mamba @ file:///home/conda/feedstock_root/build_artifacts/mamba-split_1643117251182/work/mamba Markdown @ file:///home/conda/feedstock_root/build_artifacts/markdown_1637220118004/work MarkupSafe @ file:///home/conda/feedstock_root/build_artifacts/markupsafe_1635833544680/work matplotlib @ file:///home/conda/feedstock_root/build_artifacts/matplotlib-suite_1639359646028/work matplotlib-inline @ file:///home/conda/feedstock_root/build_artifacts/matplotlib-inline_1631080358261/work mistune @ file:///home/conda/feedstock_root/build_artifacts/mistune_1635844679781/work mock @ file:///home/conda/feedstock_root/build_artifacts/mock_1635819531655/work modis-tools==1.0.0 mpmath @ file:///home/conda/feedstock_root/build_artifacts/mpmath_1612895720168/work msgpack @ file:///home/conda/feedstock_root/build_artifacts/msgpack-python_1637771281198/work multidict @ file:///home/conda/feedstock_root/build_artifacts/multidict_1643055352339/work munch==2.5.0 munkres==1.1.4 mypy-extensions @ file:///home/conda/feedstock_root/build_artifacts/mypy_extensions_1635839744187/work nbclassic @ file:///home/conda/feedstock_root/build_artifacts/nbclassic_1641934401341/work nbclient @ file:///home/conda/feedstock_root/build_artifacts/nbclient_1642144614450/work nbconvert @ file:///home/conda/feedstock_root/build_artifacts/nbconvert_1643237836825/work nbformat @ file:///home/conda/feedstock_root/build_artifacts/nbformat_1617383142101/work nest-asyncio @ file:///home/conda/feedstock_root/build_artifacts/nest-asyncio_1638419302549/work networkx @ file:///home/conda/feedstock_root/build_artifacts/networkx_1635253012265/work notebook @ file:///home/conda/feedstock_root/build_artifacts/notebook_1643149609895/work numba @ file:///home/conda/feedstock_root/build_artifacts/numba_1642189988883/work numexpr @ file:///home/conda/feedstock_root/build_artifacts/numexpr_1643220666587/work numpy @ file:///home/conda/feedstock_root/build_artifacts/numpy_1626681925468/work oauthlib @ file:///home/conda/feedstock_root/build_artifacts/oauthlib_1622563202229/work olefile @ file:///home/conda/feedstock_root/build_artifacts/olefile_1602866521163/work opt-einsum @ file:///home/conda/feedstock_root/build_artifacts/opt_einsum_1617859230218/work packaging @ file:///home/conda/feedstock_root/build_artifacts/packaging_1637239678211/work pamela==1.0.0 pandas==1.4.0 pandocfilters @ file:///home/conda/feedstock_root/build_artifacts/pandocfilters_1631603243851/work parso @ file:///home/conda/feedstock_root/build_artifacts/parso_1638334955874/work partd @ file:///home/conda/feedstock_root/build_artifacts/partd_1617910651905/work pathspec @ file:///home/conda/feedstock_root/build_artifacts/pathspec_1626613672358/work patsy @ file:///home/conda/feedstock_root/build_artifacts/patsy_1632667180946/work pexpect @ file:///home/conda/feedstock_root/build_artifacts/pexpect_1602535608087/work pickleshare @ file:///home/conda/feedstock_root/build_artifacts/pickleshare_1602536217715/work Pillow @ file:///home/conda/feedstock_root/build_artifacts/pillow_1636558800840/work platformdirs @ file:///home/conda/feedstock_root/build_artifacts/platformdirs_1630400214373/work prometheus-client @ file:///home/conda/feedstock_root/build_artifacts/prometheus_client_1643149052602/work prompt-toolkit @ file:///home/conda/feedstock_root/build_artifacts/prompt-toolkit_1643301131884/work protobuf==3.18.1 psutil @ file:///home/conda/feedstock_root/build_artifacts/psutil_1640887138049/work ptyprocess @ file:///home/conda/feedstock_root/build_artifacts/ptyprocess_1609419310487/work/dist/ptyprocess-0.7.0-py2.py3-none-any.whl pure-eval @ file:///home/conda/feedstock_root/build_artifacts/pure_eval_1642875951954/work pyasn1==0.4.8 pyasn1-modules==0.2.7 pycosat @ file:///home/conda/feedstock_root/build_artifacts/pycosat_1636020362356/work pycparser @ file:///home/conda/feedstock_root/build_artifacts/pycparser_1636257122734/work pycurl==7.44.1 pydantic==1.9.0 Pygments @ file:///home/conda/feedstock_root/build_artifacts/pygments_1641580240686/work PyJWT @ file:///home/conda/feedstock_root/build_artifacts/pyjwt_1638819640841/work pyOpenSSL @ file:///home/conda/feedstock_root/build_artifacts/pyopenssl_1633192417276/work pyparsing @ file:///home/conda/feedstock_root/build_artifacts/pyparsing_1642753572664/work pyproj==3.3.0 pyrsistent @ file:///home/conda/feedstock_root/build_artifacts/pyrsistent_1642534390768/work PySocks @ file:///home/conda/feedstock_root/build_artifacts/pysocks_1635862404942/work python-dateutil @ file:///home/conda/feedstock_root/build_artifacts/python-dateutil_1626286286081/work python-json-logger @ file:///home/conda/feedstock_root/build_artifacts/python-json-logger_1602545356084/work pytz @ file:///home/conda/feedstock_root/build_artifacts/pytz_1633452062248/work pyu2f @ file:///home/conda/feedstock_root/build_artifacts/pyu2f_1604248910016/work PyWavelets @ file:///home/conda/feedstock_root/build_artifacts/pywavelets_1638209721487/work PyYAML @ file:///home/conda/feedstock_root/build_artifacts/pyyaml_1636139833192/work pyzmq @ file:///home/conda/feedstock_root/build_artifacts/pyzmq_1635877397296/work regex @ file:///home/conda/feedstock_root/build_artifacts/regex_1642551759955/work requests @ file:///home/conda/feedstock_root/build_artifacts/requests_1641580202195/work requests-oauthlib @ file:///home/conda/feedstock_root/build_artifacts/requests-oauthlib_1595492159598/work rsa @ file:///home/conda/feedstock_root/build_artifacts/rsa_1637781155505/work ruamel-yaml-conda @ file:///home/conda/feedstock_root/build_artifacts/ruamel_yaml_1636009144459/work ruamel.yaml @ file:///home/conda/feedstock_root/build_artifacts/ruamel.yaml_1640809661358/work ruamel.yaml.clib @ file:///home/conda/feedstock_root/build_artifacts/ruamel.yaml.clib_1636815619750/work scikit-image @ file:///home/conda/feedstock_root/build_artifacts/scikit-image_1639751509164/work scikit-learn @ file:///home/conda/feedstock_root/build_artifacts/scikit-learn_1640464150285/work scipy @ file:///home/conda/feedstock_root/build_artifacts/scipy_1637806669496/work seaborn @ file:///home/conda/feedstock_root/build_artifacts/seaborn-split_1629095986539/work Send2Trash @ file:///home/conda/feedstock_root/build_artifacts/send2trash_1628511208346/work Shapely==1.8.1.post1 six @ file:///home/conda/feedstock_root/build_artifacts/six_1590081179328/work sniffio @ file:///home/conda/feedstock_root/build_artifacts/sniffio_1635844660462/work sortedcontainers @ file:///home/conda/feedstock_root/build_artifacts/sortedcontainers_1621217038088/work soupsieve @ file:///home/conda/feedstock_root/build_artifacts/soupsieve_1638550740809/work SQLAlchemy @ file:///home/conda/feedstock_root/build_artifacts/sqlalchemy_1642761938807/work stack-data @ file:///home/conda/feedstock_root/build_artifacts/stack_data_1642255706390/work statsmodels @ file:///home/conda/feedstock_root/build_artifacts/statsmodels_1636765196796/work sympy @ file:///home/conda/feedstock_root/build_artifacts/sympy_1636237085010/work tables @ file:///home/conda/feedstock_root/build_artifacts/pytables_1610156075642/work tblib @ file:///home/conda/feedstock_root/build_artifacts/tblib_1616261298899/work tensorboard @ file:///home/conda/feedstock_root/build_artifacts/tensorboard_1629677129676/work/tensorboard-2.6.0-py3-none-any.whl tensorboard-data-server @ file:///home/conda/feedstock_root/build_artifacts/tensorboard-data-server_1636045763324/work/tensorboard_data_server-0.6.0-py3-none-manylinux2010_x86_64.whl tensorboard-plugin-wit @ file:///home/conda/feedstock_root/build_artifacts/tensorboard-plugin-wit_1641458951060/work/tensorboard_plugin_wit-1.8.1-py3-none-any.whl tensorflow @ file:///home/conda/feedstock_root/build_artifacts/tensorflow-split_1637975456804/work/tensorflow_pkg/tensorflow-2.6.2-cp39-cp39-linux_x86_64.whl tensorflow-estimator @ file:///home/conda/feedstock_root/build_artifacts/tensorflow-split_1637975456804/work/tensorflow-estimator/wheel_dir/tensorflow_estimator-2.6.0-py2.py3-none-any.whl termcolor==1.1.0 terminado @ file:///home/conda/feedstock_root/build_artifacts/terminado_1643296497241/work testpath @ file:///home/conda/feedstock_root/build_artifacts/testpath_1621261527237/work threadpoolctl @ file:///home/conda/feedstock_root/build_artifacts/threadpoolctl_1633102299089/work tifffile @ file:///home/conda/feedstock_root/build_artifacts/tifffile_1635944860688/work tomli @ file:///home/conda/feedstock_root/build_artifacts/tomli_1635181214134/work toolz @ file:///home/conda/feedstock_root/build_artifacts/toolz_1636177962100/work tornado @ file:///home/conda/feedstock_root/build_artifacts/tornado_1635819584296/work tqdm @ file:///home/conda/feedstock_root/build_artifacts/tqdm_1632160078689/work traitlets @ file:///home/conda/feedstock_root/build_artifacts/traitlets_1635260543454/work typed-ast @ file:///home/conda/feedstock_root/build_artifacts/typed-ast_1643045772392/work typing-extensions @ file:///home/conda/feedstock_root/build_artifacts/typing_extensions_1602702424206/work unicodedata2 @ file:///home/conda/feedstock_root/build_artifacts/unicodedata2_1640031013105/work urllib3 @ file:///home/conda/feedstock_root/build_artifacts/urllib3_1641584929973/work wcwidth @ file:///home/conda/feedstock_root/build_artifacts/wcwidth_1600965781394/work webencodings==0.5.1 websocket-client @ file:///home/conda/feedstock_root/build_artifacts/websocket-client_1638949723239/work Werkzeug @ file:///home/conda/feedstock_root/build_artifacts/werkzeug_1642714909499/work widgetsnbextension @ file:///home/conda/feedstock_root/build_artifacts/widgetsnbextension_1637174147962/work wrapt @ file:///home/conda/feedstock_root/build_artifacts/wrapt_1610094886776/work xlrd @ file:///home/conda/feedstock_root/build_artifacts/xlrd_1610224409810/work yarl @ file:///home/conda/feedstock_root/build_artifacts/yarl_1636046810561/work zict==2.0.0 zipp @ file:///home/conda/feedstock_root/build_artifacts/zipp_1641731510635/work

Any Additional context Add any other context about the problem here.

Thanks again for your prompt reply yesterday.

[sgmurphy216]

Added a quick try except to see how many rows were skipped:

[jtanwk]

Hi, thanks for submitting the issue. We were able to investigate and diagnose the issue.

In the code you sent, we have this loop:

erikson_granules = []
for col, row in gdf.iterrows():
    print(col)
    erikson_bbox = row['geometry']
    row_granules = granule_client.query(start_date=row['Daterange_'], end_date=row['Daterang_1'], bounding_box=erikson_bbox,limit=20000)
    #GranuleHandler.download_from_granules(erikson_granules, modis_session=session, threads=4)
    erikson_granules.extend(list(row_granules))

Specifically, the line withgranule_client.query(...) is making an API call to the NASA Earthdata API. Since your GDF has ~1500 rows (at least in the data you sent by email), you're making 1500+ individual calls to the API (one per row). The result is that you're getting rate-limited because of too many requests in a row after the first 600 or so -- hence the skipped rows.

We've added support for rate limited queries to our feature list for now, but probably can't give an ETA on resolution for now.

[iamchrisearle]

Closing this issue as it has become stale. If the recommended fix has not worked, feel free to open a new issue.