search

fraymio / modis-tools

Tools for working with the MODIS API and MODIS data.
Apache License 2.0
23 stars 12 forks source link

Granule appears to only account for ~4 files when iterating over 1572 feature gdf #10

Closed sgmurphy216 closed 2 years ago

sgmurphy216 commented 2 years ago

Hello! Thank you for producing this tool, I am currently using it to download oceanic/coastal MOD09A1 V006 tiles for use in my thesis.

Describe the bug A clear and concise description of what the bug is.

When iterating through an entire ~1580 row gdf (from a .json), it appears modis-tools is only downloading 4 files at once. However, generating granules from different iloc subsets of the same gdf yields different files (still to a maximum of 4 at a time). Geometries in gdf are pre-generated bounding boxes, date range is 22 days.

To Reproduce Steps to reproduce the behavior:

  1. Go to '...'
  2. Click on '....'
  3. Scroll down to '....'
  4. See error

I've included screenshots of my code and of the outputs from iterating over the entire dataframe vs subsets of it. The tiles I'm looking for do appear to be present in the EE dataset, and the code runs without error- I would be happy to provide my code/dataset, but cannot submit .ipynb type through an issue.

Here is a link on how to make a good reproducible example.

Expected behavior and actual behavior Expected an output of ~1580 .hdf files, received 3-4 at a time.

For example: I expected to download Nigeria MOD13A2 tiles. However, the following exception occurred:

Screenshots image

image

image

image

Desktop (please complete the following information):

Your Environment:

alembic @ file:///home/conda/feedstock_root/build_artifacts/alembic_1647367721563/work altair @ file:///home/conda/feedstock_root/build_artifacts/altair_1640799865332/work anyio @ file:///home/conda/feedstock_root/build_artifacts/anyio_1641898043316/work/dist argon2-cffi @ file:///home/conda/feedstock_root/build_artifacts/argon2-cffi_1640817743617/work argon2-cffi-bindings @ file:///home/conda/feedstock_root/build_artifacts/argon2-cffi-bindings_1640855140139/work asttokens @ file:///home/conda/feedstock_root/build_artifacts/asttokens_1618968359944/work async-generator==1.10 attrs @ file:///home/conda/feedstock_root/build_artifacts/attrs_1640799537051/work Babel @ file:///home/conda/feedstock_root/build_artifacts/babel_1619719576210/work backcall @ file:///home/conda/feedstock_root/build_artifacts/backcall_1592338393461/work backports.functools-lru-cache @ file:///home/conda/feedstock_root/build_artifacts/backports.functools_lru_cache_1618230623929/work beautifulsoup4 @ file:///home/conda/feedstock_root/build_artifacts/beautifulsoup4_1631087867185/work bleach @ file:///home/conda/feedstock_root/build_artifacts/bleach_1629908509068/work blinker==1.4 bokeh @ file:///home/conda/feedstock_root/build_artifacts/bokeh_1637615122801/work Bottleneck @ file:///home/conda/feedstock_root/build_artifacts/bottleneck_1645538067161/work brotlipy @ file:///home/conda/feedstock_root/build_artifacts/brotlipy_1636012194889/work cached-property @ file:///home/conda/feedstock_root/build_artifacts/cached_property_1615209429212/work certifi==2021.10.8 certipy==0.1.3 cffi @ file:///home/conda/feedstock_root/build_artifacts/cffi_1636046055389/work charset-normalizer @ file:///home/conda/feedstock_root/build_artifacts/charset-normalizer_1644853463426/work click @ file:///home/conda/feedstock_root/build_artifacts/click_1645238100595/work click-plugins==1.1.1 cligj==0.7.2 cloudpickle @ file:///home/conda/feedstock_root/build_artifacts/cloudpickle_1631273254894/work colorama @ file:///home/conda/feedstock_root/build_artifacts/colorama_1602866480661/work conda==4.11.0 conda-package-handling @ file:///home/conda/feedstock_root/build_artifacts/conda-package-handling_1636021710069/work cryptography @ file:///home/conda/feedstock_root/build_artifacts/cryptography_1639699280527/work cycler @ file:///home/conda/feedstock_root/build_artifacts/cycler_1635519461629/work Cython @ file:///home/conda/feedstock_root/build_artifacts/cython_1645125462806/work cytoolz==0.11.2 dask @ file:///home/conda/feedstock_root/build_artifacts/dask-core_1647641559712/work debugpy @ file:///home/conda/feedstock_root/build_artifacts/debugpy_1636043249011/work decorator @ file:///home/conda/feedstock_root/build_artifacts/decorator_1641555617451/work defusedxml @ file:///home/conda/feedstock_root/build_artifacts/defusedxml_1615232257335/work dill @ file:///home/conda/feedstock_root/build_artifacts/dill_1623610058511/work distributed @ file:///home/conda/feedstock_root/build_artifacts/distributed_1647650208834/work entrypoints @ file:///home/conda/feedstock_root/build_artifacts/entrypoints_1643888246732/work executing @ file:///home/conda/feedstock_root/build_artifacts/executing_1646044401614/work Fiona==1.8.21 flit_core @ file:///home/conda/feedstock_root/build_artifacts/flit-core_1645629044586/work/source/flit_core fonttools @ file:///home/conda/feedstock_root/build_artifacts/fonttools_1647621553730/work fsspec @ file:///home/conda/feedstock_root/build_artifacts/fsspec_1645566723803/work geopandas==0.10.2 gmpy2 @ file:///home/conda/feedstock_root/build_artifacts/gmpy2_1641732907770/work greenlet @ file:///home/conda/feedstock_root/build_artifacts/greenlet_1635836790477/work h5py @ file:///home/conda/feedstock_root/build_artifacts/h5py_1637963928098/work HeapDict==1.0.1 idna @ file:///home/conda/feedstock_root/build_artifacts/idna_1642433548627/work imagecodecs @ file:///home/conda/feedstock_root/build_artifacts/imagecodecs_1645704981320/work imageio @ file:///home/conda/feedstock_root/build_artifacts/imageio_1646036686179/work importlib-metadata @ file:///home/conda/feedstock_root/build_artifacts/importlib-metadata_1647210358856/work importlib-resources @ file:///home/conda/feedstock_root/build_artifacts/importlib_resources_1635615662634/work ipykernel @ file:///home/conda/feedstock_root/build_artifacts/ipykernel_1647271115035/work/dist/ipykernel-6.9.2-py3-none-any.whl ipympl @ file:///home/conda/feedstock_root/build_artifacts/ipympl_1645174282670/work ipython @ file:///home/conda/feedstock_root/build_artifacts/ipython_1646324680646/work ipython-genutils==0.2.0 ipywidgets @ file:///home/conda/feedstock_root/build_artifacts/ipywidgets_1647456365981/work jedi @ file:///home/conda/feedstock_root/build_artifacts/jedi_1637175083648/work Jinja2 @ file:///home/conda/feedstock_root/build_artifacts/jinja2_1636510082894/work joblib @ file:///home/conda/feedstock_root/build_artifacts/joblib_1633637554808/work json5 @ file:///home/conda/feedstock_root/build_artifacts/json5_1600692310011/work jsonschema @ file:///home/conda/feedstock_root/build_artifacts/jsonschema-meta_1642000296051/work jupyter-client @ file:///home/conda/feedstock_root/build_artifacts/jupyter_client_1642858610849/work jupyter-core @ file:///home/conda/feedstock_root/build_artifacts/jupyter_core_1645024267147/work jupyter-server @ file:///home/conda/feedstock_root/build_artifacts/jupyter_server_1647472848795/work jupyter-telemetry @ file:///home/conda/feedstock_root/build_artifacts/jupyter_telemetry_1605173804246/work jupyterhub @ file:///home/conda/feedstock_root/build_artifacts/jupyterhub-feedstock_1647267198467/work jupyterlab @ file:///home/conda/feedstock_root/build_artifacts/jupyterlab_1647279328649/work jupyterlab-pygments @ file:///home/conda/feedstock_root/build_artifacts/jupyterlab_pygments_1601375948261/work jupyterlab-server @ file:///home/conda/feedstock_root/build_artifacts/jupyterlab_server_1647439851306/work jupyterlab-widgets @ file:///home/conda/feedstock_root/build_artifacts/jupyterlab_widgets_1647446862951/work kiwisolver @ file:///home/conda/feedstock_root/build_artifacts/kiwisolver_1647351495879/work libmambapy @ file:///home/conda/feedstock_root/build_artifacts/mamba-split_1644831758535/work/libmambapy llvmlite==0.38.0 locket==0.2.0 Mako @ file:///home/conda/feedstock_root/build_artifacts/mako_1646959760357/work mamba @ file:///home/conda/feedstock_root/build_artifacts/mamba-split_1644831758535/work/mamba MarkupSafe @ file:///home/conda/feedstock_root/build_artifacts/markupsafe_1647364298597/work matplotlib @ file:///home/conda/feedstock_root/build_artifacts/matplotlib-suite_1639359646028/work matplotlib-inline @ file:///home/conda/feedstock_root/build_artifacts/matplotlib-inline_1631080358261/work mistune @ file:///home/conda/feedstock_root/build_artifacts/mistune_1635844679781/work modis-tools==1.0.0 mpmath @ file:///home/conda/feedstock_root/build_artifacts/mpmath_1612895720168/work msgpack @ file:///home/conda/feedstock_root/build_artifacts/msgpack-python_1637771281198/work munch==2.5.0 munkres==1.1.4 nbclassic @ file:///home/conda/feedstock_root/build_artifacts/nbclassic_1647450696711/work nbclient @ file:///home/conda/feedstock_root/build_artifacts/nbclient_1646999386773/work nbconvert @ file:///home/conda/feedstock_root/build_artifacts/nbconvert_1647040509207/work nbformat @ file:///home/conda/feedstock_root/build_artifacts/nbformat_1646951096007/work nest-asyncio @ file:///home/conda/feedstock_root/build_artifacts/nest-asyncio_1638419302549/work networkx @ file:///home/conda/feedstock_root/build_artifacts/networkx_1646497321764/work notebook @ file:///home/conda/feedstock_root/build_artifacts/notebook_1647377876077/work notebook-shim @ file:///home/conda/feedstock_root/build_artifacts/notebook-shim_1646330736330/work numba @ file:///home/conda/feedstock_root/build_artifacts/numba_1644003844419/work numexpr @ file:///home/conda/feedstock_root/build_artifacts/numexpr_1643220666587/work numpy @ file:///home/conda/feedstock_root/build_artifacts/numpy_1640083068831/work oauthlib @ file:///home/conda/feedstock_root/build_artifacts/oauthlib_1643507977997/work packaging @ file:///home/conda/feedstock_root/build_artifacts/packaging_1637239678211/work pamela==1.0.0 pandas==1.4.1 pandocfilters @ file:///home/conda/feedstock_root/build_artifacts/pandocfilters_1631603243851/work parso @ file:///home/conda/feedstock_root/build_artifacts/parso_1638334955874/work partd @ file:///home/conda/feedstock_root/build_artifacts/partd_1617910651905/work patsy @ file:///home/conda/feedstock_root/build_artifacts/patsy_1632667180946/work pexpect @ file:///home/conda/feedstock_root/build_artifacts/pexpect_1602535608087/work pickleshare @ file:///home/conda/feedstock_root/build_artifacts/pickleshare_1602536217715/work Pillow @ file:///home/conda/feedstock_root/build_artifacts/pillow_1645323160399/work prometheus-client @ file:///home/conda/feedstock_root/build_artifacts/prometheus_client_1643395600215/work prompt-toolkit @ file:///home/conda/feedstock_root/build_artifacts/prompt-toolkit_1644497866770/work protobuf==3.19.4 psutil @ file:///home/conda/feedstock_root/build_artifacts/psutil_1640887138049/work ptyprocess @ file:///home/conda/feedstock_root/build_artifacts/ptyprocess_1609419310487/work/dist/ptyprocess-0.7.0-py2.py3-none-any.whl pure-eval @ file:///home/conda/feedstock_root/build_artifacts/pure_eval_1642875951954/work pycosat @ file:///home/conda/feedstock_root/build_artifacts/pycosat_1636020362356/work pycparser @ file:///home/conda/feedstock_root/build_artifacts/pycparser_1636257122734/work pycurl==7.45.1 pydantic==1.9.0 Pygments @ file:///home/conda/feedstock_root/build_artifacts/pygments_1641580240686/work PyJWT @ file:///home/conda/feedstock_root/build_artifacts/pyjwt_1638819640841/work pyOpenSSL @ file:///home/conda/feedstock_root/build_artifacts/pyopenssl_1643496850550/work pyparsing @ file:///home/conda/feedstock_root/build_artifacts/pyparsing_1642753572664/work pyproj==3.3.0 pyrsistent @ file:///home/conda/feedstock_root/build_artifacts/pyrsistent_1642534390768/work PySocks @ file:///home/conda/feedstock_root/build_artifacts/pysocks_1635862404942/work python-dateutil @ file:///home/conda/feedstock_root/build_artifacts/python-dateutil_1626286286081/work python-json-logger @ file:///home/conda/feedstock_root/build_artifacts/python-json-logger_1602545356084/work pytz @ file:///home/conda/feedstock_root/build_artifacts/pytz_1633452062248/work PyWavelets @ file:///home/conda/feedstock_root/build_artifacts/pywavelets_1647035547589/work PyYAML @ file:///home/conda/feedstock_root/build_artifacts/pyyaml_1636139833192/work pyzmq @ file:///home/conda/feedstock_root/build_artifacts/pyzmq_1635877397296/work requests @ file:///home/conda/feedstock_root/build_artifacts/requests_1641580202195/work ruamel-yaml-conda @ file:///home/conda/feedstock_root/build_artifacts/ruamel_yaml_1636009144459/work ruamel.yaml @ file:///home/conda/feedstock_root/build_artifacts/ruamel.yaml_1644759508211/work ruamel.yaml.clib @ file:///home/conda/feedstock_root/build_artifacts/ruamel.yaml.clib_1636815619750/work scikit-image @ file:///home/conda/feedstock_root/build_artifacts/scikit-image_1645196657675/work scikit-learn @ file:///home/conda/feedstock_root/build_artifacts/scikit-learn_1640464150285/work scipy @ file:///home/conda/feedstock_root/build_artifacts/scipy_1644357253444/work seaborn @ file:///home/conda/feedstock_root/build_artifacts/seaborn-split_1629095986539/work Send2Trash @ file:///home/conda/feedstock_root/build_artifacts/send2trash_1628511208346/work Shapely==1.8.1.post1 six @ file:///home/conda/feedstock_root/build_artifacts/six_1620240208055/work sniffio @ file:///home/conda/feedstock_root/build_artifacts/sniffio_1635844660462/work sortedcontainers @ file:///home/conda/feedstock_root/build_artifacts/sortedcontainers_1621217038088/work soupsieve @ file:///home/conda/feedstock_root/build_artifacts/soupsieve_1638550740809/work SQLAlchemy @ file:///home/conda/feedstock_root/build_artifacts/sqlalchemy_1646615291967/work stack-data @ file:///home/conda/feedstock_root/build_artifacts/stack_data_1644872665635/work statsmodels @ file:///home/conda/feedstock_root/build_artifacts/statsmodels_1644535581977/work sympy @ file:///home/conda/feedstock_root/build_artifacts/sympy_1647802532210/work tables @ file:///home/conda/feedstock_root/build_artifacts/pytables_1643135938562/work tblib @ file:///home/conda/feedstock_root/build_artifacts/tblib_1616261298899/work terminado @ file:///home/conda/feedstock_root/build_artifacts/terminado_1646684455438/work testpath @ file:///home/conda/feedstock_root/build_artifacts/testpath_1645693042223/work threadpoolctl @ file:///home/conda/feedstock_root/build_artifacts/threadpoolctl_1643647933166/work tifffile @ file:///home/conda/feedstock_root/build_artifacts/tifffile_1647539359111/work toolz @ file:///home/conda/feedstock_root/build_artifacts/toolz_1636177962100/work tornado @ file:///home/conda/feedstock_root/build_artifacts/tornado_1635819584296/work tqdm @ file:///home/conda/feedstock_root/build_artifacts/tqdm_1632160078689/work traitlets @ file:///home/conda/feedstock_root/build_artifacts/traitlets_1635260543454/work typing_extensions @ file:///home/conda/feedstock_root/build_artifacts/typing_extensions_1644850595256/work unicodedata2 @ file:///home/conda/feedstock_root/build_artifacts/unicodedata2_1640031013105/work urllib3 @ file:///home/conda/feedstock_root/build_artifacts/urllib3_1641584929973/work wcwidth @ file:///home/conda/feedstock_root/build_artifacts/wcwidth_1600965781394/work webencodings==0.5.1 websocket-client @ file:///home/conda/feedstock_root/build_artifacts/websocket-client_1645884408572/work widgetsnbextension @ file:///home/conda/feedstock_root/build_artifacts/widgetsnbextension_1647446890517/work xlrd @ file:///home/conda/feedstock_root/build_artifacts/xlrd_1610224409810/work zict @ file:///home/conda/feedstock_root/build_artifacts/zict_1645832288898/work zipp @ file:///home/conda/feedstock_root/build_artifacts/zipp_1643828507773/work

Any Additional context Add any other context about the problem here.

jtanwk commented 2 years ago

Hi Sean, thanks for submitting an issue. Looking at your screenshots, I think the issue is with how you're assembling the list of granules to download.

In your third screenshot, you're executing something like:

for col, row in gdf.iterrows():
    erikson_bbox = row['geometry']
    erikson_granules = granule_client.query(....)

GranuleHandler.download_from_granules(erikson_granules, ...)

The problem is that erikson_granules is overwritten at each loop. So when you call GranuleHandler.download_from_granules afterwards, erikson_granules only lists the granules that correspond to the bounding box in the last row of gdf. That's why, for subsequent loops over gdf.iloc[50:1000] and gdf.iloc[1001:1500], you end up only downloading 3-4 files - they are the files for gdf.iloc[999] and gdf.iloc[1499] respectively.

Two ways to handle this:

  1. Just put the GranuleHandler.download_from_granules call inside your for loop. This will be very slow. 
    for col, row in gdf.iterrows():
erikson_bbox = row['geometry']
erikson_granules = granule_client.query(....)
GranuleHandler.download_from_granules(erikson_granules, ...)
  2. Per @lmcindewar 's suggestion in our prior email exchange, use the for loop to accumulate the list of granules (add to the list with every loop instead of reassigning to it) and download all at once after, like so: 
    
erikson_granules = []
for col, row in gdf.iterrows():
# erikson_bbox = [row['xmax'], row['ymax'], row['xmin'], row['ymin']]
erikson_bbox = row['geometry']
row_granules = granule_client.query(start_date=row['Daterange_'], end_date=row['Daterang_1'], bounding_box=erikson_bbox, limit = 5000)
erikson_granules.extend(list(row_granules))

GranuleHandler.download_from_granules(erikson_granules, ...)



Do try that out and let us know if it works! We tried it on our side with no issues.
sgmurphy216 commented 2 years ago

Implemented your change - thank you so much for correcting my little indentation error. Looks to be running perfectly now. Thanks again for the module, you all couldn't have made it easier to use.