search

fraymio / modis-tools

Tools for working with the MODIS API and MODIS data.
Apache License 2.0
23 stars 12 forks source link

'pydantic.error_wrappers.ValidationError: 12 validation errors for GranuleFeed' error when download the MOD13Q1.061 product #27

Closed xeno0 closed 1 year ago

xeno0 commented 1 year ago

Describe the bug

When donwload the MOD13Q1.061 product using the API (code is attached below), the pydantic.error_wrappers.ValidationError: 12 validation errors for GranuleFeed comes out.

To Reproduce Steps to reproduce the behavior:

snwe = [34.5, 34.74, -91.5, -91.26]
start_date = "2022-01-01"
end_date = "2022-02-01"

S, N, W, E = snwe

# Reusable session
session = ModisSession(username=username, password=password)
collection_client = CollectionApi(session=session)

collections = collection_client.query(short_name="MOD13Q1", version="061")
granule_client = GranuleApi.from_collection(collections[0], session=session)

granules = granule_client.query(
    start_date=start_date, end_date=end_date, bounding_box=[W, S, E, N]
)

print(list(granules))

Expected behavior and actual behavior I expected to download MOD13Q1.061 tiles. However, the following exception occurred:

Traceback (most recent call last): File "/Users/xeno/Documents/carbon/carbon/rsinputs/modis/modis.py", line 90, in print(list(granules)) File "/Users/xeno/miniconda3/envs/co2/lib/python3.9/site-packages/modis_tools/resources.py", line 119, in query granule_feed = GranuleFeed(**feed) File "pydantic/main.py", line 341, in pydantic.main.BaseModel.init pydantic.error_wrappers.ValidationError: 12 validation errors for GranuleFeed entry -> 0 -> links -> 1 -> href URL scheme not permitted (type=value_error.url.scheme; allowed_schemes={'https', 'http'}) entry -> 0 -> links -> 4 -> href URL scheme not permitted (type=value_error.url.scheme; allowed_schemes={'https', 'http'}) entry -> 0 -> links -> 7 -> href URL scheme not permitted (type=value_error.url.scheme; allowed_schemes={'https', 'http'}) entry -> 0 -> links -> 9 -> href URL scheme not permitted (type=value_error.url.scheme; allowed_schemes={'https', 'http'}) entry -> 1 -> links -> 1 -> href URL scheme not permitted (type=value_error.url.scheme; allowed_schemes={'https', 'http'}) entry -> 1 -> links -> 4 -> href URL scheme not permitted (type=value_error.url.scheme; allowed_schemes={'https', 'http'}) entry -> 1 -> links -> 7 -> href URL scheme not permitted (type=value_error.url.scheme; allowed_schemes={'https', 'http'}) entry -> 1 -> links -> 9 -> href URL scheme not permitted (type=value_error.url.scheme; allowed_schemes={'https', 'http'}) entry -> 2 -> links -> 1 -> href URL scheme not permitted (type=value_error.url.scheme; allowed_schemes={'https', 'http'}) entry -> 2 -> links -> 4 -> href URL scheme not permitted (type=value_error.url.scheme; allowed_schemes={'https', 'http'}) entry -> 2 -> links -> 7 -> href URL scheme not permitted (type=value_error.url.scheme; allowed_schemes={'https', 'http'}) entry -> 2 -> links -> 9 -> href URL scheme not permitted (type=value_error.url.scheme; allowed_schemes={'https', 'http'})

Screenshots If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

Your Environment:

Any Additional context Add any other context about the problem here.

jamie-sgro commented 1 year ago

Hi @xeno0, thanks for submitting this issue. Our team has received it and loaded it into our internal issue tracking system. We hope to address it in the coming weeks

ilsep93 commented 1 year ago

Hi @xeno0 ! The modis-tools team recently updated our pydantic validation to use anAnyUrl format (a more flexible validation than the previous allowed_schemes={'https', 'http'}) that you were receiving as an error (see https://github.com/fraymio/modis-tools/commit/c2d8c6fb3c37a530affc7b910dc03c0d419fe908).

I was able to replicate your code just now and downloaded the requested granules.

After pulling the latest code, let us know if you have any similar problems! Thanks for raising this issue

xeno0 commented 1 year ago

Hi @ilsep93 I tested this commit, it seems that there are still some errors on my end, please see below, thank you

Downloading:   0%|                                                                                             | 0/2 [00:00<?, ?file/s]
concurrent.futures.process._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/Users/xeno/miniconda3/envs/co2/lib/python3.9/concurrent/futures/process.py", line 246, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/Users/xeno/miniconda3/envs/co2/lib/python3.9/concurrent/futures/process.py", line 205, in _process_chunk
    return [fn(*args) for args in chunk]
  File "/Users/xeno/miniconda3/envs/co2/lib/python3.9/concurrent/futures/process.py", line 205, in <listcomp>
    return [fn(*args) for args in chunk]
  File "/Users/xeno/Documents/modis-tools/modis_tools/granule_handler.py", line 80, in wrapper_download_from_urls
    return cls.download_from_urls(*args)
  File "/Users/xeno/Documents/modis-tools/modis_tools/granule_handler.py", line 139, in download_from_urls
    req = cls._get(url, modis_session)
  File "/Users/xeno/Documents/modis-tools/modis_tools/granule_handler.py", line 167, in _get
    location = cls._get_location(url, modis_session)
  File "/Users/xeno/Documents/modis-tools/modis_tools/granule_handler.py", line 182, in _get_location
    location_resp = session.get(https_url, allow_redirects=False)
  File "/Users/xeno/miniconda3/envs/co2/lib/python3.9/site-packages/requests/sessions.py", line 600, in get
    return self.request("GET", url, **kwargs)
  File "/Users/xeno/miniconda3/envs/co2/lib/python3.9/site-packages/requests/sessions.py", line 573, in request
    prep = self.prepare_request(req)
  File "/Users/xeno/miniconda3/envs/co2/lib/python3.9/site-packages/requests/sessions.py", line 484, in prepare_request
    p.prepare(
  File "/Users/xeno/miniconda3/envs/co2/lib/python3.9/site-packages/requests/models.py", line 368, in prepare
    self.prepare_url(url, params)
  File "/Users/xeno/miniconda3/envs/co2/lib/python3.9/site-packages/requests/models.py", line 445, in prepare_url
    raise InvalidURL(f"Invalid URL {url!r}: No host supplied")
requests.exceptions.InvalidURL: Invalid URL 'https:///h': No host supplied
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/xeno/Documents/carbon/carbon/rsinputs/modis/modis.py", line 176, in <module>
    modis_lists = download_modis250m_ndvi(
  File "/Users/xeno/Documents/carbon/carbon/rsinputs/modis/modis.py", line 136, in download_modis250m_ndvi
    modis_lists = GranuleHandler.download_from_granules(
  File "/Users/xeno/Documents/modis-tools/modis_tools/granule_handler.py", line 62, in download_from_granules
    result = process_map(
  File "/Users/xeno/miniconda3/envs/co2/lib/python3.9/site-packages/tqdm/contrib/concurrent.py", line 105, in process_map
    return _executor_map(ProcessPoolExecutor, fn, *iterables, **tqdm_kwargs)
  File "/Users/xeno/miniconda3/envs/co2/lib/python3.9/site-packages/tqdm/contrib/concurrent.py", line 51, in _executor_map
    return list(tqdm_class(ex.map(fn, *iterables, chunksize=chunksize), **kwargs))
  File "/Users/xeno/miniconda3/envs/co2/lib/python3.9/site-packages/tqdm/std.py", line 1178, in __iter__
    for obj in iterable:
  File "/Users/xeno/miniconda3/envs/co2/lib/python3.9/concurrent/futures/process.py", line 562, in _chain_from_iterable_of_lists
    for element in iterable:
  File "/Users/xeno/miniconda3/envs/co2/lib/python3.9/concurrent/futures/_base.py", line 609, in result_iterator
    yield fs.pop().result()
  File "/Users/xeno/miniconda3/envs/co2/lib/python3.9/concurrent/futures/_base.py", line 446, in result
    return self.__get_result()
  File "/Users/xeno/miniconda3/envs/co2/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
requests.exceptions.InvalidURL: Invalid URL 'https:///h': No host supplied
ilsep93 commented 1 year ago

@xeno0 let's troubleshoot a bit more. Would you mind confirming if you pulled the most recent version of main (git pull main)? I ask because there was another recent update to the repo to use MODIS 061, which I noticed you were using in the reproducible example

xeno0 commented 1 year ago

Hi @ilsep93 thank you for quick reply! I first run the git pull origin main and then pip install -e ., after that, I run this code, 

   snwe = [34.5, 34.74, -91.5, -91.26]
   start_date = "2022-01-01"
   end_date = "2022-02-01"

    S, N, W, E = snwe

    # Reusable session
    session = ModisSession(username=username, password=password)
    collection_client = CollectionApi(session=session)

    collections = collection_client.query(short_name="MOD13Q1", version="061")
    granule_client = GranuleApi.from_collection(collections[0], session=session)

    granules = granule_client.query(
        start_date=start_date, end_date=end_date, bounding_box=[W, S, E, N]
    )

    # Download the MODIS file
    modis_lists = GranuleHandler.download_from_granules(
        granules, threads=4, modis_session=session, path=output_dir
    )
ilsep93 commented 1 year ago

Hi @xeno0, a few of us have not been able to replicate the error with the code you provided. The downloads work fine on our end. I'm closing this for now, but let us know if you keep running into issues