search

sentinel-hub / sentinelhub-py

Download and process satellite imagery in Python using Sentinel Hub services.
http://sentinelhub-py.readthedocs.io/en/latest/
MIT License
810 stars 248 forks source link

[BUG] `MissingTokenError` setting `config.sh_base_url` to point to CREODIAS #285

Closed AdrienWehrle closed 2 years ago

AdrienWehrle commented 2 years ago

Hi everyone,

This issue has been reported and described here. In my case, this happens setting config.sh_base_url to "https://creodias.sentinel-hub.com" to include Sentinel-3 and get a search iterator through a catalog.search() which is then used in a process API request as described here.

Thank you a lot in advance for your help!

zigaLuksic commented 2 years ago

Hi, we can try and pin down the problem.

Could you provide the following information so I get a better grip on the situation:

  1. Does the error happen already with the catalog.search (or within the iterator) or later when doing the requests?
  2. Full contents of the config (omit the id info) just before the problem, in case some weird modification happens somewhere.
  3. A stack trace would also help.
AdrienWehrle commented 2 years ago

Hi @zigaLuksic, thank you for your answer!

Does the error happen already with the catalog.search (or within the iterator) or later when doing the requests?

Only at the request step, I am able to get the search iterator.

To make the issue reproducible:

I am using earthspy on the include-s3 branch (in development), run the following and get the MissingTokenError: (missing_token) Missing access token parameter error.


import earthspy.earthspy as es

# initialize job
job = es.EarthSpy("auth.txt")

with open(
    "evalscript_olci_cop30.js", "r"
) as f:
    evalscript = f.read()

# pass string to evaluation_script
job.set_query_parameters(
    bounding_box=[-49.68, 69.02, -48.84, 69.23],
    time_interval=["2022-05-11", "2022-05-11"],
    evaluation_script=evalscript,
    data_collection="SENTINEL3_OLCI",
    algorithm="SICE",
)

# and off it goes!
job.send_sentinelhub_requests()

The evalscript is doing some operations on COP_30 and OLCI datasets. The actual request is happening in earthspy there.

cat /etc/os-release:

NAME="Ubuntu"
VERSION="20.04.4 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.4 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
zigaLuksic commented 2 years ago

I tried playing with the above example a bit, and I got errors in the job.set_query_parameters that I get due to get_available_data.

I noticed a few things, that might be worth investigating:

  1. I got a missing token problem, but it turns out I wrote the credentials into the auth.txt wrong. I would suggest you print out the config at the end of configure_connection and doublecheck there are no \" character in there, or similar oddities. I had "sh_client_id": "\"blablabla\""
  2. After fixing that, I actually get a DownloadFailedException because the catalog tried snooping around services.sentinel-hub.com and not creodias. I looked at the code and you create a catalog in configure_connection, but at that point you don't adjust the config.sh_base_url yet. The catalog saves the base url at initialization (see here) so I suspect this may cause issues for you as well.
AdrienWehrle commented 2 years ago
  1. I got a missing token problem, but it turns out I wrote the credentials into the auth.txt wrong. I would suggest you print out the config at the end of configure_connection and doublecheck there are no \" character in there, or similar oddities. I had "sh_client_id": "\"blablabla\""

Thank you a lot for this suggestion!

  1. After fixing that, I actually get a DownloadFailedException because the catalog tried snooping around services.sentinel-hub.com and not creodias. I looked at the code and you create a catalog in configure_connection, but at that point you don't adjust the config.sh_base_url yet. The catalog saves the base url at initialization (see here) so I suspect this may cause issues for you as well.

Thank you. I fixed this by moving the catalog initialization, however, I still get the same MissingTokenError: (missing_token) Missing access token parameter which I still don't really understand... Thank you a lot in advance for your help!

AleksMat commented 2 years ago

Hi @AdrienWehrle,

Thanks for reporting this. I ran your code with my credentials but it works fine. Do you get this issue every time or only occasionally?

We are investigating Sentinel Hub Authentication service for a potential cause of the problem.

AdrienWehrle commented 2 years ago

Thank you a lot for jumping in @AleksMat!

Thanks for reporting this. I ran your code with my credentials but it works fine. Do you get this issue every time or only occasionally?

So far I always got this error trying to run the code I shared above, every time. Just as a note, the example application of earthspy still works just fine, so it looks like it's really linked to this case.

We are investigating Sentinel Hub Authentication service for a potential cause of the problem.

Thank you a lot. I will also run more tests on my end to try and narrow it down.

Maybe is it linked to the account I use to make this request, maybe not giving access to creodias?

AleksMat commented 2 years ago

So far I always got this error trying to run the code I shared above, every time. Just as a note, the example application of earthspy still works just fine, so it looks like it's really linked to this case.

I'm quite sure the error happens at the beginning when a request to Sentinel Hub authentication service is made and a response somehow doesn't contain a token. However, this part of the process is the same for both of your use cases because authentication is always done to the main Sentinel Hub deployment (even in case of creodias). So I find it strange that in one case it works and in the other it doesn't.

Would you mind sharing a full stack trace of the error? Perhaps I'm mistaken and the error somehow happens later in the process. :thinking:

AdrienWehrle commented 2 years ago

Hi @AleksMat, here is the log: log.txt

I can go deeper in debug to get a more detailed log, just let me know if that is not enough!

AleksMat commented 2 years ago

Good news, we've identified the 1st part of the problem and prepared a solution: https://github.com/sentinel-hub/sentinelhub-py/pull/286. The problem was that requests_oauthlib package doesn't check for response status and because of that it always raised a completely wrong and misleading error instead of an actual error. We'll try to release a new package version early next week. But until then feel free to try using the branch that contains the fix - the difference will be that you'll now get the real error.

The 2nd part of the problem is to understand why you are getting an error in the first place. I think it might be one of the options:

Let me know if this solves the problem.

AdrienWehrle commented 2 years ago

Thank you a lot for your help on this @AleksMat! Indeed I had an old sentinelhub version, updating moved my issue to the next step with a more useful error report! Now, using your sice evalscript, I get the following: 

Initial bounding box split into a (2, 2) grid
---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
File ~/mambaforge3/envs/EO-IO/lib/python3.9/site-packages/sentinelhub/download/handlers.py:22, in fail_user_errors.<locals>.new_download_func(self, request)
     21 try:
---> 22     return download_func(self, request)
     23 except requests.HTTPError as exception:

File ~/mambaforge3/envs/EO-IO/lib/python3.9/site-packages/sentinelhub/download/sentinelhub_client.py:74, in SentinelHubDownloadClient._execute_download(self, request)
     72     continue
---> 74 response.raise_for_status()
     76 LOGGER.debug('Successful %s request to %s', request.request_type.value, request.url)

File ~/mambaforge3/envs/EO-IO/lib/python3.9/site-packages/requests/models.py:1022, in Response.raise_for_status(self)
   1021 if http_error_msg:
-> 1022     raise HTTPError(http_error_msg, response=self)

HTTPError: 400 Client Error: Bad Request for url: https://creodias.sentinel-hub.com/api/v1/process

The above exception was the direct cause of the following exception:

DownloadFailedException                   Traceback (most recent call last)
Input In [4], in <cell line: 1>()
----> 1 import sys, codecs, os, ast;__pyfile = codecs.open('''/tmp/pyfUKT1g''', encoding='''utf-8''');__code = __pyfile.read().encode('''utf-8''');__pyfile.close();os.remove('''/tmp/pyfUKT1g''');__block = ast.parse(__code, '''/home/adrien/EO-IO/earthspy-for-SICE/apply_SICE_with_earthspy.py''', mode='exec'); __block.body = (__block.body if not isinstance(__block.body[0], ast.If) else __block.body if not isinstance(__block.body[0].test, ast.Name) else __block.body if not __block.body[0].test.id == 'True' else __block.body[0].body) if sys.version_info[0] < 3 else (__block.body if not isinstance(__block.body[0], ast.If) else __block.body if not isinstance(__block.body[0].test, ast.NameConstant) else __block.body if not __block.body[0].test.value is True else __block.body[0].body);__last = __block.body[-1];__isexpr = isinstance(__last,ast.Expr);_ = __block.body.pop() if __isexpr else None;exec(compile(__block, '''/home/adrien/EO-IO/earthspy-for-SICE/apply_SICE_with_earthspy.py''', mode='exec'));eval(compile(ast.Expression(__last.value), '''/home/adrien/EO-IO/earthspy-for-SICE/apply_SICE_with_earthspy.py''', mode='eval')) if __isexpr else None

File ~/EO-IO/earthspy-for-SICE/apply_SICE_with_earthspy.py:29, in <module>
     20 job.set_query_parameters(
     21     bounding_box=[-49.68, 69.02, -48.84, 69.23],
     22     time_interval=["2022-05-11", "2022-05-11"],
   (...)
     25     algorithm="SICE",
     26 )
     28 # and off it goes!
---> 29 job.send_sentinelhub_requests()
     31 # %%
     32 import sentinelhub as shb

File ~/UZH/earthspy/earthspy/earthspy.py:855, in EarthSpy.send_sentinelhub_requests(self)
    852     start_local_time = time.ctime(start_time)
    854 # the actual Sentinel Hub download
--> 855 self.outputs = shb.SentinelHubDownloadClient(config=self.config).download(
    856     self.download_list, max_threads=self.nb_cores
    857 )
    859 # store raw folders created by Sentinel Hub API
    860 self.raw_filenames = [
    861     r.get_filename_list()[0].split(os.sep)[0] for r in self.requests_list
    862 ]

File ~/mambaforge3/envs/EO-IO/lib/python3.9/site-packages/sentinelhub/download/sentinelhub_client.py:51, in SentinelHubDownloadClient.download(self, *args, **kwargs)
     49 self.lock = Lock()
     50 try:
---> 51     return super().download(*args, **kwargs)
     52 finally:
     53     self.lock = None

File ~/mambaforge3/envs/EO-IO/lib/python3.9/site-packages/sentinelhub/download/client.py:87, in DownloadClient.download(self, download_requests, max_threads, decode_data, show_progress)
     85 else:
     86     for future in as_completed(download_list):
---> 87         data_list[future_order[future]] = self._process_download_future(future)
     89 if is_single_request:
     90     return data_list[0]

File ~/mambaforge3/envs/EO-IO/lib/python3.9/site-packages/sentinelhub/download/client.py:101, in DownloadClient._process_download_future(self, future)
     99 if self.raise_download_errors:
    100     traceback = sys.exc_info()[2]
--> 101     raise download_exception.with_traceback(traceback)
    103 warnings.warn(str(download_exception), category=SHRuntimeWarning)
    104 return None

File ~/mambaforge3/envs/EO-IO/lib/python3.9/site-packages/sentinelhub/download/client.py:97, in DownloadClient._process_download_future(self, future)
     94 """ Unpacks the future and correctly handles exceptions
     95 """
     96 try:
---> 97     return future.result()
     98 except DownloadFailedException as download_exception:
     99     if self.raise_download_errors:

File ~/mambaforge3/envs/EO-IO/lib/python3.9/concurrent/futures/_base.py:439, in Future.result(self, timeout)
    437     raise CancelledError()
    438 elif self._state == FINISHED:
--> 439     return self.__get_result()
    441 self._condition.wait(timeout)
    443 if self._state in [CANCELLED, CANCELLED_AND_NOTIFIED]:

File ~/mambaforge3/envs/EO-IO/lib/python3.9/concurrent/futures/_base.py:391, in Future.__get_result(self)
    389 if self._exception:
    390     try:
--> 391         raise self._exception
    392     finally:
    393         # Break a reference cycle with the exception in self._exception
    394         self = None

File ~/mambaforge3/envs/EO-IO/lib/python3.9/concurrent/futures/thread.py:58, in _WorkItem.run(self)
     55     return
     57 try:
---> 58     result = self.fn(*self.args, **self.kwargs)
     59 except BaseException as exc:
     60     self.future.set_exception(exc)

File ~/mambaforge3/envs/EO-IO/lib/python3.9/site-packages/sentinelhub/download/client.py:120, in DownloadClient._single_download(self, request, decode_data)
    117         return read_data(response_path, data_format=request.data_type if decode_data else MimeType.RAW)
    118     return None
--> 120 response_content = self._execute_download(request)
    122 if request_path and request.save_response and (self.redownload or not os.path.exists(request_path)):
    123     request_info = request.get_request_params(include_metadata=True)

File ~/mambaforge3/envs/EO-IO/lib/python3.9/site-packages/sentinelhub/download/handlers.py:44, in retry_temporary_errors.<locals>.new_download_func(self, request)
     42 for attempt_num in range(download_attempts):
     43     try:
---> 44         return download_func(self, request)
     45     except requests.RequestException as exception:
     47         if not (_is_temporary_problem(exception) or
     48                 (isinstance(exception, requests.HTTPError) and
     49                  exception.response.status_code >= requests.status_codes.codes.INTERNAL_SERVER_ERROR)):

File ~/mambaforge3/envs/EO-IO/lib/python3.9/site-packages/sentinelhub/download/handlers.py:27, in fail_user_errors.<locals>.new_download_func(self, request)
     23 except requests.HTTPError as exception:
     24     if (exception.response.status_code < requests.status_codes.codes.INTERNAL_SERVER_ERROR and
     25             exception.response.status_code != requests.status_codes.codes.TOO_MANY_REQUESTS):
---> 27         raise DownloadFailedException(_create_download_failed_message(exception, request.url)) from exception
     28     raise exception from exception

DownloadFailedException: Failed to download from:
https://creodias.sentinel-hub.com/api/v1/process
with HTTPError:
400 Client Error: Bad Request for url: https://creodias.sentinel-hub.com/api/v1/process
Server response: "{"error":{"status":400,"reason":"Bad Request","message":"Dataset with id: OLCI not found.","code":"RENDERER_EXCEPTION"}}"

working on it!

AdrienWehrle commented 2 years ago

You might have changed a parameter "sh_auth_base_url" in the package config to Creodias base URL. This would not work because for now authentication must always be done with the main deployment. So this parameter has to be set to "https://services.sentinel-hub.com". You can check your configuration file in command line with sentinelhub.config --show.

job.config with job being the instance of earthspy shared above, I get:


SHConfig(
  instance_id='',
  sh_client_id=<my_id>,
  sh_client_secret=<my_secret>,
  sh_base_url='https://creodias.sentinel-hub.com',
  sh_auth_base_url='https://services.sentinel-hub.com',
  geopedia_wms_url='https://service.geopedia.world',
  geopedia_rest_url='https://www.geopedia.world/rest',
  aws_access_key_id='',
  aws_secret_access_key='',
  aws_session_token='',
  aws_metadata_url='https://roda.sentinel-hub.com',
  aws_s3_l1c_bucket='sentinel-s2-l1c',
  aws_s3_l2a_bucket='sentinel-s2-l2a',
  opensearch_url='http://opensearch.sentinel-hub.com/resto/api/collections/Sentinel2',
  max_wfs_records_per_query=100,
  max_opensearch_records_per_query=500,
  max_download_attempts=4,
  download_sleep_time=5.0,
  download_timeout_seconds=120.0,
  number_of_download_processes=1
)

So this looks fine!

AleksMat commented 2 years ago

Excellent, the last remaining problem should be easier to resolve. I see that you are running a data fusion use case and that here you set an identifier name "OLCI". I'm not sure which evalscript you are using but the solution is that you have to use the same identifier also in the evalscript in the input part of the setup function:

...

function setup() {
  return {
    input: [
      {
        datasource: "OLCI",  // <- the identifier has to be written here
        bands: ...
      },
      ...
    ],
    ...

I also recommend checking the data fusion example in the sentinelhub-py documentation.

AdrienWehrle commented 2 years ago

Thanks for your help! I am using data_fusion_olci_dem.js that you shared as part of the SICE project and which contains the right identifier name:

//VERSION=3
function setup() {
  return {
    input: [
      {
        datasource: "OLCI",
        bands: ["B01", "B02", "B03", "B04", "B05", "B06", "B07", "B08", "B09", "B10", "B11", "B12", "B13", "B14", "B15", "B16", "B17", "B18", "B19", "B20", "B21", "SZA", "VZA", "SAA", "VAA", "TOTAL_COLUMN_OZONE"],
      },
      {
        datasource: "COP_30",
        bands: ["DEM"]
      },
      ...
    ],
    ...
mperse commented 2 years ago

Adrien do you define the datasources in your request? Under "input_data=" :

 request = SentinelHubRequest(
            evalscript=evalscript,
            data_folder=folderPath,
            input_data=[
                SentinelHubRequest.input_data(
                    data_collection=DataCollection.DEM_COPERNICUS_30,
                    identifier="COP_30",
                    upsampling="NEAREST",
                    downsampling="NEAREST",
                ),
                SentinelHubRequest.input_data(
                    data_collection=DataCollection.SENTINEL3_OLCI,
                    identifier="OLCI",
                    time_interval=date_range,
                    upsampling="NEAREST",
                    downsampling="NEAREST",
                ),
            ],
            responses=[
                SentinelHubRequest.output_response('r_TOA_01_valid', MimeType.TIFF),
                SentinelHubRequest.output_response('r_TOA_06_valid', MimeType.TIFF),
                SentinelHubRequest.output_response('r_TOA_17_valid', MimeType.TIFF),
                SentinelHubRequest.output_response('r_TOA_21_valid', MimeType.TIFF),
                SentinelHubRequest.output_response('Snow_Grain_Diameter', MimeType.TIFF),
                SentinelHubRequest.output_response('Snow_Specific_Surface_Area', MimeType.TIFF),
            ],
            bbox=bbox,
            size=size,
            config=sh_config,
        )
        resp = request.get_data(save_data=True)
AleksMat commented 2 years ago

I checked eartshpy code a bit more into detail and it turns out that in this line you initialize an instance of SentinelHubRequest correctly, however you don't assign this to shb_request variable. Instead shb_request holds some completely different request.

Today we released sentinelhub-py 3.6.2 which solves the problem with MissingTokenError. So I'm closing this issue.

AdrienWehrle commented 2 years ago

Thanks a lot @AleksMat, dummy mistake on my side that wasn't caught properly by earthspy. Thanks for your help!

AdrienWehrle commented 2 years ago

Hi @AleksMat, unfortunately an issue persists after this fix. Please see the error log below for the code shared above. I get the same error when running a simple Sentinel Hub request, independently of earthspy.

Thank you a lot in advance for your help!


---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
File ~/mambaforge3/envs/EO-IO/lib/python3.9/site-packages/sentinelhub/download/handlers.py:22, in fail_user_errors.<locals>.new_download_func(self, request)
     21 try:
---> 22     return download_func(self, request)
     23 except requests.HTTPError as exception:

File ~/mambaforge3/envs/EO-IO/lib/python3.9/site-packages/sentinelhub/download/sentinelhub_client.py:74, in SentinelHubDownloadClient._execute_download(self, request)
     72     continue
---> 74 response.raise_for_status()
     76 LOGGER.debug('Successful %s request to %s', request.request_type.value, request.url)

File ~/mambaforge3/envs/EO-IO/lib/python3.9/site-packages/requests/models.py:1022, in Response.raise_for_status(self)
   1021 if http_error_msg:
-> 1022     raise HTTPError(http_error_msg, response=self)

HTTPError: 400 Client Error: Bad Request for url: https://creodias.sentinel-hub.com/api/v1/process

The above exception was the direct cause of the following exception:

DownloadFailedException                   Traceback (most recent call last)
Input In [2], in <cell line: 1>()
----> 1 import sys, codecs, os, ast;__pyfile = codecs.open('''/tmp/pydW49oZ''', encoding='''utf-8''');__code = __pyfile.read().encode('''utf-8''');__pyfile.close();os.remove('''/tmp/pydW49oZ''');__block = ast.parse(__code, '''/home/adrien/EO-IO/earthspy-for-SICE/apply_SICE_with_earthspy.py''', mode='exec'); __block.body = (__block.body if not isinstance(__block.body[0], ast.If) else __block.body if not isinstance(__block.body[0].test, ast.Name) else __block.body if not __block.body[0].test.id == 'True' else __block.body[0].body) if sys.version_info[0] < 3 else (__block.body if not isinstance(__block.body[0], ast.If) else __block.body if not isinstance(__block.body[0].test, ast.NameConstant) else __block.body if not __block.body[0].test.value is True else __block.body[0].body);__last = __block.body[-1];__isexpr = isinstance(__last,ast.Expr);_ = __block.body.pop() if __isexpr else None;exec(compile(__block, '''/home/adrien/EO-IO/earthspy-for-SICE/apply_SICE_with_earthspy.py''', mode='exec'));eval(compile(ast.Expression(__last.value), '''/home/adrien/EO-IO/earthspy-for-SICE/apply_SICE_with_earthspy.py''', mode='eval')) if __isexpr else None

File ~/EO-IO/earthspy-for-SICE/apply_SICE_with_earthspy.py:29, in <module>
     20 job.set_query_parameters(
     21     bounding_box=[-49.68, 69.02, -48.84, 69.23],
     22     time_interval=["2022-05-11", "2022-05-11"],
   (...)
     25     algorithm="SICE",
     26 )
     28 # and off it goes!
---> 29 job.send_sentinelhub_requests()
     31 # %%
     32 import sentinelhub as shb

File ~/UZH/earthspy/earthspy/earthspy.py:855, in EarthSpy.send_sentinelhub_requests(self)
    852     start_local_time = time.ctime(start_time)
    854 # the actual Sentinel Hub download
--> 855 self.outputs = shb.SentinelHubDownloadClient(config=self.config).download(
    856     self.download_list, max_threads=self.nb_cores
    857 )
    859 # store raw folders created by Sentinel Hub API
    860 self.raw_filenames = [
    861     r.get_filename_list()[0].split(os.sep)[0] for r in self.requests_list
    862 ]

File ~/mambaforge3/envs/EO-IO/lib/python3.9/site-packages/sentinelhub/download/sentinelhub_client.py:51, in SentinelHubDownloadClient.download(self, *args, **kwargs)
     49 self.lock = Lock()
     50 try:
---> 51     return super().download(*args, **kwargs)
     52 finally:
     53     self.lock = None

File ~/mambaforge3/envs/EO-IO/lib/python3.9/site-packages/sentinelhub/download/client.py:87, in DownloadClient.download(self, download_requests, max_threads, decode_data, show_progress)
     85 else:
     86     for future in as_completed(download_list):
---> 87         data_list[future_order[future]] = self._process_download_future(future)
     89 if is_single_request:
     90     return data_list[0]

File ~/mambaforge3/envs/EO-IO/lib/python3.9/site-packages/sentinelhub/download/client.py:101, in DownloadClient._process_download_future(self, future)
     99 if self.raise_download_errors:
    100     traceback = sys.exc_info()[2]
--> 101     raise download_exception.with_traceback(traceback)
    103 warnings.warn(str(download_exception), category=SHRuntimeWarning)
    104 return None

File ~/mambaforge3/envs/EO-IO/lib/python3.9/site-packages/sentinelhub/download/client.py:97, in DownloadClient._process_download_future(self, future)
     94 """ Unpacks the future and correctly handles exceptions
     95 """
     96 try:
---> 97     return future.result()
     98 except DownloadFailedException as download_exception:
     99     if self.raise_download_errors:

File ~/mambaforge3/envs/EO-IO/lib/python3.9/concurrent/futures/_base.py:439, in Future.result(self, timeout)
    437     raise CancelledError()
    438 elif self._state == FINISHED:
--> 439     return self.__get_result()
    441 self._condition.wait(timeout)
    443 if self._state in [CANCELLED, CANCELLED_AND_NOTIFIED]:

File ~/mambaforge3/envs/EO-IO/lib/python3.9/concurrent/futures/_base.py:391, in Future.__get_result(self)
    389 if self._exception:
    390     try:
--> 391         raise self._exception
    392     finally:
    393         # Break a reference cycle with the exception in self._exception
    394         self = None

File ~/mambaforge3/envs/EO-IO/lib/python3.9/concurrent/futures/thread.py:58, in _WorkItem.run(self)
     55     return
     57 try:
---> 58     result = self.fn(*self.args, **self.kwargs)
     59 except BaseException as exc:
     60     self.future.set_exception(exc)

File ~/mambaforge3/envs/EO-IO/lib/python3.9/site-packages/sentinelhub/download/client.py:120, in DownloadClient._single_download(self, request, decode_data)
    117         return read_data(response_path, data_format=request.data_type if decode_data else MimeType.RAW)
    118     return None
--> 120 response_content = self._execute_download(request)
    122 if request_path and request.save_response and (self.redownload or not os.path.exists(request_path)):
    123     request_info = request.get_request_params(include_metadata=True)

File ~/mambaforge3/envs/EO-IO/lib/python3.9/site-packages/sentinelhub/download/handlers.py:44, in retry_temporary_errors.<locals>.new_download_func(self, request)
     42 for attempt_num in range(download_attempts):
     43     try:
---> 44         return download_func(self, request)
     45     except requests.RequestException as exception:
     47         if not (_is_temporary_problem(exception) or
     48                 (isinstance(exception, requests.HTTPError) and
     49                  exception.response.status_code >= requests.status_codes.codes.INTERNAL_SERVER_ERROR)):

File ~/mambaforge3/envs/EO-IO/lib/python3.9/site-packages/sentinelhub/download/handlers.py:27, in fail_user_errors.<locals>.new_download_func(self, request)
     23 except requests.HTTPError as exception:
     24     if (exception.response.status_code < requests.status_codes.codes.INTERNAL_SERVER_ERROR and
     25             exception.response.status_code != requests.status_codes.codes.TOO_MANY_REQUESTS):
---> 27         raise DownloadFailedException(_create_download_failed_message(exception, request.url)) from exception
     28     raise exception from exception

DownloadFailedException: Failed to download from:
https://creodias.sentinel-hub.com/api/v1/process
with HTTPError:
400 Client Error: Bad Request for url: https://creodias.sentinel-hub.com/api/v1/process
Server response: "{"error":{"status":400,"reason":"Bad Request","message":"Collection null is not supported here. Please use of the following endpoints: https://services.sentinel-hub.com.","code":"RENDERER_EXCEPTION"}}"
AdrienWehrle commented 2 years ago

I found out sh_base_url of config needs to be set to https://services.sentinel-hub.com for the download and to https://creodias.sentinel-hub.com to get the catalog before download. I was using the same config for both, and as a result could get the catalog but couldn't download the actual data. I therefore now have two different config in earthspy, one for the download request and the other for the catalog. Just to keep everything sorted! Do you confirm this observation @AleksMat? thank you a lot! :+1:

AleksMat commented 2 years ago

Hi @AleksMat, unfortunately an issue persists after this fix.

I believe this is a completely different issue that is not related to MissingTokenError.

I found out sh_base_url of config needs to be set to https://services.sentinel-hub.com for the download and to https://creodias.sentinel-hub.com to get the catalog before download. I was using the same config for both, and as a result could get the catalog but couldn't download the actual data. I therefore now have two different config in earthspy, one for the download request and the other for the catalog. Just to keep everything sorted! Do you confirm this observation @AleksMat? thank you a lot! :+1:

Note that in your case you are running a data fusion of 2 data collections:

With a Catalog API request you only check availability of Sentinel-3 data so https://creodias.sentinel-hub.com has to be used for that. However, a Process API request has to collect and join data from both https://creodias.sentinel-hub.com and https://services.sentinel-hub.com. In such cases the main Sentinel Hub deployment always takes a priority over other deployments. Therefore, you had to make a request to https://services.sentinel-hub.com. In case you would want to do a Process API request that would only work with Sentinel-3 collection you would have to send it to https://creodias.sentinel-hub.com.

AdrienWehrle commented 2 years ago

Great @AleksMat, this is indeed what I observed, thank you for the full explanation!! :+1: