Authorization problem for AWS s3

[paugalles]

I am trying to execute the download with boto3 notebook:

https://github.com/nasa/LPDAAC-Data-Resources/blob/main/python/how-tos/Earthdata_Cloud__Download_file_from_S3.ipynb

I successfuly login with my earthdata username and password and I also get the temporary credentials for aws. But when I execute client.download_file I get the following error:

what am I missing?

---------------------------------------------------------------------------
ClientError                               Traceback (most recent call last)
Cell In[31], line 3
      1 key = "HLSS30.020[/HLS.S30.T56QPM.2023001T002959.v2.0/HLS.S30.T56QPM.2023001T002959.v2.0.B03.tif](http://localhost:8888/HLS.S30.T56QPM.2023001T002959.v2.0/HLS.S30.T56QPM.2023001T002959.v2.0.B03.tif)"
      2 filename = 'temp_download_example.tif'
----> 3 client.download_file(Bucket=bucket, Key=key, Filename=filename)

File [~/miniconda3/envs/hls/lib/python3.11/site-packages/boto3/s3/inject.py:192](http://localhost:8888/home/pauravent/miniconda3/envs/hls/lib/python3.11/site-packages/boto3/s3/inject.py#line=191), in download_file(self, Bucket, Key, Filename, ExtraArgs, Callback, Config)
    157 """Download an S3 object to a file.
    158 
    159 Usage::
   (...)
    189     transfer.
    190 """
    191 with S3Transfer(self, Config) as transfer:
--> 192     return transfer.download_file(
    193         bucket=Bucket,
    194         key=Key,
    195         filename=Filename,
    196         extra_args=ExtraArgs,
    197         callback=Callback,
    198     )

File [~/miniconda3/envs/hls/lib/python3.11/site-packages/boto3/s3/transfer.py:405](http://localhost:8888/home/pauravent/miniconda3/envs/hls/lib/python3.11/site-packages/boto3/s3/transfer.py#line=404), in S3Transfer.download_file(self, bucket, key, filename, extra_args, callback)
    401 future = self._manager.download(
    402     bucket, key, filename, extra_args, subscribers
    403 )
    404 try:
--> 405     future.result()
    406 # This is for backwards compatibility where when retries are
    407 # exceeded we need to throw the same error from boto3 instead of
    408 # s3transfer's built in RetriesExceededError as current users are
    409 # catching the boto3 one instead of the s3transfer exception to do
    410 # their own retries.
    411 except S3TransferRetriesExceededError as e:

File [~/miniconda3/envs/hls/lib/python3.11/site-packages/s3transfer/futures.py:103](http://localhost:8888/home/pauravent/miniconda3/envs/hls/lib/python3.11/site-packages/s3transfer/futures.py#line=102), in TransferFuture.result(self)
     98 def result(self):
     99     try:
    100         # Usually the result() method blocks until the transfer is done,
    101         # however if a KeyboardInterrupt is raised we want want to exit
    102         # out of this and propagate the exception.
--> 103         return self._coordinator.result()
    104     except KeyboardInterrupt as e:
    105         self.cancel()

File [~/miniconda3/envs/hls/lib/python3.11/site-packages/s3transfer/futures.py:266](http://localhost:8888/home/pauravent/miniconda3/envs/hls/lib/python3.11/site-packages/s3transfer/futures.py#line=265), in TransferCoordinator.result(self)
    263 # Once done waiting, raise an exception if present or return the
    264 # final result.
    265 if self._exception:
--> 266     raise self._exception
    267 return self._result

File [~/miniconda3/envs/hls/lib/python3.11/site-packages/s3transfer/tasks.py:269](http://localhost:8888/home/pauravent/miniconda3/envs/hls/lib/python3.11/site-packages/s3transfer/tasks.py#line=268), in SubmissionTask._main(self, transfer_future, **kwargs)
    265     self._transfer_coordinator.set_status_to_running()
    267     # Call the submit method to start submitting tasks to execute the
    268     # transfer.
--> 269     self._submit(transfer_future=transfer_future, **kwargs)
    270 except BaseException as e:
    271     # If there was an exception raised during the submission of task
    272     # there is a chance that the final task that signals if a transfer
   (...)
    281 
    282     # Set the exception, that caused the process to fail.
    283     self._log_and_set_exception(e)

File [~/miniconda3/envs/hls/lib/python3.11/site-packages/s3transfer/download.py:354](http://localhost:8888/home/pauravent/miniconda3/envs/hls/lib/python3.11/site-packages/s3transfer/download.py#line=353), in DownloadSubmissionTask._submit(self, client, config, osutil, request_executor, io_executor, transfer_future, bandwidth_limiter)
    325 """
    326 :param client: The client associated with the transfer manager
    327 
   (...)
    349     downloading streams
    350 """
    351 if transfer_future.meta.size is None:
    352     # If a size was not provided figure out the size for the
    353     # user.
--> 354     response = client.head_object(
    355         Bucket=transfer_future.meta.call_args.bucket,
    356         Key=transfer_future.meta.call_args.key,
    357         **transfer_future.meta.call_args.extra_args,
    358     )
    359     transfer_future.meta.provide_transfer_size(
    360         response['ContentLength']
    361     )
    363 download_output_manager = self._get_download_output_manager_cls(
    364     transfer_future, osutil
    365 )(osutil, self._transfer_coordinator, io_executor)

File [~/miniconda3/envs/hls/lib/python3.11/site-packages/botocore/client.py:565](http://localhost:8888/home/pauravent/miniconda3/envs/hls/lib/python3.11/site-packages/botocore/client.py#line=564), in ClientCreator._create_api_method.<locals>._api_call(self, *args, **kwargs)
    561     raise TypeError(
    562         f"{py_operation_name}() only accepts keyword arguments."
    563     )
    564 # The "self" in this scope is referring to the BaseClient.
--> 565 return self._make_api_call(operation_name, kwargs)

File [~/miniconda3/envs/hls/lib/python3.11/site-packages/botocore/client.py:1021](http://localhost:8888/home/pauravent/miniconda3/envs/hls/lib/python3.11/site-packages/botocore/client.py#line=1020), in BaseClient._make_api_call(self, operation_name, api_params)
   1017     error_code = error_info.get("QueryErrorCode") or error_info.get(
   1018         "Code"
   1019     )
   1020     error_class = self.exceptions.from_code(error_code)
-> 1021     raise error_class(parsed_response, operation_name)
   1022 else:
   1023     return parsed_response

ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden

[amfriesz]

@paugalles,

Are you running the code from an environment running in AWS us-west-2?

[paugalles]

That was in my local machine. I see that I will need to run that in a aws instance, correct? Is there anyway to run it in a local machine?

[amfriesz]

Yes, you do need to be in an AWS instance in us-west-2 to run that particular script. If you are looking to do a straight download of the data from the archive, I'd recommend using the earthaccess package for Python. The repo and the documentation page have some examples to get you started.

[paugalles]

ok, thanks for the info. I will close the issue

[paugalles]

Is there any way to run this from outside an AWS instance? any way at all? Thanks for your help

[amfriesz]

Yes, you would use the HTTPS links rather than the S3 links. You can use earthaccess for that too. This resource should help > https://github.com/CU-ESIIL/HYR-SENSE/blob/main/notebooks/how_to/earthaccess_Introduction.ipynb