Permission error of accessing S3 data

[zxdawn]

I have tried to run How_to_Direct_S3_Access.ipynb and got an error for the part of opening s3 url:

# Open s3 url
fp = fs_s3.open(s3_url, mode='rb')
# Open dataset with xarray
ds = xr.open_dataset(fp) #Note this only opens the root group (reflectance)
ds

---------------------------------------------------------------------------
ClientError                               Traceback (most recent call last)
File [~/miniconda3/envs/emit_tutorials/lib/python3.9/site-packages/s3fs/core.py:113](https://file+.vscode-resource.vscode-cdn.net/Users/xinz/Documents/github/EMIT-Data-Resources/python/how-tos/~/miniconda3/envs/emit_tutorials/lib/python3.9/site-packages/s3fs/core.py:113), in _error_wrapper(func, args, kwargs, retries)
    112 try:
--> 113     return await func(*args, **kwargs)
    114 except S3_RETRYABLE_ERRORS as e:

File [~/miniconda3/envs/emit_tutorials/lib/python3.9/site-packages/aiobotocore/client.py:383](https://file+.vscode-resource.vscode-cdn.net/Users/xinz/Documents/github/EMIT-Data-Resources/python/how-tos/~/miniconda3/envs/emit_tutorials/lib/python3.9/site-packages/aiobotocore/client.py:383), in AioBaseClient._make_api_call(self, operation_name, api_params)
    382     error_class = self.exceptions.from_code(error_code)
--> 383     raise error_class(parsed_response, operation_name)
    384 else:

ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden

The above exception was the direct cause of the following exception:

PermissionError                           Traceback (most recent call last)
[/Users/xinz/Documents/github/EMIT-Data-Resources/python/how-tos/How_to_Direct_S3_Access.ipynb](https://file+.vscode-resource.vscode-cdn.net/Users/xinz/Documents/github/EMIT-Data-Resources/python/how-tos/How_to_Direct_S3_Access.ipynb) Cell 16 line 2
      [1](vscode-notebook-cell:/Users/xinz/Documents/github/EMIT-Data-Resources/python/how-tos/How_to_Direct_S3_Access.ipynb#X21sZmlsZQ%3D%3D?line=0) # Open s3 url
----> [2](vscode-notebook-cell:/Users/xinz/Documents/github/EMIT-Data-Resources/python/how-tos/How_to_Direct_S3_Access.ipynb#X21sZmlsZQ%3D%3D?line=1) fp = fs_s3.open(s3_url, mode='rb')
      [3](vscode-notebook-cell:/Users/xinz/Documents/github/EMIT-Data-Resources/python/how-tos/How_to_Direct_S3_Access.ipynb#X21sZmlsZQ%3D%3D?line=2) # Open dataset with xarray
      [4](vscode-notebook-cell:/Users/xinz/Documents/github/EMIT-Data-Resources/python/how-tos/How_to_Direct_S3_Access.ipynb#X21sZmlsZQ%3D%3D?line=3) ds = xr.open_dataset(fp) #Note this only opens the root group (reflectance)

File [~/miniconda3/envs/emit_tutorials/lib/python3.9/site-packages/fsspec/spec.py:1309](https://file+.vscode-resource.vscode-cdn.net/Users/xinz/Documents/github/EMIT-Data-Resources/python/how-tos/~/miniconda3/envs/emit_tutorials/lib/python3.9/site-packages/fsspec/spec.py:1309), in AbstractFileSystem.open(self, path, mode, block_size, cache_options, compression, **kwargs)
   1307 else:
...
    138         err = e
    139 err = translate_boto_error(err)
--> 140 raise err

PermissionError: Forbidden

[ebolch]

From the output message it looks like you're trying to access the S3 object from your local machine. Earthdata Cloud S3 buckets/objects are only accessible if a user is working from an instance within AWS us-west2. If you want to work with EMIT data locally, from another AWS region, or cloud vendor, you'll need to use the HTTPS URL to access the data asset.

There is an example of streaming data using the HTTPS in the Stream Data section of the How to Find and Access EMIT Data

Depending on your internet speed, you may want to download the data as done in the Exploring EMIT L2A Reflectance Tutorial, rather than stream EMIT data because of its size.

[zxdawn]

Thanks a lot for your reply. Because I'm interested in processing L1 data by myself, it sounds better to download it using the Earth Search web tool or script.