Read satellite data from AWS

[remi-braun]

EOReader is able to read products stored on the cloud. The main issue is that the format of these products may be different from the one delivred elsewhere (i.e. no more SAFE format)

There is a community need for that, see #102

---

Sentinel-2

• [x] sentinel-2-l1c - from arn:aws:s3:::sentinel-s2-l1c (Sinergise) - ✔️
• [x] sentinel-2-l2a - from arn:aws:s3:::sentinel-cogs (Element84) - ✔️
• [x] sentinel-s2-l2a - from arn:aws:s3:::sentinel-s2-l2a (Sinergise) - ✔️

  Sentinel-1

• ❌ sentinel-1-grd - from arn:aws:s3:::sentinel-s1-l1c (Sinergise) - ❌ WON'T DO for now as SNAP doesn't handle AWS-stored products

  Other

• [x] landsat-c2-l1 and landsat-c2-l2 - from arn:aws:s3:::usgs-landsat/collection02/ (USGS) - ✔ ️Was already OK as their formatting are the same on cloud and on disk
• ❌ STAC data stored in one bucket pointing to another such as E84 landsat, s2-L1C catalogs : too complicated to manage right account. Maybe will be managed with #118 ?
• Umbra open data, Maxar open data, Landsat etc... (other issues)
  • Handle Umbra as a new product #114
  • Handle STAC Items as entry points #118

Then look at MPC data (delayed, as I need to register)

• MPC

Don't forget to had documentation and a notebook about Cloud-stored products

[remi-braun]

@bcerripromethee If you clone the 0.21.0 branch, could you tell me if it fits your needs ? Note that right now EOReader is not able to ingest a STAC Item, but it'll be soon I hope.

[bcerripromethee]

Alright, i'll test it ;)

I just initiate the Reader with the path of the Element84 item (such as a random Sentinel-2 product) ?

[remi-braun]

You can find some help here: https://github.com/sertit/eoreader/blob/0.21.0/docs/notebooks/aws.ipynb :)

[bcerripromethee]

I feel awkward but I can't test your upgrade because you develop your tool on the products that needs AWS credentials. I actually use the element84 links 😅.

For example, in the Element84's API, let's take this product : S2A_40VDR_20231114_0_L2A

It has your AWS URL that EOReader is now able to read : s3://sentinel-s2-l2a/tiles/40/V/DR/2023/11/14/0/

And the Element84 URL that I usually take as an input in my processes : https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/40/V/DR/2023/11/S2A_40VDR_20231114_0_L2A/

As far i know, the last one doesn't need any credentials or account.

[remi-braun]

This should work:

import tempenv
from sertit import s3
from eoreader.reader import Reader
from eoreader.bands import BLUE

with tempenv.TemporaryEnvironment({
    "AWS_S3_ENDPOINT": "s3.us-west-2.amazonaws.com",
    "AWS_SECRET_ACCESS_KEY": "",
    "AWS_ACCESS_KEY_ID": "",
}):
    with s3.temp_s3():
        path = r"s3://sentinel-cogs/sentinel-s2-l2a-cogs/40/V/DR/2023/11/S2A_40VDR_20231114_0_L2A"
        prod = Reader().open(path)
        blue = prod.load([BLUE])[BLUE]
        blue

<xarray.DataArray <SpectralBandNames.BLUE: 'BLUE'> (band: 1, y: 10980, x: 10980)>
dask.array<clip, shape=(1, 10980, 10980), dtype=float32, chunksize=(1, 1024, 1024), chunktype=numpy.ndarray>
Coordinates:
  * x            (x) float64 4e+05 4e+05 4e+05 ... 5.097e+05 5.097e+05 5.098e+05
  * y            (y) float64 7.1e+06 7.1e+06 7.1e+06 ... 6.99e+06 6.99e+06
    spatial_ref  int32 0
  * band         (band) int32 1
Attributes: (12/13)
    long_name:         BLUE
    constellation:     Sentinel-2 stored on AWS and processed by Element84
    constellation_id:  S2_E84
    product_path:      s3://sentinel-cogs/sentinel-s2-l2a-cogs/40/V/DR/2023/1...
    product_name:      S2A_MSIL2A_20231114T074151_N0509_R092_T40VDR_20231114T...
    product_filename:  S2A_40VDR_20231114_0_L2A
    ...                ...
    product_type:      MSIL2A
    acquisition_date:  20231114T074151
    condensed_name:    20231114T074151_S2_T40VDR_L2A_081855
    orbit_direction:   DESCENDING
    radiometry:        reflectance
    cloud_cover:       99.999058

Give the S3 path you find in your catalogue:

[bcerripromethee]

I tried your methodology but it doesn't seems to work, i got this error : botocore.exceptions.NoCredentialsError: Unable to locate credentials

[remi-braun]

Sorry, I updated the code hereover.

The only difference is to set "" to "AWS_SECRET_ACCESS_KEY" and "AWS_ACCESS_KEY_ID"

[bcerripromethee]

You mean like this ? ->

s2_path = r"s3://sentinel-s2-l2a/tiles/40/V/DR/2023/11/14/0/"

with tempenv.TemporaryEnvironment({"AWS_S3_ENDPOINT": "s3.eu-central-1.amazonaws.com",
                                   "AWS_SECRET_ACCESS_KEY": "",
                                   "AWS_ACCESS_KEY_ID": ""}):
    with s3.temp_s3(requester_pays=True):
        prod = Reader().open(s2_path)
        print(prod)

[remi-braun]

No, copy paste my piece of code 😉 You need to remove requester_pays=True (or set it to False)

And here you are trying to open a product from Sinergise and not E84, and for them you need credentials I think

[bcerripromethee]

Even by copy-pasting your code i still get the message botocore.exceptions.NoCredentialsError: Unable to locate credentials, it's weird

[remi-braun]

Ok, let's use only rasterio and cloudpathlib, and not sertit libraries. It will complicate a bit your code, though:

import os
from cloudpathlib import S3Client, AnyPath
import rasterio
from eoreader.reader import Reader

# Create your S3 path with cloudpathlib
aws_endpoint = "s3.us-west-2.amazonaws.com"
client = S3Client(endpoint_url=f"https://{aws_endpoint}")
client.set_as_default_client()

# Path
path = "s3://sentinel-cogs/sentinel-s2-l2a-cogs/40/V/DR/2023/11/S2A_40VDR_20231114_0_L2A"

# Create a rasterio env to enable reading on S3
with rasterio.Env(AWS_S3_ENDPOINT=aws_endpoint):
    # Open your product
    prod = Reader().open(path)
    print(prod)

[bcerripromethee]

I still get the same error message 😅

[remi-braun]

Maybe this:

import os
from cloudpathlib import S3Client, AnyPath
import rasterio
from eoreader.reader import Reader

# Create your S3 path with cloudpathlib
aws_endpoint = "s3.us-west-2.amazonaws.com"
client = S3Client(endpoint_url=f"https://{aws_endpoint}", no_sign_request=True)
client.set_as_default_client()

# Path
path = "s3://sentinel-cogs/sentinel-s2-l2a-cogs/40/V/DR/2023/11/S2A_40VDR_20231114_0_L2A"

# Create a rasterio env to enable reading on S3
with rasterio.Env(AWS_S3_ENDPOINT=aws_endpoint, AWS_NO_SIGN_REQUEST="YES"):
    # Open your product
    prod = Reader().open(path)
    print(prod)

[bcerripromethee]

Your last solution worked !!!! 🍾🍾🍾🍾

[remi-braun]

Note that with sertit[full]>=1.32.1, the following will work (if the environment variable AWS_S3_ENDPOINT is empty, if not set it with tempenv like above):

  with s3.temp_s3("s3.us-west-2.amazonaws.com", no_sign_request=True):
      path = r"s3://sentinel-cogs/sentinel-s2-l2a-cogs/40/V/DR/2023/11/S2A_40VDR_20231114_0_L2A"
      prod = Reader().open(path)
      print(prod)

eoreader.S2E84Product 'S2A_MSIL2A_20231114T074151_N0509_R092_T40VDR_20231114T081855'
Attributes:
    condensed_name: 20231114T074151_S2_T40VDR_L2A_081855
    path: s3://sentinel-cogs/sentinel-s2-l2a-cogs/40/V/DR/2023/11/S2A_40VDR_20231114_0_L2A
    constellation: Sentinel-2 stored on AWS and processed by Element84
    sensor type: Optical
    product type: MSIL2A
    default pixel size: 10.0
    default resolution: 10.0
    acquisition datetime: 2023-11-14T07:41:51
    band mapping:
        COASTAL_AEROSOL: 01
        BLUE: 02
        GREEN: 03
        RED: 04
        VEGETATION_RED_EDGE_1: 05
        VEGETATION_RED_EDGE_2: 06
        VEGETATION_RED_EDGE_3: 07
        NIR: 08
        NARROW_NIR: 8A
        WATER_VAPOUR: 09
        SWIR_1: 11
        SWIR_2: 12
    needs extraction: False
    cloud cover: 99.999058
    tile name: T40VDR

[remi-braun]

I'm closing this issue, but it still links to others (Umbra, STAC, MPC...) that are open.

[remi-braun]

0.21.0 is up today :)