MathewNWSH
commented
1 month ago Hey @j08lue,
here is how I do it in CDSE:
os.environ['GDAL_HTTP_TCP_KEEPALIVE'] = "YES" os.environ['AWS_S3_ENDPOINT'] = "eodata.dataspace.copernicus.eu" os.environ['AWS_ACCESS_KEY_ID'] = "" os.environ['AWS_SECRET_ACCESS_KEY'] = "" os.environ['AWS_HTTPS'] = "YES" os.environ['AWS_VIRTUAL_HOSTING'] = "FALSE" os.environ['GDAL_HTTP_UNSAFESSL'] = "YES"
and this is how I do it from my machine on waw3-2 (should also work on fra1-2 cloud):
export AWS_S3_ENDPOINT=eodata.cloudferro.com
export AWS_ACCESS_KEY_ID=****
export AWS_SECRET_ACCESS_KEY=****
export CPL_CURL_VERBOSE=YES
export AWS_HTTPS=YES
export AWS_VIRTUAL_HOSTING=FALSE
export GDAL_HTTP_TCP_KEEPALIVE=YES
export CPL_VSIL_CURL_CHUNK_SIZE=98304
export GDAL_INGESTED_BYTES_AT_OPEN=16384
export GDAL_HTTP_UNSAFESSL=YES
export CPL_DEBUG=OFF
export GDAL_NUM_THREADS=-1
export PROJ_DEBUG=OFFSome docs: S3 key gen and configuration of s3 on CDSE: https://documentation.dataspace.copernicus.eu/APIs/S3.html
Creodias (how to extract credentials and endpoint out of VM): https://creodias.docs.cloudferro.com/en/latest/eodata/How-to-get-credentials-used-for-accessing-EODATA-on-a-cloud-VM-on-Creodias.html#waw3-2-and-fra1-2-clouds-using-custom-and-default-linux-vms-executing-curl
And some of my code how I connect to our pgstac.demo.cloudferro.com using stackstac:
os.environ['GDAL_HTTP_TCP_KEEPALIVE'] = "YES"
os.environ['AWS_S3_ENDPOINT'] = "eodata.dataspace.copernicus.eu"
os.environ['AWS_ACCESS_KEY_ID'] = ""
os.environ['AWS_SECRET_ACCESS_KEY'] = ""
os.environ['AWS_HTTPS'] = "YES"
os.environ['AWS_VIRTUAL_HOSTING'] = "FALSE"
os.environ['GDAL_HTTP_UNSAFESSL'] = "YES"
lon, lat = 14, 50
# Main endpoint for STAC
URL='https://pgstac.demo.cloudferro.com/'
catalog = pystac_client.Client.open(URL)
catalog.add_conforms_to("ITEM_SEARCH")
items = catalog.search(
collections=['sentinel-2-l2a'],
intersects=dict(type="Point", coordinates=[lon, lat]),
datetime="2024-01-01/2024-02-01",
query = {"eo:cloud_cover":{"lte":50}}
).item_collection()
for x in items:
for k, asset in list(x.assets.items()):
if "alternate" in asset.extra_fields:
asset.href = asset.extra_fields["alternate"]["s3"]["href"].replace('s3://', '/vsis3/')
if '60m' not in k:
del x.assets[k]
stack = stackstac.stack(items = items, resolution = (60, 60), bounds_latlon = (14.254, 50.014, 14.587, 50.133), chunksize = 98304, gdal_env=stackstac.DEFAULT_GDAL_ENV.updated(
{'GDAL_NUM_THREADS': -1,
'GDAL_HTTP_UNSAFESSL': 'YES',
'GDAL_HTTP_TCP_KEEPALIVE': 'YES',
'AWS_VIRTUAL_HOSTING': 'FALSE',
'AWS_HTTPS': 'YES',
}),)I hope this will help to get the idea of how to generate and pass env variables to raster api gdal backend.
gitguardian[bot]
This adds an environment variable to the raster API settings that should tell it to resolve
s3://URLs to CloudFerro S3.Addresses