[Data]: Updated GLO30 data

[nmt28]

Dataset Name

Copernicus GLO30 (updated)

Dataset Description

The DEM GLO30 has been updated several times. Old versions did not include certain countries, but these are included in update releases. A table of the updates and release dates is given in this table: https://spacedata.copernicus.eu/collections/copernicus-digital-elevation-model

Requestor Name/Affiliation

NATHAN THOMAS GSFC/EHU

Platform/Method/Sensor

NA

URL or DOI to Dataset Description

NA

URL to Download or Access the Data

NA

Data License

NA

Intended Science Use Case

NA

Format of the Data

NA

Approximate Size of the Data

NA

Date Needed By

No response

Additional Information

A list of the updates and release dates is given in the table here: https://spacedata.copernicus.eu/collections/copernicus-digital-elevation-model

[wildintellect]

@nmt28 I did some initial looking into this. I'm not sure that the latest product versions are in the AWS Open Data bucket. I could not find the Armenia tiles in the bucket by name.

Let us do a little more digging and reach out Singergise to find out if they plan to update the bucket. As of now, I believe the currently indexed data matches what is in the bucket, we could compare against the latest bucket list textfile to verity. cc: @jjfrench

[nmt28]

@wildintellect OK, thanks. Is the path of least resistance right now to pull what I need directly from Copernicus (if available somewhere) and upload them manually, updating the spatial index so that it uses a local location for those files?

[wildintellect]

@nmt28 I looked into this, and the data should be available from dataspace.copernicus.eu however once you login, it appears the data present is actually hosted by SentinelHub and not available as original files for download. If you go over to SentinelHub (made and operated by Sinergise) the reference in the browse appears to suggest the same AWS bucket is used which is not the most recent release of the data, also the request appear to be WCS based, and they fill any gaps with GLO90. So I haven't found the original tiles there either.

I'm going to ask the ESA MAAP group about where to find the files.

[wildintellect]

@nmt28 the files do appear to be on https://panda.copernicus.eu but require a CDS SSO account, the links given on https://ssoidp.copernicus.eu/idp/umsso20/login?faq are all dead for registering for such an account.

Example: Armenia Copernicus_DSM_10_N41_00_E043_00_DEM.tif

[wildintellect]

I found the right place to register, here are the instructions

I have yet to find any other source that has the 2023 updated files. However even though I can now get in, the Download is broken... will ask ESA tomorrow.

[wildintellect]

@nmt28 ESA said we need to use a newer interface described on

• https://documentation.dataspace.copernicus.eu/Data/ComplementaryData/Additional.html#copernicus-digital-elevation-model-cop-dem
• https://documentation.dataspace.copernicus.eu/APIs/STAC.html

I've made progress https://gist.github.com/wildintellect/1e49d36f81aab3c710687b211252fc57 on the STAC query, however it doesn't look like the missing tiles are in this catalog (not pictured in the gist since it does show shell output. I need to double check with OpenSearch instead of STAC next week.

Also a todo, test how to authenticate and retrieve a file from this system.

[nmt28]

So it looks like we're going in circles between dead links or missing data.....?

[wildintellect]

Update from ESA - the data is in progress but not available yet on the new distribution platform (check again in a week or two).

[jjfrench]

Still missing, ping ESA :)

[wildintellect]

Email sent to Cristiano, added @freitagb to the CC.

[wildintellect]

Feedback from my colleagues. The newer versions of the Copernicus-DEM are now part of the “Copernicus Contributing Missions” collection, described at Copernicus Contributing Missions Collections Description | Copernicus Data Space Ecosystem.

To search for data, one needs to search in the CCM collection and then filter by datasetFull to find the specific dataset that one needs. In STAC this is not available in server side but with OData you can use something like the following:

catalogue.dataspace.copernicus.eu/odata/v1/Products?$filter=Attributes/OData.CSC.StringAttribute/any(att:att/Name eq %27datasetFull%27 and att/OData.CSC.StringAttribute/Value eq %27COP-DEM_GLO-30-DTED%27) and OData.CSC.Intersects(area=geography%27SRID=4326;POLYGON ((42 37,42 42, 50 42, 50 37, 42 37))%27)&$expand=Attributes%27)&$expand=Attributes)

We'll look into what needs to change in the notebook.

[wildintellect]

@nmt28 update

I've confirmed the data is there: https://gist.github.com/wildintellect/1e49d36f81aab3c710687b211252fc57 see the odata.ipynb

The issue is I haven't figured out how to download the data yet. The S3 requests are blocked.

[nmt28]

Thanks @wildintellect! It's the gift that keeps on giving...

[wildintellect]

We're blocked on permissions, which seem to work for other collections but not this one. I'll contact ESA again to follow up.

[wildintellect]

Getting a token works fine

curl --location --request POST 'https://identity.dataspace.copernicus.eu/auth/realms/CDSE/protocol/openid-connect/token'   --header 'Content-Type: application/x-www-form-urlencoded'   --data-urlencode 'grant_type=password'   --data-urlencode 'username=***'   --data-urlencode 'password=***'   --data-urlencode 'client_id=cdse-public' > token.json

But getting a file does not, do we have the wrong url?

curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://catalogue.dataspace.copernicus.eu/odata/v1/Products(f8c55947-7def-431f-b88b-539c1b4d3708)/$value' --location-trusted --output /tmp/product

[wildintellect]

I need to open a ticket with the helpdesk https://helpcenter.dataspace.copernicus.eu/hc/en-gb

[wildintellect]

OpenTopography announced they have a copy of the latest version now. Here's a notebook showing how to access, no account needed. Probably performs best in the US, as I think the data is hosted at San Diego Super Computer center. https://gist.github.com/wildintellect/9485dc082744f47d915d410e97437671