Add additional assets that are available.

[wildintellect]

Is your feature request related to a problem? Please describe. The original package only includes the core Data asset but not any of the other auxiliary files which are available. This includes files that some might find useful: Quicklook shaded relief for visualization, mask layers for water, xml metadata, etc...

Describe the solution you'd like Add the additional assets that are available on the same AWS open data bucket.

Describe alternatives you've considered Since the core DEM is what most users want, this can be skipped/optional for many people.

Additional context From the latest product handbook (June 2022)

[philvarner]

@gadomski seems like this can be closed?

[gadomski]

I don't see any additional assets here: https://github.com/stactools-packages/cop-dem/blob/f7e2207f610f9c5ca2fa828b9d22e5a106493b9b/src/stactools/cop_dem/stac.py#L77-L96

None in examples/ either -- so I don't think so, unless I'm misunderstanding.

[wildintellect]

If you look at the image above the additional assets exist on the AWS bucket. The current code ignores their existence. The fact that it's not in the stac.py Assets is the point the of this ticket.

$ aws s3 ls --recursive --no-sign-request s3://copernicus-dem-30m/Copernicus_DSM_COG_10_N80_00_E098_00_DEM/
2022-05-09 07:51:22        670 Copernicus_DSM_COG_10_N80_00_E098_00_DEM/AUXFILES/Copernicus_DSM_10_N80_00_E098_00_ACM.kml
2022-05-09 07:51:22      41817 Copernicus_DSM_COG_10_N80_00_E098_00_DEM/AUXFILES/Copernicus_DSM_COG_10_N80_00_E098_00_EDM.tif
2022-05-09 07:51:22      41371 Copernicus_DSM_COG_10_N80_00_E098_00_DEM/AUXFILES/Copernicus_DSM_COG_10_N80_00_E098_00_FLM.tif
2022-05-09 07:51:22     425942 Copernicus_DSM_COG_10_N80_00_E098_00_DEM/AUXFILES/Copernicus_DSM_COG_10_N80_00_E098_00_HEM.tif
2022-05-09 07:51:22      39735 Copernicus_DSM_COG_10_N80_00_E098_00_DEM/AUXFILES/Copernicus_DSM_COG_10_N80_00_E098_00_WBM.tif
2022-05-09 07:51:22      44714 Copernicus_DSM_COG_10_N80_00_E098_00_DEM/Copernicus_DSM_10_N80_00_E098_00.xml
2022-05-09 07:51:22     369514 Copernicus_DSM_COG_10_N80_00_E098_00_DEM/Copernicus_DSM_COG_10_N80_00_E098_00_DEM.tif
2022-05-09 07:51:22     117522 Copernicus_DSM_COG_10_N80_00_E098_00_DEM/INFO/eula_F.pdf
2022-05-09 07:51:22     317172 Copernicus_DSM_COG_10_N80_00_E098_00_DEM/PREVIEW/Copernicus_DSM_10_N80_00_E098_00_DEM_ABS_QL.tif
2022-05-09 07:51:22     317172 Copernicus_DSM_COG_10_N80_00_E098_00_DEM/PREVIEW/Copernicus_DSM_10_N80_00_E098_00_DEM_QL.tif
2022-05-09 07:51:22      24220 Copernicus_DSM_COG_10_N80_00_E098_00_DEM/PREVIEW/Copernicus_DSM_10_N80_00_E098_00_EDM_QL.tif
2022-05-09 07:51:22      24208 Copernicus_DSM_COG_10_N80_00_E098_00_DEM/PREVIEW/Copernicus_DSM_10_N80_00_E098_00_FLM_QL.tif
2022-05-09 07:51:22     111310 Copernicus_DSM_COG_10_N80_00_E098_00_DEM/PREVIEW/Copernicus_DSM_10_N80_00_E098_00_HEM_QL.tif
2022-05-09 07:51:22       2106 Copernicus_DSM_COG_10_N80_00_E098_00_DEM/PREVIEW/Copernicus_DSM_10_N80_00_E098_00_QL.kml
2022-05-09 07:51:22      43207 Copernicus_DSM_COG_10_N80_00_E098_00_DEM/PREVIEW/Copernicus_DSM_10_N80_00_E098_00_SRC.kml
2022-05-09 07:51:22      24222 Copernicus_DSM_COG_10_N80_00_E098_00_DEM/PREVIEW/Copernicus_DSM_10_N80_00_E098_00_WBM_QL.tif

[philvarner]

Thanks @wildintellect -- I was just asking because this issue was referenced from a merge PR, but I didn't know (or look to see) if that PR had resolved the issue.

[wildintellect]

@philvarner Ah the comment in the PR specifically was about if this needed to be solved for that PR. The decision was no, to put off adding the Assets to a later time.

[muelj12]

Hello everyone. Do you currently plan to include the above mentioned aux files as assets in stac.py?

[wildintellect]

@muelj12 no one is actively working on this. Can you explain the need for some of the other files to help us prioritize their inclusion?

[jonas-eberle]

@wildintellect I can provide some details. We work on generating a Sentinel-1 Normalized Radar Backscatter (https://sentinels.copernicus.eu/web/sentinel/sentinel-1-ard-normalised-radar-backscatter-nrb-product) product. We make use of the Copernicus 30m DEM for the terrain correction and as part of the output layers a water body mask is being generated (e.g., see a prototype implementation from ESA https://github.com/SAR-ARD/S1_NRB). We request the DEM tiles from our STAC API and would like to do so for the auxiliary layers.

We plan to conduct an implementation using stactools-cop-dem, but we have a slightly different structure of the files (our basis is the original folder structure and file naming as presented in the screenshot above. The Copernicus DEM data is now available for direct download (https://sentinels.copernicus.eu/web/sentinel/-/copernicus-dem-new-direct-data-download-access). In comparison to the data on AWS. The original structure of files has a dedicated sub-folder for the DEM and "COG" is not included in the file name. I hope we do not need to have separate packages for Copernicus DEM data hosted on AWS and hosted elsewhere with the original folder structure.

[wildintellect]

@jonas-eberle I see what you mean about the slightly different structure of the folders in the tar version of the data. I think we should be able to handle that with a command line switch to indicate the source being AWS or Copernicus.eu direct. For the record I'm going to note that the files that come in the tar are not COGs just GeoTiffs. There's only a couple of places that will need modification to handle the non-cog alternative.

It does seem like the masks in the AUXFiles are important for use cases so I'll see if I can work on it next week, but if someone wants to collaborate on that I'm open to it.

[muelj12]

@wildintellect We have seen that there is already a COP-DEM STAC collection on AWS, where the AUXFiles are integrated as additional assets. One could possibly use their tags as a guide (e.g.: aws s3 ls --no-sign-request s3://copernicus-dem-30m-stac/). Exactly, the data inside the tar are not COGs for now, but we plan to process the data into COGs so that the corresponding benefits can be used when retrieving the data. We would then follow the file naming in the post above https://github.com/stactools-packages/cop-dem/issues/6#issuecomment-1462550800

[wildintellect]

@gadomski @philvarner can you assign this ticket to me :bow:

[wildintellect]

Question: should these additional asset be optional arguments to the cli/functions for create_item? Also taking suggestions on additional tests that should be added. See the Draft PR for the current state.

[philvarner]

I don't think there needs to be an option. If someone doesn't want them for whatever reason, they can remove them.