SSL Error when trying to download data with force-level1-csd

[bountrisv]

Hello,

the force-leve1-csd tool fails to download some requested data from a server due to the server not having valid SSL certificates.

[2021-10-08 12:13:59,233] {pod_launcher.py:136} INFO - ERROR 1: server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
[2021-10-08 12:13:59,234] {pod_launcher.py:136} INFO - ERROR 1: Error returned by server : server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none (60)
[2021-10-08 12:13:59,235] {pod_launcher.py:136} INFO - FAILURE:
[2021-10-08 12:13:59,236] {pod_launcher.py:136} INFO - Unable to open datasource `WFS:http://ows.geo.hu-berlin.de/cgi-bin/qgis_mapserv.fcgi?MAP=/owsprojects/grids.qgs&SERVICE=WFS&REQUEST=GetCapabilities&typename=landsat&bbox=' with the following drivers.
[2021-10-08 12:13:59,236] {pod_launcher.py:136} INFO -   -> `PCIDSK'
.
.
.

Command that causes it (omitting daterange and path arguments as they seem to be irrelevant):

force-level1-csd -s LT04,LT05,LE07,S2A -d {daterange} -c 0,70 {image_metadata_folderpath} {image_folderpath} {queue_filepath} {aoi_filepath}

FORCE ver: 3.6.5

The server seems to be owned by Humbold University of Berlin. Who would be the right person to contact for SSL certificate renewal here?

Thanks!

[davidfrantz]

Hi @bountrisv this would be @jakimowb Cheers, David

[bountrisv]

Thanks, I'll close the issue when it's up and running again. Hi @jakimowb! Would you be able to check the SSL error here?

[ernstste]

Just in case you need the data asap: You can manually select which Landsat footprints you would like to include or define a bounding box to work around the issue.

[bountrisv]

Thanks for the pointer @ernstste! Fortunately, this is not the case.

[jakimowb]

I updated the CA certtification. @bountrisv does it work now?

[bountrisv]

Hey @jakimowb, thanks for having a look at it. Unfortunately, the error is not fixed, and the error message seems to remain unchanged.

[2021-10-09 11:02:41,178] {pod_launcher.py:149} INFO - Extracting compressed Landsat metadata catalogue...
[2021-10-09 11:03:02,187] {pod_launcher.py:149} INFO - 
[2021-10-09 11:03:02,188] {pod_launcher.py:149} INFO - Downloading Sentinel-2 metadata catalogue...
[2021-10-09 11:04:19,629] {pod_launcher.py:149} INFO - Extracting compressed Sentinel-2 metadata catalogue...
[2021-10-09 11:06:42,756] {pod_launcher.py:149} INFO - 
[2021-10-09 11:06:42,756] {pod_launcher.py:149} INFO - Done. You can run this script without option -u to download data now.
[2021-10-09 11:06:42,756] {pod_launcher.py:149} INFO - 
[2021-10-09 11:06:43,136] {pod_launcher.py:149} INFO - 
[2021-10-09 11:06:43,136] {pod_launcher.py:149} INFO - Landsat and Sentinel-2 data requested.
[2021-10-09 11:06:43,136] {pod_launcher.py:149} INFO - Landsat data will be queried and downloaded first.
[2021-10-09 11:06:43,144] {pod_launcher.py:149} INFO - 
[2021-10-09 11:06:43,144] {pod_launcher.py:149} INFO - Searching for footprints / tiles intersecting with geometries of AOI shapefile...
[2021-10-09 11:06:43,405] {pod_launcher.py:149} INFO - Warning 1: One or several characters couldn't be converted correctly from UTF-8 to ISO-8859-1.  This warning will not be emitted anymore.
[2021-10-09 11:06:43,706] {pod_launcher.py:149} INFO - ERROR 1: server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
[2021-10-09 11:06:43,706] {pod_launcher.py:149} INFO - ERROR 1: Error returned by server : server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none (60)
[2021-10-09 11:06:43,707] {pod_launcher.py:149} INFO - FAILURE:
[2021-10-09 11:06:43,707] {pod_launcher.py:149} INFO - Unable to open datasource `WFS:http://ows.geo.hu-berlin.de/cgi-bin/qgis_mapserv.fcgi?MAP=/owsprojects/grids.qgs&SERVICE=WFS&REQUEST=GetCapabilities&typename=landsat&bbox=' with the following drivers.
[2021-10-09 11:06:43,710] {pod_launcher.py:149} INFO -   -> `PCIDSK'
[2021-10-09 11:06:43,711] {pod_launcher.py:149} INFO -   -> `netCDF'
[2021-10-09 11:06:43,711] {pod_launcher.py:149} INFO -   -> `JP2OpenJPEG'
[2021-10-09 11:06:43,711] {pod_launcher.py:149} INFO -   -> `PDF'
[2021-10-09 11:06:43,712] {pod_launcher.py:149} INFO -   -> `ESRI Shapefile'
[2021-10-09 11:06:43,712] {pod_launcher.py:149} INFO -   -> `MapInfo File'
[2021-10-09 11:06:43,712] {pod_launcher.py:149} INFO -   -> `UK .NTF'
[2021-10-09 11:06:43,712] {pod_launcher.py:149} INFO -   -> `OGR_SDTS'
[2021-10-09 11:06:43,712] {pod_launcher.py:149} INFO -   -> `S57'
[2021-10-09 11:06:43,713] {pod_launcher.py:149} INFO -   -> `DGN'
[2021-10-09 11:06:43,713] {pod_launcher.py:149} INFO -   -> `OGR_VRT'
[2021-10-09 11:06:43,713] {pod_launcher.py:149} INFO -   -> `REC'
[2021-10-09 11:06:43,713] {pod_launcher.py:149} INFO -   -> `Memory'
[2021-10-09 11:06:43,713] {pod_launcher.py:149} INFO -   -> `BNA'
[2021-10-09 11:06:43,713] {pod_launcher.py:149} INFO -   -> `CSV'
[2021-10-09 11:06:43,713] {pod_launcher.py:149} INFO -   -> `NAS'
[2021-10-09 11:06:43,714] {pod_launcher.py:149} INFO -   -> `GML'
[2021-10-09 11:06:43,714] {pod_launcher.py:149} INFO -   -> `GPX'
[2021-10-09 11:06:43,714] {pod_launcher.py:149} INFO -   -> `LIBKML'
[2021-10-09 11:06:43,714] {pod_launcher.py:149} INFO -   -> `KML'
[2021-10-09 11:06:43,714] {pod_launcher.py:149} INFO -   -> `GeoJSON'
[2021-10-09 11:06:43,714] {pod_launcher.py:149} INFO -   -> `Interlis 1'
[2021-10-09 11:06:43,714] {pod_launcher.py:149} INFO -   -> `Interlis 2'
[2021-10-09 11:06:43,714] {pod_launcher.py:149} INFO -   -> `OGR_GMT'
[2021-10-09 11:06:43,715] {pod_launcher.py:149} INFO -   -> `GPKG'
[2021-10-09 11:06:43,715] {pod_launcher.py:149} INFO -   -> `SQLite'
[2021-10-09 11:06:43,715] {pod_launcher.py:149} INFO -   -> `OGR_DODS'
[2021-10-09 11:06:43,715] {pod_launcher.py:149} INFO -   -> `ODBC'
[2021-10-09 11:06:43,715] {pod_launcher.py:149} INFO -   -> `WAsP'
[2021-10-09 11:06:43,715] {pod_launcher.py:149} INFO -   -> `PGeo'
[2021-10-09 11:06:43,715] {pod_launcher.py:149} INFO -   -> `MSSQLSpatial'
[2021-10-09 11:06:43,716] {pod_launcher.py:149} INFO -   -> `OGR_OGDI'
[2021-10-09 11:06:43,716] {pod_launcher.py:149} INFO -   -> `PostgreSQL'
[2021-10-09 11:06:43,716] {pod_launcher.py:149} INFO -   -> `MySQL'
[2021-10-09 11:06:43,716] {pod_launcher.py:149} INFO -   -> `OpenFileGDB'
[2021-10-09 11:06:43,716] {pod_launcher.py:149} INFO -   -> `XPlane'
[2021-10-09 11:06:43,716] {pod_launcher.py:149} INFO -   -> `DXF'
[2021-10-09 11:06:43,716] {pod_launcher.py:149} INFO -   -> `CAD'
[2021-10-09 11:06:43,716] {pod_launcher.py:149} INFO -   -> `Geoconcept'
[2021-10-09 11:06:43,717] {pod_launcher.py:149} INFO -   -> `GeoRSS'
[2021-10-09 11:06:43,717] {pod_launcher.py:149} INFO -   -> `GPSTrackMaker'
[2021-10-09 11:06:43,717] {pod_launcher.py:149} INFO -   -> `VFK'
[2021-10-09 11:06:43,717] {pod_launcher.py:149} INFO -   -> `PGDUMP'
[2021-10-09 11:06:43,717] {pod_launcher.py:149} INFO -   -> `OSM'
[2021-10-09 11:06:43,717] {pod_launcher.py:149} INFO -   -> `GPSBabel'
[2021-10-09 11:06:43,717] {pod_launcher.py:149} INFO -   -> `SUA'
[2021-10-09 11:06:43,717] {pod_launcher.py:149} INFO -   -> `OpenAir'
[2021-10-09 11:06:43,718] {pod_launcher.py:149} INFO -   -> `OGR_PDS'
[2021-10-09 11:06:43,718] {pod_launcher.py:149} INFO -   -> `WFS'
[2021-10-09 11:06:43,718] {pod_launcher.py:149} INFO -   -> `SOSI'
[2021-10-09 11:06:43,718] {pod_launcher.py:149} INFO -   -> `HTF'
[2021-10-09 11:06:43,718] {pod_launcher.py:149} INFO -   -> `AeronavFAA'
[2021-10-09 11:06:43,718] {pod_launcher.py:149} INFO -   -> `Geomedia'
[2021-10-09 11:06:43,718] {pod_launcher.py:149} INFO -   -> `EDIGEO'
[2021-10-09 11:06:43,718] {pod_launcher.py:149} INFO -   -> `GFT'
[2021-10-09 11:06:43,719] {pod_launcher.py:149} INFO -   -> `SVG'
[2021-10-09 11:06:43,719] {pod_launcher.py:149} INFO -   -> `CouchDB'
[2021-10-09 11:06:43,719] {pod_launcher.py:149} INFO -   -> `Cloudant'
[2021-10-09 11:06:43,719] {pod_launcher.py:149} INFO -   -> `Idrisi'
[2021-10-09 11:06:43,719] {pod_launcher.py:149} INFO -   -> `ARCGEN'
[2021-10-09 11:06:43,719] {pod_launcher.py:149} INFO -   -> `SEGUKOOA'
[2021-10-09 11:06:43,720] {pod_launcher.py:149} INFO -   -> `SEGY'
[2021-10-09 11:06:43,720] {pod_launcher.py:149} INFO -   -> `XLS'
[2021-10-09 11:06:43,720] {pod_launcher.py:149} INFO -   -> `ODS'
[2021-10-09 11:06:43,720] {pod_launcher.py:149} INFO -   -> `XLSX'
[2021-10-09 11:06:43,720] {pod_launcher.py:149} INFO -   -> `ElasticSearch'
[2021-10-09 11:06:43,720] {pod_launcher.py:149} INFO -   -> `Walk'
[2021-10-09 11:06:43,720] {pod_launcher.py:149} INFO -   -> `Carto'
[2021-10-09 11:06:43,721] {pod_launcher.py:149} INFO -   -> `AmigoCloud'
[2021-10-09 11:06:43,721] {pod_launcher.py:149} INFO -   -> `SXF'
[2021-10-09 11:06:43,721] {pod_launcher.py:149} INFO -   -> `Selafin'
[2021-10-09 11:06:43,721] {pod_launcher.py:149} INFO -   -> `JML'
[2021-10-09 11:06:43,721] {pod_launcher.py:149} INFO -   -> `PLSCENES'
[2021-10-09 11:06:43,721] {pod_launcher.py:149} INFO -   -> `CSW'
[2021-10-09 11:06:43,721] {pod_launcher.py:149} INFO -   -> `VDV'
[2021-10-09 11:06:43,721] {pod_launcher.py:149} INFO -   -> `GMLAS'
[2021-10-09 11:06:43,722] {pod_launcher.py:149} INFO -   -> `TIGER'
[2021-10-09 11:06:43,722] {pod_launcher.py:149} INFO -   -> `AVCBin'
[2021-10-09 11:06:43,722] {pod_launcher.py:149} INFO -   -> `AVCE00'
[2021-10-09 11:06:43,722] {pod_launcher.py:149} INFO -   -> `HTTP'```

[jakimowb]

Might take some time to fix this. However, from home and HU VPN I can access the requested WFS to load the requested Landsat / Sentinel-2 tile geometries without errors (using QGIS or gdal/ogr). How does FORCE open it?

WFS 1.1: https://ows.geo.hu-berlin.de/cgi-bin/qgis_mapserv.fcgi?MAP=/owsprojects/eo-grids/eo-grids.qgs&service=WFS&request=GetCapabilities&

WFS 3.0: https://ows.geo.hu-berlin.de/cgi-bin/qgis_mapserv.fcgi/wfs3/?MAP=/owsprojects/eo-grids/eo-grids.qgs

ogrinfo "WFS:https://ows.geo.hu-berlin.de/cgi-bin/qgis_mapserv.fcgi?MAP=/owsprojects/eo-grids/eo-grids.qgs&service=WFS&request=GetCapabilities&"
INFO: Open of `WFS:https://ows.geo.hu-berlin.de/cgi-bin/qgis_mapserv.fcgi?MAP=/owsprojects/eo-grids/eo-grids.qgs&service=WFS&request=GetCapabilities&'
      using driver `WFS' successful.
Metadata:
  ABSTRACT=A QGIS Server project to provide grids and tiles like Landsat WRS-2 footprints or Sentinel-2 tiles.
  PROVIDER_NAME=Humboldt-Universität zu Berlin
  TITLE=EO-Grids
1: equi7_land (title: Equi7 (land)) (Polygon)
2: equi7_tile_t1 (title: Equi7 (T1)) (Polygon)
3: equi7_tile_t3 (title: Equi7 (T3)) (Polygon)
4: equi7_tile_t6 (title: Equi7 (T6)) (Polygon)
5: equi7_zone (title: Equi7 (zones)) (Polygon)
6: glance_land (title: GLANCE (land)) (Polygon)
7: glance_tile (title: GLANCE) (Polygon)
8: glance_zone (title: GLANCE (zones)) (Polygon)
9: modis (title: MODIS) (Polygon)
10: sen2 (title: Sentinel 2) (Polygon)
11: sen2orbits (title: Sentinel 2 (descending orbits)) (Polygon)
12: wrs1 (title: WRS-1) (Polygon)
13: wrs1a (title: WRS-1 (ascending)) (Polygon)
14: wrs2 (title: WRS-2) (Polygon)
15: wrs2a (title: WRS-2 (ascending)) (Polygon)

[ernstste]

FORCE uses ogr2ogr. I can't reproduce the error either. The WFS works without issues using FORCE and QGIS here.

[bountrisv]

I tried the force-level1-csd command on two different cluster setups to avoid any possible caching without success. One thing I forgot to note is that I am on FORCE version 3.6.5. Could this be the culprit here?

[ernstste]

That shouldn't make a difference. I'm assuming that you get the same error when running the following command? ogrinfo WFS:http://ows.geo.hu-berlin.de/cgi-bin/qgis_mapserv.fcgi?MAP=/owsprojects/grids.qgs&SERVICE=WFS&REQUEST=GetCapabilities

[bountrisv]

Yep:

ERROR 1: Error returned by server : server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none (60)
ERROR 1: server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
ERROR 1: Error returned by server : server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none (60)
FAILURE:
Unable to open datasource `WFS:http://ows.geo.hu-berlin.de/cgi-bin/qgis_mapserv.fcgi?MAP=/owsprojects/grids.qgs' with the following drivers.
  -> PCIDSK
.
.
.
  -> AVCE00
  -> HTTP

[davidfrantz]

Hi all,

I am able to reproduce this.

When running the GetCapabilities command through the FORCE Docker container, I am seeing the error message. When running through the same-version GDAL container (3.0.4, and all releases up to 3.3.2), I am seeing the error message. When running through the latest GDAL container, it works, however... This is version 3.4.0.

However, the Docker image is only 3 days old, and this release is not even tagged and described yet. Thus I am not sure what changed... https://github.com/OSGeo/gdal/releases https://hub.docker.com/r/osgeo/gdal/tags?page=1

I will try to update the image, though I am inclined to wait for the official release of the GDAL/OGR version history to avoid taking in an unstable GDAL version.

Cheers, David

[ernstste]

So there is an problem with GDAL running from within a docker container. The issue does not occur when running GDAL directly from bash (tested on 2.2.3, 3.0.4, 3.1.4, 3.3.1 just now).

If v3.4 resolves the issue - great! But I share your concerns about updating GDAL before doing some testing or at least having a proper changelog. In the meantime, the trouble here can be avoided with a simple workaround: use a tile list or bounding box instead of a vector file as AOI (or run force-level1-csd directly from the shell).

Edit: @bountrisv @steiner-sve could you please confirm that you're both facing the issue while using FORCE within docker?

[bountrisv]

@ernstste Confirming, all my tries have been through Docker containers.

[steiner-sve]

`Docker version / FORCE v. 3.6.3 C:\Users\Nordicus>docker run -it -v G:\DockerTest:/opt/data --env BOTO_CONFIG=/opt/data/credentials/.boto davidfrantz/force force-level1-csd -nkc 0,20 -d 20210827,20210930 -s S2A,S2B /opt/data/odc/meta/ /opt/data/odc/uploaded-data/ /opt/data/odc/level1-sentinel-pannonia_new.pool /opt/data/odc/pannonia/MISC/lsth_aoi.gpkg

Searching for footprints / tiles intersecting with geometries of AOI shapefile... ERROR 1: server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none ERROR 1: Error returned by server : server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none (60) FAILURE: Unable to open datasource 'WFS:http://ows.geo.hu-berlin.de/cgi-bin/qgis_mapserv.fcgi?MAP=/owsprojects/grids.qgs&SERVICE=WFS&REQUEST=GetCapabilities&typename=sentinel2&bbox=' with the following drivers. --etc`

The script above create a correct footprint ,reprojected epsg3006-->epsg4326 and store it in a local map [ l1csd-temp_2021-10-10T11-15-55 ], but no response från your WFS.

####################### The script below is working in the same workaround : docker run -it -v G:\DockerTest:/opt/data --env BOTO_CONFIG=/opt/data/credentials/.boto davidfrantz/force force-level1-csd -nkc 0,10 -d 20200827,20200930 -s S2A,S2B /opt/data/odc/meta/ /opt/data/odc/uploaded-data/ /opt/data/odc/level1-sentinel-pannonia_new.pool T33VWC,T33VXC,T33VWD,T33VXD,T33VWE,T33VXE

Greetings, J.Steiner

[ernstste]

Thanks! Adding -v /etc/ssl/certs:/etc/ssl/certs when starting the container should fix it. Can you confirm?

[davidfrantz]

Works for me. Glad to hear that this is not related to GDAL. Do you know why this mountpoint is necessary, all of a sudden?

[ernstste]

Well, is it 'all of a sudden'? I haven't used force-level1-csd with a shapefile/geopackage AOI in a docker container so far. Take my words with a grain of salt as I don't know how the certificates are handled, but from my understanding this is what happens: The handshake between server and client requires the client to have the certificates available. The docker container is lacking the certificates (or they are installed in a different location?), so the server refuses to accept the connection.

You said it was working fine with the latest GDAL version. Will be interesting to see the changelog.

[davidfrantz]

I have used it in Docker containers previously. Although, on second thought, I am not sure if this was with a vector geometry or with tile IDs... But anyways, it works with the latest GDAL Docker without mounting... So this is likely more related to the Docker build than to GDAL itself... They changed some things in the past days regarding the Docker build, but idk..

It is good though that we might have a solution here.

Can you confirm @bountrisv ? @steiner-sve , as you are running on Windows (or is it the Subsystem for Linux?), it would also be good if you could confirm whether this (with some adjustments I assume) works for you, too?

Cheers, David

[ernstste]

Thanks David. I just briefly checked their dockerfile and they do seem to set up ca-certificates as a requirement for PROJ there. However, this was also the case in earlier versions...

[bountrisv]

On the "all of a sudden" part I can confirm the setup with docker was working normally about 3 or 4 weeks ago.

On the proposed solution, I am not sure how it applies in my case. To the best of my understanding, you are running the docker image locally and adding -v /etc/ssl/certs:/etc/ssl/certs when running it. This makes visible your local certs folder to the FORCE container. In a cluster setup, I would need to upload this folder and mount it to my container. But do I upload my local /etc/ssl/certs folder? Or only the ca-certificates.crt that must be the responsible one for getting things to work?

Please confirm that my understanding of the proposed solution is not totally off before I try it. :D

Thanks

[davidfrantz]

I did some wild googling, and it seems that this is a general issue at the moment. Probably, the difference is, that the latest GDAL image is simply new.

https://superuser.com/questions/1679204/curl-on-ubuntu-14-all-lets-encrypt-certificates-are-expired-error-60/1679205#1679205

[ernstste]

@davidfrantz When interpreting the error message I assume that this is about missing certificates rather than expired ones. I may be wrong though.

@bountrisv Mounting a folder that only contains the ca-certificates.crt worked in my case. You said it worked fine before - are you 100% sure that you used a shapefile/geopackage as AOI back then rather than a bounding box or tile list?

[vincentschut]

I might be wrong, but I think this might be related to the DST Root CA X3 Certificate expiration on September 30th. See e.g. https://www.stephenwagner.com/2021/09/30/sophos-dst-root-ca-x3-expiration-problems-fix/ . Lot's of sites had trouble with that. Probably FORCE's parent docker image has not updated it's certificates, and thus fails to authenticate properly. When you -v your own certificate folder into the docker, it will use your own - probably updated - cerificates, and succeed.

[ernstste]

Thanks @vincentschut and @davidfrantz for the links. I just had a look again. Apparently the certificates on the machines I'm using where updated. The ISRG Root X1 certificate was already included in my ca-certificates.crt. Only providing this one certificate to the docker container solves the issue, so it's most likely that the scenario you're describing is what's going on: the expired certificate in the older docker containers is at fault.

[davidfrantz]

I updated the Docker base image and the FORCE images (and released v. 3.7.1)

It works on my end again (without -v)

Can you update your Docker images and confirm?

Docker tags that should work again:

• latest
• latest-debug
• dev
• dev-debug
• 3.7.1
• 3.7.1-debug

Cheers, David

[bountrisv]

@ernstste Yes, I have always been using the gpkg file.

@davidfrantz Thanks David! Unfortunately I'm stuck using the old images for now (3.6.5), but here is a workaround for the old images, using the links everybody provided.

One can manually add the required certificates by adding this to the image entry point:

sed -i -e 's=^mozilla/DST_Root_CA_X3.crt=!mozilla/DST_ROOT_CA_X3.crt=' /etc/ca-certificates.conf && \
cd /usr/local/share/ca-certificates && \
wget https://letsencrypt.org/certs/isrgrootx1.pem && \
wget https://letsencrypt.org/certs/isrg-root-x2.pem && \
update-ca-certificates --fresh

The above script comments out the old certificate, downloads the new ones and updates them from scratch. After that, the downloading should proceed as usual.

[ernstste]

Tested and working on dev and latest, thanks for updating the image @davidfrantz!

Thanks to everyone bringing this up and helping to narrow down the cause of the issue.

@bountrisv feel free close the issue if the new image works okay for you.

[bountrisv]

Yes, I will. Thank you all!