Network file loading doesn't work on QGIS 3.14 and Mac OS 10.15.5 including GDAL loading data from S3

[alexgleith]

The use of Cloud Optimised GeoTIFFs is really important to my workflows.

I've got a clean install of MacOS without any of the old Kyngchaos builds installed and I wonder if there's something missing that QGIS is assuming will be installed.

Anyhow, the issue is that on loading a TIF (and presumably any file that would load over the network) there's an error:

Invalid Layer : GDAL provider Cannot open GDAL dataset /vsis3/deafrica-data/ancillary/dem/srtm_africa.tif: CURL error: error setting certificate verify locations: CAfile: /usr/local/etc/openssl/cert.pem CApath: /usr/local/etc/openssl/certs Raster layer Provider is not valid (provider: gdal, URI: /vsis3/deafrica-data/ancillary/dem/srtm_africa.tif

If i try to load the above layer directly with Rio, it works fine, see:

rio info /vsis3/deafrica-data/ancillary/dem/srtm_africa.tif 

{"blockxsize": 512, "blockysize": 512, "bounds": [-32.00013888888889, -46.00013888889551, 61.00013888889551, 47.00013888888889], "colorinterp": ["gray"], "compress": "deflate", "count": 1, "crs": "EPSG:4326", "descriptions": [null], "driver": "GTiff", "dtype": "int16", "height": 334801, "indexes": [1], "interleave": "band", "lnglat": [14.500000000003311, 0.49999999999668887], "mask_flags": [["nodata"]], "nodata": -32768.0, "res": [0.00027777777777779756, 0.00027777777777779756], "shape": [334801, 334801], "tiled": true, "transform": [0.00027777777777779756, 0.0, -32.00013888888889, 0.0, -0.00027777777777779756, 47.00013888888889, 0.0, 0.0, 1.0], "units": [null], "width": 334801}

QGIS is 3.14.0-Pi It has GDAL 2.4.1 MacOS is 10.15.5

[gioman]

@alexgleith gdailnfo works?

[AndrewAnnex]

@alexgleith it should be possibly to override where gdal looks for the CA files using environment variables: https://gdal.org/user/virtual_file_systems.html but this project should be update to point to the files it distributes.. but maybe your error is specific to the aws support in the vs driver?

[alexgleith]

Yeah, gdalinfo works: gdalinfo /vsis3/deafrica-data/ancillary/dem/srtm_africa.tif

Driver: GTiff/GeoTIFF Files: /vsis3/deafrica-data/ancillary/dem/srtm_africa.tif /vsis3/deafrica-data/ancillary/dem/srtm_africa.tif.aux.xml Size is 334801, 334801 Coordinate System is: GEOGCS["WGS 84", DATUM["WGS_1984", SPHEROID["WGS 84",6378137,298.257223563, AUTHORITY["EPSG","7030"]], AUTHORITY["EPSG","6326"]], PRIMEM["Greenwich",0], UNIT["degree",0.0174532925199433], AUTHORITY["EPSG","4326"]] Origin = (-32.000138888888891,47.000138888888891) Pixel Size = (0.000277777777778,-0.000277777777778) Metadata: AREA_OR_POINT=Area OVR_RESAMPLING_ALG=NEAREST Image Structure Metadata: COMPRESSION=DEFLATE INTERLEAVE=BAND

[alexgleith]

Also, the error is NOT specific to the S3 implementation.

If I go to raster and add a HTTP(S) source, and use the type HTTP/HTTPS/FTP using this URL: https://deafrica-data.s3-us-west-2.amazonaws.com/ancillary/dem/srtm_africa.tif then it also fails with a CURL error.

Invalid Layer : GDAL provider Cannot open GDAL dataset /vsicurl/https://deafrica-data.s3-us-west-2.amazonaws.com/ancillary/dem/srtm_africa.tif: CURL error: error setting certificate verify locations: CAfile: /usr/local/etc/openssl/cert.pem CApath: /usr/local/etc/openssl/certs Raster layer Provider is not valid (provider: gdal, URI: /vsicurl/https://deafrica-data.s3-us-west-2.amazonaws.com/ancillary/dem/srtm_africa.tif

[alexgleith]

Plus, if I try system Curl (installed with Brew) then it works fine, see: curl --head https://deafrica-data.s3-us-west-2.amazonaws.com/ancillary/dem/srtm_africa.tif

HTTP/1.1 200 OK x-amz-id-2: /dYtR2yIXyJloZi8Iu2/uDB5wEcw4Clw5zw3VPBQ19rtspEQ87J0eNnX3nB4adICup/f6h2hDqs= x-amz-request-id: 1BC5F39AB1E3E451 Date: Mon, 22 Jun 2020 21:10:33 GMT Last-Modified: Mon, 16 Sep 2019 06:24:57 GMT ETag: "fdacf338f90fe685a78d7bb959f1718c-4900" x-amz-version-id: null Accept-Ranges: bytes Content-Type: image/tiff Content-Length: 41103318874 Server: AmazonS3

[alexgleith]

Is there anything I can do to bump this up? I'm still having issues.

[adamsteer]

heya all I'm able to load https://deafrica-data.s3-us-west-2.amazonaws.com/ancillary/dem/srtm_africa.tif in QGIS 3.14 on vanilla MacOS Catalina. I wonder if the issue can be narrowed to 'how does QGIS interact with curl provided by MacOS' vs 'curl provided by homebrew'?

[alexgleith]

Did you have any other QGIS' installed, @adamsteer?

I have only ever installed QGIS with the new packages on this system, and never with KyngChaos builds. I wondered if that had something to do with it...

[alexgleith]

Also, I think my curl is installed with MacOS actually:

which curl /bin/curl

[PeterPetrik]

duplicate of #32