Support ZSTD compression in rasters

[latot]

Feature description

Hi, actually GDAL has several ways to compress rasters:

https://gdal.org/drivers/raster/gtiff.html

I testes save rasters with ZSTD compression, sadly we can't open that rasters with QGIS, I don't know if is supported or not other compression ways.

QGIS 3.26.3 Gentoo 64 GDAL 3.5.3

Thx!

Additional context

No response

[agiudiceandrea]

@latot could you please provide a sample ZSTD compressed raster layer with wich you have the issue? Could you please test using a currently supported version of QGIS (3.28.1 - 3.22.13) with a current version of GDAL (3.6.0). If they are not available for you system, you can also try the Flatpak versions.

[nicogodet]

ZSTD compression read/write works on Windows with OSGeo4W so this feature request is not a FR. Could come from GDAL used on your system. "ZSTD is available since GDAL 2.3 when using internal libtiff and if GDAL built against libzstd >=1.0, or if built against external libtiff with zstd support."

[latot]

Its weird, I probably can test this the next monday, but the raster I tested was generated with GDAL (coded in python3), explicitly with ZSTD compression......

[latot]

Oks, system update today, now with QGIS 3.26.3 and GDAL 3.6.1, in any case is the same as before, the next thing works:

This works

import numpy as np
import rasterio
a = rasterio.open("test.tiff", "w", compress="DEFLATE", height = 10, width = 10, count = 1, dtype = "uint8")
a.write(np.ones((10, 10)), 1)
a.close()

This does not works:

import numpy as np
import rasterio
a = rasterio.open("test.tiff", "w", compress="ZSTD", height = 10, width = 10, count = 1, dtype = "uint8")
a.write(np.ones((10, 10)), 1)
a.close()

D:

I can correcyly generate the files, but when we open them with QGIS, the ZSTD one does not works.

Tested with flatpak too, fails!, says the codec can't be found..., pretty weird, without codec the code above should fails, at least if QGIS uses GDAL, the code to work should be the same.

Thx!

[agiudiceandrea]

@latot I've created the raster layer using the provided Python scripts with QGIS 3.28.1 (GDAL 3.6.0) on Windows (OSGeo4W). They are correctly imported in QGIS 3.28.1 (GDAL 3.6.0) and in QGIS 3.22.10 (GDAL 3.5.1) and QGIS 3.16.5 (GDAL 3.1.4) on Windows (OSGeo4W). test.zip

Extent  0.0000000000000000,-10.0000000000000000 : 10.0000000000000000,0.0000000000000000
Width   10
Height  10
Data type   Byte - Eight bit unsigned integer
GDAL Driver Description GTiff
GDAL Driver Metadata    GeoTIFF
Dataset Description test_deflate.tiff
Compression DEFLATE
Band 1  
STATISTICS_MAXIMUM=1
STATISTICS_MEAN=1
STATISTICS_MINIMUM=1
STATISTICS_STDDEV=0
STATISTICS_VALID_PERCENT=100
Scale: 1
Offset: 0
More information    
Dimensions  X: 10 Y: 10 Bands: 1

Extent  0.0000000000000000,-10.0000000000000000 : 10.0000000000000000,0.0000000000000000
Width   10
Height  10
Data type   Byte - Eight bit unsigned integer
GDAL Driver Description GTiff
GDAL Driver Metadata    GeoTIFF
Dataset Description test_zstd.tiff
Compression ZSTD
Band 1  
STATISTICS_MAXIMUM=1
STATISTICS_MEAN=1
STATISTICS_MINIMUM=1
STATISTICS_STDDEV=0
STATISTICS_VALID_PERCENT=100
Scale: 1
Offset: 0
More information    
Dimensions  X: 10 Y: 10 Bands: 1

So I think the issue could be due to the GDAL library (built without the ZTSD compression support or missing some dependencies related to the ZSTD compression) used by QGIS on you Gentoo system which could be different from the GDAL version used by the rasterio Python module. You can check the GDAL version used by rasterio with print(rasterio.__gdal_version__) (I don't know if rasterio actually uses GDAL to create the raster) Could you please check if gdalinfo from the GDAL version used by QGIS on your system is capable to read the ZSTD compressed layer?

Anyway reading a ZSTD compressed GeoTIFF raster doesn't depend on QGIS, but on the GDAL library. GDAL is capable of read and write a ZSTD compressed GeoTIFF raster if build in the correct way.

[agiudiceandrea]

@latot I've also tested the DEFLATE and ZSTD compressed GeoTIFF raster layers using QGIS 3.28.2 on Ubuntu 22.04 and they are both correctly imported without issues. This confirms the issue is due to a packaging issue in the GDAL built in your Gentoo system (and in Flatpak). I suggest you to contact the QGIS/GDAL packagers for the Gentoo system.

[latot]

let me check this before close plis u.u

[latot]

Oks, this is weird!

rasterio seems to work over GDAL (the compression params is a GDAL param, so is a deduction). But after checking gdalinfo:

Warning 1: test.tiff: test.tiff:ZSTD compression support is not configured
ERROR 1: test.tiff: Cannot open TIFF file due to missing codec ZSTD.
gdalinfo failed - unable to open 'test.tiff'.

So, weird that rasterio can use zstd, well I found other thing, from GDAL:

ZSTD is available since GDAL 2.3 when using internal libtiff and if GDAL built against libzstd >=1.0, or if built against external libtiff with zstd support.

So, my GDAL does have TIFF, is enabled by default (in gentoo you can choose what libs you use), I checked the TIFF lib, and has ZSTD disabled D: After enable it and recompile tiff now:

Driver: GTiff/GeoTIFF
Files: test.tiff
Size is 10, 10
Image Structure Metadata:
  COMPRESSION=ZSTD
  INTERLEAVE=BAND
Corner Coordinates:
Upper Left  (    0.0,    0.0)
Lower Left  (    0.0,   10.0)
Upper Right (   10.0,    0.0)
Lower Right (   10.0,   10.0)
Center      (    5.0,    5.0)
Band 1 Block=10x10 Type=Byte, ColorInterp=Gray

Great! now works.

And, thx to this, we know the flatpak version need a fix too.

Thx!

[agiudiceandrea]

Issue report submitted for QGIS packaged by Flatpak https://github.com/flathub/org.qgis.qgis/issues/254.