Issue with GTiff LZW compression (LZWPreDecode:Old-style LZW )

[xiaoda0801]

My GDAL can't use the compression algorithm anywhere, and the error message in my log is: LZWPreDecode:Old-style LZW codes, convert file, how can I fix this

[jratike80]

Please explain what you try to do and what is your GDAL version. It seems that you have TIFFs which are created with some old LZW compression implementation. Either your images are old, or then the software that writes them is using some old components. But the libtiff library seems to handle the case and send only a warning, not an error https://github.com/dducharme/tiff/blob/master/libtiff/tif_lzw.c#L220. Could you share one image for testing?

[xiaoda0801]

The version used is GDAL 3.6.3.I used GDAL to generate mosaics, driver to create a raster, and then warp.I've used compression algorithms everywhere else and none of them seem to work，and Images that use a compression algorithm are blank when opened in QGIS

[xiaoda0801]

I don't think it's a waring, and the error message also includes: using code not yet in table; LZWDecode: Corrupted LZW table at scanline 0

[jratike80]

Please, give us something concrete and re-producible. Do you believe that you could make an analysis yourself if someone tells you "I've used compression algorithms everywhere else and none of them seem to work"

Test image + commands that you use would be a good start.

[xiaoda0801]

I'm sorry, but the image is very large, about 15GB, and the image is confidential. When I created the mosaic data, I implemented it under C++, created a dataset using the GDALDriver pointer, and compressed the parameter papszOptions = CSLSetNameValue(papszOptions, "COMPRESS", "LZW"), and then used the function ChunkAndWarpImage for the warp operation. I also failed with LZW compression when creating pyramids. I searched the GDAL source for the error messages that showed up below libtiff, and I wondered if my libtiff wasn't configured correctly

[jratike80]

The test image does not need to be your confidential data. If the error is in LZW decompress/compress, the source data does not be 15 GB image and it is irrelevant from what part of the world it is, or it is has any real world contents at all. Good test images can be generated for example with https://gdal.org/programs/gdal_create.html. However, I believe that there is something wrong with your GDAL build. LZW compression/decompression is very common task and also continuously tested, see https://github.com/OSGeo/gdal/blob/master/autotest/gcore/tiff_write.py#L285

[xiaoda0801]

I think this problem is related to the internal implementation of gdalwarp, and I have a similar problem using the command line in QGIS. Can't use compression-related settings when gdalwarp executes mosaic.

gdalwarp input1.tif input2.tif -co COMPRESS=LZW -co TILED=YES -co BLOCKXSIZE=256 -co BLOCKYSIZE=256 -co BIGTIFF=YES -muti -co NUM_THREADS=ALL_CPUS output.tif.

I got this issue " using code not yet in table ".

[rouault]

@xiaoda0801 Please provide the input image(s). Ideally just one (and possibly a subset of it generated with gdal_translate -srcwin) if that's sufficient to reproduce

[xiaoda0801]

@rouault Hi，I use the following file, execute the mosaic command, and see that the source code error message comes from libtiff. I referred to gdalwarp.cpp to implement the image mosaic function in c++ program, and a similar error occurred.I wonder if compression cannot be used when creating output files before executing GDAL Warp.

https://drive.google.com/drive/folders/1Li-kIR1ldhYmYJDJchzI8-CbMTyS32Bo?usp=drive_link

[rouault]

@xiaoda0801 I don't reproduce any issue with a master build of GDAL with internal libtiff:

$ gdalwarp input2a.tif input2b.tif -co  COMPRESS=LZW   -co TILED=YES  -co BLOCKXSIZE=256  -co  BLOCKYSIZE=256   -co   BIGTIFF=YES  -multi  -co  NUM_THREADS=ALL_CPUS    output.tif -overwrite
Creating output file that is 18284P x 32984L.
Processing input2a.tif [1/2] : 0Using internal nodata values (e.g. 0) for image input2a.tif.
Copying nodata values from source nput2a.tif to destination output.tif.
...10...20...30...40...50...60...70...80...90...100 - done.
Processing input2b.tif [2/2] : 0Using internal nodata values (e.g. 0) for image input2b.tif.
...10...20...30...40...50...60...70...80...90...100 - done.

$ gdalinfo -checksum output.tif 
Driver: GTiff/GeoTIFF
Files: output.tif
Size is 18284, 32984
[...]
Band 1 Block=256x256 Type=Byte, ColorInterp=Red
  Checksum=26182
  NoData Value=0
Band 2 Block=256x256 Type=Byte, ColorInterp=Green
  Checksum=8026
  NoData Value=0
Band 3 Block=256x256 Type=Byte, ColorInterp=Blue
  Checksum=61052
  NoData Value=0

[xiaoda0801]

@rouault I'm very sorry. In order to facilitate the uploading of data, the above file is part of my cropping, not the entire file. Below is the data that caused the problem in my test.A file will also be generated after the execution error, but it cannot be opened using QGIS.

https://drive.google.com/drive/folders/1Li-kIR1ldhYmYJDJchzI8-CbMTyS32Bo?usp=drive_link

gdalwarp D:\c++code\data\test1.tif D:\c++code\data\test2.tif -co COMPRESS=LZW -co TILED=YES -co BLOCKXSIZE=256 -co BLOCKYSIZE=256 -co BIGTIFF=YES -multi -co NUM_THREADS=ALL_CPUS D:\c++code\data\outtest.tif Creating output file that is 28023P x 28814L. Processing D:\c++code\data\test1.tif [1/2] : 0Using internal nodata values (e.g. 0) for image D:\c++code\data\test1.tif. Copying nodata values from source D:\c++code\data\test1.tif to destination D:\c++code\data\outtest.tif. ...10...20...30...40...50...60...70...80...90...100 - done. Processing D:\c++code\data\test2.tif [2/2] : 0Using internal nodata values (e.g. 0) for image D:\c++code\data\test2.tif. ERROR 1: /vsimem/decompress_000002443E1BE1F0.tif:Using code not yet in table ERROR 1: /vsimem/decompress_000002443E1BE330.tif:Using code not yet in table ERROR 1: /vsimem/decompress_000002443E1BE3A8.tif:Using code not yet in table ERROR 1: /vsimem/decompress_000002443E1BE3F8.tif:Using code not yet in table ERROR 1: /vsimem/decompress_000002443E1BE358.tif:Using code not yet in table ERROR 1: /vsimem/decompress_000002443E1BE380.tif:Using code not yet in table ERROR 1: /vsimem/decompress_000002441B7234B0.tif:Using code not yet in table ...10...20...30...40...50...60...70...80...90...100 - done.

[rouault]

@xiaoda0801 I can't still reproduce a LZW error with your latest datasets. That said, when trying to reproduce, I hit a deadlock situation (gdalwarp hang forever), hopefully fixed by https://github.com/OSGeo/gdal/pull/8561

[rouault]

@xiaoda0801 Can you update your GDAL version to GDAL master possibly? (or the latest stable release 3.7.2) and see if you still reproduce the issue

[xiaoda0801]

@rouault Hi. The GDAL version used by my QGIS is release 3.7.2, but this problem still occurs. I haven't upgraded to master yet to reproduce this problem. I think this may be an internal defect of GDAL. I can solve this problem by using WRITE_FLUSH=YES in the code, but this method seems to reduce the processing speed. And I don't think 'using WRITE_FLUSH=YES' is a wise approach.

[rouault]

can you add "--debug on" to your gdalwarp command line and report the output ? Of interest are the debug information about GDAL_CACHEMAX and the number of threads

[xiaoda0801]

OK, the output of my execution using --debug on: D:\Program Files\QGIS 3.32.3\bin>gdalwarp D:\c++code\data\test1.tif D:\c++code\data\test2.tif -co COMPRESS=DEFLATE -co TILED=YES -co BLOCKXSIZE=256 -co BLOCKYSIZE=256 -co BIGTIFF=YES -multi -co NUM_THREADS=ALL_CPUS D:\c++code\ data\outtest.tif --debug on GDAL: GDALOpen(D:\c++code\data\test1.tif, this=00000195A1FC80D0) succeeds as GTiff. GDAL: GDALOpen(D:\c++code\data\test2.tif, this=00000195A3B43A60) succeeds as GTiff. GDAL: Using GTiff driver GDAL: Computing area of interest: 104.649, 30.3862, 104.817, 30.572 OGRCT: Wrap source at 104.733. GDAL: Computing area of interest: 104.59, 30.3386, 104.756, 30.5171 OGRCT: Wrap source at 104.673. Creating output file that is 28023P x 28814L. GDAL: GDALDriver::Create(GTiff,D:\c++code\data\outtest.tif,28023,28814,3,Byte,00000195A3BC5930) GTiff: File being created as a BigTIFF. GTiff: Using up to 8 threads for compression/decompression Processing D:\c++code\data\test1.tif [1/2] : 0WARP: Copying metadata from first source to destination dataset GDAL: Computing area of interest: 104.649, 30.3862, 104.817, 30.572 GTiff: ScanDirectories() GDAL: GDALDefaultOverviews::OverviewScan() Using internal nodata values (e.g. 0) for image D:\c++code\data\test1.tif. Copying nodata values from source D:\c++code\data\test1.tif to destination D:\c++code\data\outtest.tif. WARP: srcNoData=0.000000 dstNoData=0.000000 WARP: calling GDALSetRasterNoDataValue() for band#0 WARP: srcNoData=0.000000 dstNoData=0.000000 WARP: calling GDALSetRasterNoDataValue() for band#1 WARP: srcNoData=0.000000 dstNoData=0.000000 WARP: calling GDALSetRasterNoDataValue() for band#2 GDALWARP: Defining SKIP_NOSOURCE=YES GDAL: Start chunk 0 / 78. GDAL: Start chunk 1 / 78. GDAL: GDAL_CACHEMAX = 808 MB GDAL: GDALWarpKernel()::GWKNearestByte() Src=0,0,3276x1800 Dst=7005,0,3503x1800 GDAL: GDALWarpKernel()::GWKNearestByte() Src=3276,0,3503x1800 Dst=10508,0,3503x1800 GDAL: Finished chunk 0 / 78. GDAL: Start chunk 2 / 78. GDAL: GDALWarpKernel()::GWKNearestByte() Src=6779,0,3503x1800 Dst=14011,0,3503x1800 GDAL: Finished chunk 1 / 78. GDAL: Start chunk 3 / 78. GDAL: GDALWarpKernel()::GWKNearestByte() Src=10282,0,3503x1800 Dst=17514,0,3503x1800 .GDAL: Finished chunk 2 / 78. GDAL: Start chunk 4 / 78. GDAL: GDALWarpKernel()::GWKNearestByte() Src=13785,0,3503x1800 Dst=21017,0,3503x1800 GDAL: Finished chunk 3 / 78. GDAL: Start chunk 5 / 78. GDAL: GDALWarpKernel()::GWKNearestByte() Src=17288,0,3503x1800 Dst=24520,0,3503x1800 GDAL: Finished chunk 4 / 78. GDAL: Start chunk 6 / 78. GDAL: GDALWarpKernel()::GWKNearestByte() Src=0,1800,3276x1801 Dst=7005,1800,3503x1801 GDAL: Finished chunk 5 / 78. GDAL: Start chunk 7 / 78. .GDAL: GDALWarpKernel()::GWKNearestByte() Src=3276,1800,3503x1801 Dst=10508,1800,3503x1801 GDAL: Finished chunk 6 / 78. GDAL: Start chunk 8 / 78. GDAL: GDALWarpKernel()::GWKNearestByte() Src=6779,1800,3503x1801 Dst=14011,1800,3503x1801 GDAL: Finished chunk 7 / 78. GDAL: Start chunk 9 / 78. GDAL: Finished chunk 8 / 78. GDAL: Start chunk 10 / 78. GDAL: GDALWarpKernel()::GWKNearestByte() Src=10282,1800,3503x1801 Dst=17514,1800,3503x1801 .GDAL: GDALWarpKernel()::GWKNearestByte() Src=13785,1800,3503x1801 Dst=21017,1800,3503x1801 GDAL: Finished chunk 9 / 78. GDAL: Start chunk 11 / 78. GDAL: GDALWarpKernel()::GWKNearestByte() Src=17288,1800,3503x1801 Dst=24520,1800,3503x1801 GDAL: Finished chunk 10 / 78. GDAL: Start chunk 12 / 78. GDAL: GDALWarpKernel()::GWKNearestByte() Src=0,3601,3276x1801 Dst=7005,3601,3503x1801 GDAL: Finished chunk 11 / 78. GDAL: Start chunk 13 / 78. 10GDAL: Finished chunk 12 / 78. GDAL: Start chunk 14 / 78. GDAL: GDALWarpKernel()::GWKNearestByte() Src=3276,3601,3503x1801 Dst=10508,3601,3503x1801 GDAL: Finished chunk 13 / 78. GDAL: GDALWarpKernel()::GWKNearestByte() Src=6779,3601,3503x1801 Dst=14011,3601,3503x1801 GDAL: Start chunk 15 / 78. GDAL: Finished chunk 14 / 78. GDAL: Start chunk 16 / 78. GDAL: GDALWarpKernel()::GWKNearestByte() Src=10282,3601,3503x1801 Dst=17514,3601,3503x1801 GDAL: GDALWarpKernel()::GWKNearestByte() Src=13785,3601,3503x1801 Dst=21017,3601,3503x1801 .GDAL: Finished chunk 15 / 78. GDAL: Start chunk 17 / 78. GDAL: GDALWarpKernel()::GWKNearestByte() Src=17288,3601,3503x1801 Dst=24520,3601,3503x1801 GDAL: Finished chunk 16 / 78. GDAL: Start chunk 18 / 78. GDAL: GDALWarpKernel()::GWKNearestByte() Src=0,5402,3276x1801 Dst=7005,5402,3503x1801 GDAL: Finished chunk 17 / 78. GDAL: Start chunk 19 / 78. GDAL: GDALWarpKernel()::GWKNearestByte() Src=3276,5402,3503x1801 Dst=10508,5402,3503x1801 GDAL: Finished chunk 18 / 78. GDAL: Start chunk 20 / 78. .GTIFF: Waiting for worker job to finish handling block 27 GTIFF: Waiting for worker job to finish handling block 28 GDAL: GDALWarpKernel()::GWKNearestByte() Src=6779,5402,3503x1801 Dst=14011,5402,3503x1801 GTIFF: Waiting for worker job to finish handling block 36 GTIFF: Waiting for worker job to finish handling block 139 GTIFF: Waiting for worker job to finish handling block 143 GTIFF: Waiting for worker job to finish handling block 148 GTIFF: Waiting for worker job to finish handling block 150 GTIFF: Waiting for worker job to finish handling block 251 GTIFF: Waiting for worker job to finish handling block 253 GTIFF: Waiting for worker job to finish handling block 257 GTIFF: Waiting for worker job to finish handling block 260 GTIFF: Waiting for worker job to finish handling block 358 GTIFF: Waiting for worker job to finish handling block 363 GDAL: Finished chunk 19 / 78. GDAL: Start chunk 21 / 78. GTIFF: Waiting for worker job to finish handling block 469 GDAL: GDALWarpKernel()::GWKNearestByte() Src=10282,5402,3503x1801 Dst=17514,5402,3503x1801 GTIFF: Waiting for worker job to finish handling block 475 GTIFF: Waiting for worker job to finish handling block 476 GTIFF: Waiting for worker job to finish handling block 479 GTIFF: Waiting for worker job to finish handling block 580 GTIFF: Waiting for worker job to finish handling block 586 GTIFF: Waiting for worker job to finish handling block 589 GTIFF: Waiting for worker job to finish handling block 689 GTIFF: Waiting for worker job to finish handling block 691 GTIFF: Waiting for worker job to finish handling block 695 GTIFF: Waiting for worker job to finish handling block 700 GTIFF: Waiting for worker job to finish handling block 42 GTIFF: Waiting for worker job to finish handling block 44 GTIFF: Waiting for worker job to finish handling block 53 GTIFF: Waiting for worker job to finish handling block 151 GTIFF: Waiting for worker job to finish handling block 160 GTIFF: Waiting for worker job to finish handling block 261 GTIFF: Waiting for worker job to finish handling block 270 GTIFF: Waiting for worker job to finish handling block 371 GTIFF: Waiting for worker job to finish handling block 372 GTIFF: Waiting for worker job to finish handling block 381 GTIFF: Waiting for worker job to finish handling block 481 GTIFF: Waiting for worker job to finish handling block 491 GTIFF: Waiting for worker job to finish handling block 591 GTIFF: Waiting for worker job to finish handling block 592 GTIFF: Waiting for worker job to finish handling block 602 GTIFF: Waiting for worker job to finish handling block 701 GTIFF: Waiting for worker job to finish handling block 710 GTIFF: Waiting for worker job to finish handling block 713 GTIFF: Waiting for worker job to finish handling block 62 GTIFF: Waiting for worker job to finish handling block 167 GTIFF: Waiting for worker job to finish handling block 176 GTIFF: Waiting for worker job to finish handling block 281 GTIFF: Waiting for worker job to finish handling block 286 、、、、、 ERROR 1: TIFFFetchDirectory:D:\c++code\data\outtest.tif: Can not read TIFF directory count ERROR 1: TIFFReadDirectory:Failed to read directory at offset 821212854 ERROR 1: D:\c++code\data\outtest.tif: FillEmptyTiles() failed because panByteCounts == NULL GDAL: GDALClose(D:\c++code\data\outtest.tif, this=00000195A3C62200) GDAL: GDALClose(D:\c++code\data\test1.tif, this=00000195A1FC80D0) GDAL: GDALClose(D:\c++code\data\test2.tif, this=00000195A3B43A60)

[xiaoda0801]

The output shows that the chunk has not been processed and the work job is waiting. 、、、、、、 GTIFF: Waiting for worker job to finish handling block 7982 GTIFF: Waiting for worker job to finish handling block 8085 GTIFF: Waiting for worker job to finish handling block 8091 GDAL: Finished chunk 6 / 96. GDAL: Start chunk 8 / 96. ERROR 1: /vsimem/decompress_000001C5A7C43698.tif:Using code not yet in table GTIFF: Waiting for worker job to finish handling block 2883 GTIFF: Waiting for worker job to finish handling block 2993 GDAL: GDALWarpKernel()::GWKNearestByte() Src=6129,0,876x423 Dst=6129,6752,876x451 GTIFF: Waiting for worker job to finish handling block 7769 GTIFF: Waiting for worker job to finish handling block 7776 GTIFF: Waiting for worker job to finish handling block 7780 GTIFF: Waiting for worker job to finish handling block 7881 GDAL: Finished chunk 7 / 96. GDAL: Start chunk 9 / 96. GDAL: Finished chunk 8 / 96. ERROR 1: /vsimem/decompress_000001C5A7CE41C8.tif:Using code not yet in table ERROR 1: /vsimem/decompress_000001C5A7CE4178.tif:Using code not yet in table ERROR 1: /vsimem/decompress_000001C5A7CE4218.tif:Using code not yet in table ERROR 1: LZWDecode:Not enough data at scanline 0 (short 196608 bytes) ERROR 1: /vsimem/decompress_000001C5A7CE41F0.tif:Using code not yet in table ...10...20...30...40...50...60...70...80...90...100 - done. GDAL: Flushing dirty blocks: 0GTIFF: Waiting for worker job to finish handling block 2868 GTIFF: Waiting for worker job to finish handling block 2874 GTIFF: Waiting for worker job to finish handling block 2875 GTIFF: Waiting for worker job to finish handling block 2878 GTIFF: Waiting for worker job to finish handling block 2884 、、、、、

[jratike80]

Have you already tested if there is any difference between using a single VRT file that contains files D:\c++code\data\test1.tif D:\c++code\data\test2.tif as input and your approach to give those two files as a list?

[xiaoda0801]

@jratike80 I didn't find any difference between these two methods. You can use the following data to reproduce the problem. https://drive.google.com/drive/folders/1Li-kIR1ldhYmYJDJchzI8-CbMTyS32Bo

[jratike80]

Your case seems to be hard to debug. Your command works for me without errors. I used OSGeo4W installation of GDAL 3.8.0dev-1d1044ce33, released 2023/09/28 on Windows. I got the same number of threads from my computer: "GTiff: Using up to 8 threads for compression/decompression". GDAL 3.8.0dev from gisinternals.com worked also.

How did you install your GDAL? Could you try some other installations? As a layman I guess that some part of the code says that it has finalized the job and the next part can continue, but in your environment the essential bits of data are not available for the second part of code in a place where the bits are supposed to be. So where those bits are? In some cache or what?

[xiaoda0801]

I used code similar to gdalwarp to implement mosaic in the code, and found that the image could not be opened after compression. Then using the command line in QGIS3.32.3 (GDAL3.7.2), the error in the code was also reproduced. The error message is the one listed above. I think that GDAL's internal GDAChunkAndMulti will have chunk waiting problems when executing on compressed images. I used the parameter WRITE_FLUSH=TRUE to avoid this situation, but I don't think this is a wise choice.

[jratike80]

It was a good idea to check how the result looks. The multithreaded option with OSGeo4W that seems to be successful creates an image that is mostly black. A single-threaded command probably makes the result that you would like to see. The simple command was

gdalwarp test1.tif test2.tif -co COMPRESS=DEFLATE -co TILED=YES outtest2.tif

[rouault]

ok, I've finally tested on a Windows VM, and I can report that I could consistently get corruptions without the fix of https://github.com/OSGeo/gdal/pull/8561, and I no longer get them once it is applied. On reflection, the fix isn't just about potential deadlock but also about the multithreaded GTiff decoder potentially trying to read a tile that the multithreaed GTiff encoder would write. Closing as resolved (for GDAL 3.7.3 and 3.8.0)