search

OSGeo / gdal

GDAL is an open source MIT licensed translator library for raster and vector geospatial data formats.
https://gdal.org
Other
4.67k stars 2.46k forks source link

Lack of progress report for FlushCache() (causes apparent hang with GTIFF and WEBP compression) #1473

Open palmerj opened 5 years ago

palmerj commented 5 years ago

Expected behavior and actual behavior.

gdal_translate should finish but just hangs. Putting on debug produces:

gdal_translate -of GTiff -co COMPRESS=WEBP -co WEBP_LOSSLESS=TRUE -co TILED=YES BA31_2623.tif BA31_2623_webplossless.tif --debug on

GNM: GNMRegisterAllInternal
GNM: RegisterGNMFile
GNM: RegisterGNMdatabase
GDAL: GDALOpen(BA31_2623.tif, this=0x7fd823029200) succeeds as GTiff.
Input file size is 6400, 9600
GDAL: QuietDelete(BA31_2623_webplossless.tif) invoking Delete()
GDAL: GDALOpen(BA31_2623_webplossless.tif, this=0x7fd82302b600) succeeds as GTiff.
GDAL: GDALDefaultOverviews::OverviewScan()
MDReaderPleiades: Not a Pleiades product
MDReaderPleiades: Not a Pleiades product
GDAL: GDALClose(BA31_2623_webplossless.tif, this=0x7fd82302b600)
MDReaderPleiades: Not a Pleiades product
MDReaderPleiades: Not a Pleiades product
MDReaderPleiades: Not a Pleiades product
MDReaderPleiades: Not a Pleiades product
0GDAL: GDAL_CACHEMAX = 819 MB
GDAL: GDALDatasetCopyWholeRaster(): 6400*256 swaths, bInterleave=1
VSI: Sorry: GetExtentStatus() not implemented for this operating system
...10...20...30...40...50...60...70...80...90...100 - done.

I have to control+c to exit program. The output file is corrupted.

Steps to reproduce the problem.

Data is here: https://www.dropbox.com/s/7w1hwm73l676hae/BA31_2623.tif.zip?dl=1

Operating system

MacOSX 10.14.4

GDAL version and provenance

2.4.1 installed from MacOS osgeo-homebrew

palmerj commented 5 years ago

Same issue occurs on Ubuntu with GDAL 2.4.1 installed from source.

rouault commented 5 years ago

@palmerj You just lack patience ;-) Lossless WEBP is super slow. Adding -co NUM_THREADS=ALL_CPUS will make things faster. Eventually it completed in a few minutes. Actually what happens is that the raster is small enough to fit in GDAL block cache in the translation phase. The real compression work is deferred in the FlushCache() step, which unfortunately has no proper progress report mechanism. In e3511844a11959bd81f5956a1e7e2de96900e7ac , I've added a rather poor-man progress report for that step when CPL_DEBUG=ON and for console applications. A proper fix would be much more involved and require revising the virtual FlushCache() method in the API and drivers to add a progress report, and also have GDALTranslate() takes that into account.

rouault commented 5 years ago

If you want more accurate progress report, you can also decrease GDAL_CACHEMAX, like --config GDAL_CACHEMAX 1 ... (but might affect performance in some cases of course)

palmerj commented 5 years ago

Wow ok thanks WebP encoding is very slow! The progress reporting certainly got me thinking there was something wrong.

rouault commented 5 years ago

I'm re-opening the issue since FlushCache() inherent lack of progress mechanism should ideally be fixed in a cleaner way that I did, and thatcould benefit to non-console applications