search

qissue-bot / QGIS

QGIS is a free, open source, cross platform (lin/win/mac) geographical information system (GIS)
http://qgis.org
GNU General Public License v2.0
0 stars 0 forks source link

Building pyramids causes segfault #592

Closed qissue-bot closed 5 years ago

qissue-bot commented 5 years ago

Author Name: Gary Sherman (Gary Sherman) Original Redmine Issue: 605, https://issues.qgis.org/issues/605

Original Assignee: Tim Sutton

Building pyramids segfaults. Image size was 8192x4096.

Backtrace:

(gdb) bt
#0  0xb62424c7 in memset () from /lib/tls/i686/cmov/libc.so.6
#3905  0x00002000 in ?? ()
#3906  0xb6be5c85 in _TIFFmemset () from /usr/local/lib/libgdal.so.1
#3907  0xb6bd4ab2 in LZWPreDecode () from /usr/local/lib/libgdal.so.1
#3908  0xb6be372c in TIFFStartTile () from /usr/local/lib/libgdal.so.1
#3909  0xb6be330d in TIFFFillTile () from /usr/local/lib/libgdal.so.1
#3910  0xb6be2bce in TIFFReadEncodedTile () from /usr/local/lib/libgdal.so.1
#3911  0xb6ab8bd5 in GTiffDataset::LoadBlockBuf () from /usr/local/lib/libgdal.so.1
#3912  0xb6aba436 in GTiffRasterBand::IReadBlock () from /usr/local/lib/libgdal.so.1
#3913  0xb6c10691 in GDALRasterBand::GetLockedBlockRef () from /usr/local/lib/libgdal.so.1
#10 0xb6c22235 in GDALRasterBand::IRasterIO () from /usr/local/lib/libgdal.so.1
#3914 0xb6c0f4fc in GDALRasterBand::RasterIO () from /usr/local/lib/libgdal.so.1
#3915 0xb6c1efc8 in GDALDownsampleChunk32R () from /usr/local/lib/libgdal.so.1
#3916 0xb6c20156 in GDALRegenerateOverviews () from /usr/local/lib/libgdal.so.1
#3917 0xb6abd0c1 in GTiffDataset::IBuildOverviews () from /usr/local/lib/libgdal.so.1
#3918 0xb6bff0b6 in GDALDataset::BuildOverviews () from /usr/local/lib/libgdal.so.1
#3919 0xb7ee7529 in [[QgsRasterLayer]]::buildPyramids (this=0x82abfa0, theRasterPyramidList=@0xbfd31eac,
    theResamplingMethod=@0xbfd31eb0) at qgsrasterlayer.cpp:4216
#3920 0xb65ccaf1 in [[QgsRasterLayerProperties]]::on_buttonBuildPyramids_clicked (this=0x8319d80)
    at qgsrasterlayerproperties.cpp:478
qissue-bot commented 5 years ago

Original Redmine Comment Author Name: Tim Sutton (Tim Sutton) Original Date: 2007-02-25T18:18:46.000Z

Moved to milestone 0.8.2 since we wont be fixing any further issues before the 0.8.1 release

qissue-bot commented 5 years ago

Original Redmine Comment Author Name: Frank Warmerdam - (Frank Warmerdam -) Original Date: 2007-12-07T12:51:36.000Z

I believe an upgrade to GDAL 1.4.4 with internal libtiff, or libtiff 3.9 (from CVS) will resolve this problem. It is a libtiff bug, not a GDAL or QGIS bug.

qissue-bot commented 5 years ago

Original Redmine Comment Author Name: Maciej Sieczka - (Maciej Sieczka -) Original Date: 2008-03-08T09:46:27.000Z

I have just checked in QGIS , built against GDAL 1.5.0+SVN 2008-03-04 (internal libtiff). No crashes building overviews in QGIS whatsoever. Checked with several Byte and Uint16 [[GeoTIFFs]], all having colortables, compressed with lzw, deflate or packbits.

I still believe however that building external overviews in Erdas Imagine format would be more bulletproof and universal. The benefits of external overviews:

  1. The input data are not modified.

  2. Ergo, input data remain compatible with software that don't tolerate internal overviews and

  3. there's no need to duplicate data for safety/usability reasons before building overviews.

  4. Other software support external Erdas Imagine overviews that GDAL creates - eg. [[ArcGIS]], ERDAS Imagine.

What do developers think about it?

For now, to create an external Erdas Imagin overwievs from CLI use eg.:

gdaladdo --config USE_RRD YES some.tif 2 4 8 16
qissue-bot commented 5 years ago

Original Redmine Comment Author Name: Maciej Sieczka - (Maciej Sieczka -) Original Date: 2008-08-01T02:47:28.000Z

Since my last #comment:4 regarding overwievs, I have learned that external overviews in GeoTIFF are also a nice option. They meet points 1-3 too, and although they are not so compatible as Erdas overviews, they are more compressable beacuse you can use DEFLATE for TIFF, which is more efficient than Erdas RLC compression method (the only available for Erdas, applied by default to Erdas overviews; and can't be disabled for overviews AFAIK).

Compare (the input is a 361 MB uncompressed [[GeoTIFF]]):

$ gdaladdo --config USE_RRD YES atif.tif 2 4 8 16
$ ls -sh atif.aux
125M atif.aux

$ chmod a-w atif.tif
$ gdaladdo --config COMPRESS_OVERVIEW DEFLATE atif.tif 2 4 8 16
$ ls -sh atif.tif.ovr
103M atif.tif.ovr

So, ideally, QGIS should let the user choose if he wants internal overviews (and if/what compresssion scheme) or external overviews - in Erdas or [[GeoTIFF]] format (the former for better compatibility, the latter for space saving - if chosen let the user choose DEFLATE/LZW/PACKBITS; hint that DEFLATE compresses best and better than Erdas compressions).

qissue-bot commented 5 years ago

Original Redmine Comment Author Name: jaadfoo - (jaadfoo -) Original Date: 2008-08-01T09:08:35.000Z

One question:

In the first example you show a ".aux" file of 125M. Shouldn't this be the ".rrd" file? From my understanding of aux files, this should contain a small amount of information, usually histogram / coordinate / projection info / etc. The rrd should contain the pyramids, and in my local tests, this is the case.

qissue-bot commented 5 years ago

Original Redmine Comment Author Name: Maciej Sieczka - (Maciej Sieczka -) Original Date: 2008-08-01T09:21:42.000Z

Replying to [comment:8 dickeya]:

One question:

In the first example you show a ".aux" file of 125M. Shouldn't this be the ".rrd" file?

My *.aux file is a direct output of a command like 'gdaladdo --config USE_RRD YES some.tif 2 4 8 16' and the command does not produce anything else. If you get a file with different extension then I don't know why. I'm using GDAL 1.5.2 + SVN FWIW.

The rrd should contain the pyramids, and in my local tests, this is the case.

Do you mean gdaladdo output? Please copy&paste the command syntax, gdalinfo of input file, the list of output files, gdalinfo of what you believe is the overviews file and your GDAL version.

qissue-bot commented 5 years ago

Original Redmine Comment Author Name: jaadfoo - (jaadfoo -) Original Date: 2008-08-01T09:49:00.000Z

As it turns out, I had HFA_USE_RRD set to YES in my environment. After removal, gdaladdo only created an aux file.

Thanks.

qissue-bot commented 5 years ago

Original Redmine Comment Author Name: Tim Sutton (Tim Sutton) Original Date: 2008-08-01T10:34:53.000Z

With you can now specify internal or external overviews. Compressed overviews are not yet supported since http://www.gdal.org/classGDALDataset.html#3906aa6f88b3bbc840a5696236af11dde15 provides no obvious way to do this (I would need to go and read gdaladdo code and see how they did it there). If you really want this feature, please file a separate enhancement ticket since we are wondering way off the topic of the original ticket.

Best regards

Tim

qissue-bot commented 5 years ago

Original Redmine Comment Author Name: Maciej Sieczka - (Maciej Sieczka -) Original Date: 2008-08-01T13:57:33.000Z

Replying to [comment:11 timlinux]:

With you can now specify internal or external overviews.

Cool. Smallish issue - there is a redundant test for file write access if the user chooses to build external overviews. It effectively disables external overviews creation for read-only rasters. As we are not going to alter the file, we don't care if it is writeable.

Compressed overviews (...) . If you really want this feature, please file a separate enhancement ticket since we are wondering way off the topic of the original ticket.

Done in #1199.

qissue-bot commented 5 years ago

Original Redmine Comment Author Name: Tim Sutton (Tim Sutton) Original Date: 2008-08-01T20:45:07.000Z

Hi

Its strange, because that message is propogated from GDAL as shown in the following snippet:

1601 if (!res.isNull()) 1602 { 1603 if (res == "ERROR_WRITE_ACCESS") 1604 { 1605 QMessageBox::warning(this, tr("Write access denied"), 1606 tr("Write access denied. Adjust the file permissions and try again.\ \ ") ); 1607 } 1608 else if (res == "ERROR_WRITE_FORMAT") 1609 { 1610 QMessageBox::warning(this, tr("Building pyramids failed."), 1611 tr("The file was not writeable. Some formats do not support pyramid overviews. Consult the GDAL documentation if in doubt.") ); 1612 } 1613 else if (res == "FAILED_NOT_SUPPORTED") 1614 { 1615 QMessageBox::warning(this, tr("Building pyramids failed."), 1616 tr("Building pyramid overviews is not supported on this type of raster.") ); 1617 } 1618 }

If gdal reports an error my feeling is that we should report it. Perhaps I will just add a message saying that the warning can be ignored if building external overviews and the dir is writable.

Regards

Tim

qissue-bot commented 5 years ago

Original Redmine Comment Author Name: Maciej Sieczka - (Maciej Sieczka -) Original Date: 2008-08-11T06:50:06.000Z

Replying to [comment:13 timlinux]:

Hi

Its strange, because that message is propogated from GDAL as shown in the following snippet:

1601 if (!res.isNull()) 1602 { 1603 if (res == "ERROR_WRITE_ACCESS") 1604 { 1605 QMessageBox::warning(this, tr("Write access denied"), 1606 tr("Write access denied. Adjust the file permissions and try again.\ \ ") ); 1607 } 1608 else if (res == "ERROR_WRITE_FORMAT") 1609 { 1610 QMessageBox::warning(this, tr("Building pyramids failed."), 1611 tr("The file was not writeable. Some formats do not support pyramid overviews. Consult the GDAL documentation if in doubt.") ); 1612 } 1613 else if (res == "FAILED_NOT_SUPPORTED") 1614 { 1615 QMessageBox::warning(this, tr("Building pyramids failed."), 1616 tr("Building pyramid overviews is not supported on this type of raster.") ); 1617 } 1618 }

If gdal reports an error my feeling is that we should report it.

Hmm, probably GDAL reports this error in error. It is logical to discard it IMO.

Perhaps I will just add a message saying that the warning can be ignored if building external overviews and the dir is writable.

I think QGIS should not bother about theis error at all and just proceed. There is no point in reporting that the file is not writeable if we don't try to modify it.

Another small issue I noticed: even if QGIS fails to build the overviews for whatever reason, the icons for selected overviews change like if the overviews were actually created. Re-entering raster's properties shows they were really were not.

Regards

Tim

qissue-bot commented 5 years ago

Original Redmine Comment Author Name: Anónimo (Anónimo) Original Date: 2009-08-21T22:57:01.000Z

Milestone Version 1.0.0 deleted