search

qgis / QGIS

QGIS is a free, open source, cross platform (lin/win/mac) geographical information system (GIS)
https://qgis.org
GNU General Public License v2.0
10.63k stars 3.01k forks source link

Opening an hdf5 file causes application to quit on macOS #45416

Open g-studer-ellis opened 3 years ago

g-studer-ellis commented 3 years ago

What is the bug or the crash?

Scenario: I am working with multispectral data downloaded from NASA Earthdata and am having trouble importing and opening the file in QGIS. Specifically, I have downloaded several granules of VIIRS/NPP Surface Reflectance 8-Day L3 Global 1km SIN Grid V001 products and cannot open them successfully. I can start a new project and open various other file types including NetCDF, GeoTIFF, and png files. However, when I attempt to load a new layer from an HDF5 file QGIS will crash and close.

QGIS Version: QGIS-LTR V 3.16.11-Hannover

Example File Name: VNP09A1.A2021265.h17v05.001.2021277005531.h5

Desired Outcome: The ability to open my files in QGIS and work with the multispectral data to produce various false color images.

What I’ve Tried: I have tried a variety of different HDF5 files, including different multispectral types. I did run one file through a conversion tool from NASA, which produced a NetCDF file which did open in QGIS. However this adds a few extra steps and is something I would like to avoid if possible.

Operating System: macOS Big Sur 11.6 QGIS Install: Official All-in-one signed installer: QGIS macOS Installer Version 3.16 Long Term Release

Steps to reproduce the issue

In a blank new project:

  1. Go to "Layer"
  2. Scroll down to "Add Layer"
  3. Roll mouse over to “Add Raster Layer”
  4. Click the "..." box to the right of the "Source" field
  5. Navigate to the HDF5 file and click "open"
  6. Click "Add"
  7. Select bands to render
  8. Verify that add layers to a group box is checked (I have tried this checked and unchecked with the same result)
  9. Click "OK"
Screen Shot 2021-10-05 at 3 19 04 PM

At this point QGIS quits.

Versions

QGIS version 3.16.11-Hannover QGIS code revision 26cc1c762f
Compiled against Qt 5.14.2 Running against Qt 5.14.2
Compiled against GDAL/OGR 3.2.1 Running against GDAL/OGR 3.2.1
Compiled against GEOS 3.9.1-CAPI-1.14.2 Running against GEOS 3.9.1-CAPI-1.14.2
Compiled against SQLite 3.31.1 Running against SQLite 3.31.1
PostgreSQL Client Version 12.3 SpatiaLite Version 4.3.0a
QWT Version 6.1.4 QScintilla2 Version 2.11.4
Compiled against PROJ 6.3.2 Running against PROJ Rel. 6.3.2, May 1st, 2020
OS Version macOS 11.6
Active python plugins processing; db_manager; MetaSearch
Screen Shot 2021-10-05 at 3 22 21 PM

Supported QGIS version

New profile

Additional context

No response

gioman commented 3 years ago

Navigate to the HDF5 file and click "open"

@g-studer-ellis can you add a link for the download on one of such files?

g-studer-ellis commented 3 years ago

Navigate to: https://lpdaac.usgs.gov/products/vnp09a1v001/ Click "Access Data" Click the "Download Symbol" for the "NASA Earthdata Search" Navigate to new tab Click "VIIRS/NPP Surface Reflectance 8-Day L3 Global 1km SIN Grid V001" Scroll down to "VNP09A1.A2021265.h17v04.001.2021277005427.h5" Click "Download Symbol" for "Download single granule data"

A few things to note: You need a NASA Earthdata account to download this data I have tried with multiple granules from multiple HDF5 data from this site, and they all crash I gave directions to the specific granule I mentioned above

Screen Shot 2021-10-05 at 3 58 53 PM
gioman commented 3 years ago

You need a NASA Earthdata account to download this data

@g-studer-ellis can you download the data and upload it to a service like dropbox, google drive, etc. ? thanks.

g-studer-ellis commented 3 years ago

Try this link: https://drive.google.com/file/d/1w2QV2og2c4mLVc2dhDDdPvVnp9vWRzb4/view?usp=sharing

gioman commented 3 years ago

https://drive.google.com/file/d/1w2QV2og2c4mLVc2dhDDdPvVnp9vWRzb4/view?usp=sharing

can't replicate the problem on Linux and Windows. Will try also on my mac mini.

gioman commented 3 years ago

Yes, crash confirmed on macOS Big Sur.

g-studer-ellis commented 3 years ago

Any update on when this might be addressed?

g-studer-ellis commented 2 years ago

QGIS still crashes for me when importing an hdf5 file. Is there an eta on when this will be addressed?

g-studer-ellis commented 2 years ago

Checking in about an update for this crash issue.

fabianlang commented 2 years ago

Happens as well on macOS 12.5 running QGIS 3.22 and QGIS 3.26.

richardhyy commented 1 year ago

Also happens on macOS 13.0.1 running QGIS 3.28.

GSeyfried commented 1 year ago

Still happening to me :(

roger-downing commented 1 year ago

Crash also happens with hdf5 files holding line geometry such as: https://github.com/ioos/compliance-checker/blob/master/compliance_checker/tests/data/line_geometry.cdl

Just selecting the line geometry layer works and displays the line features, but choosing to load data layers alongside causes a segfault. This is on Mac OS 13.2.1 with QGIS 3.28.3.

I should say, this CDL file can be converted to a netCDF file with: ncgen -b -k hdf5 -o ~/test.nc line_geometry.cdl

vbrancat commented 1 year ago

Confirming that qGIS Firenze LTR crashes when opening a HDF5 on Mac OS Ventura 13.6. However, if when opening the HDF5 file, I specify the path to a specific dataset like "HDF5:"path_to_HDF5_product":/path_to_hdf5_dataset then I am able to correctly visualize the dataset without qGIS crashing. However, the dataset is not geolocated properly.

If the same HDF5 file is converted to NetCDF, then it can be opened without any problem.

Is there a plan to address this issue? Most of the dataset generated by NASA mission used HDF5 as their main data format. Not able to correctly visualize them in qGIS is a big loss (and missed opportunity) for all the scientific community.

Samweli commented 8 months ago

More inspection on the issue shows that the crash originates from the upstream macOS hdf5 library used by the QGIS GDAL data provider, see the below backtrace

thread #54, name = 'Thread (pooled)', stop reason = EXC_BAD_ACCESS 
(code=1, address=0x0)
    frame #0: 0x000000011002c24c libhdf5.103.dylib`H5SL_search + 3004
    frame #1: 0x000000010ffe5931 libhdf5.103.dylib`H5P_create_id + 337
    frame #2: 0x000000010ffbc727 libhdf5.103.dylib`H5Pcreate + 135
    frame #3: 0x00000001201eee91 
libgdal.29.dylib`GDAL_HDF5Open(std::__1::basic_string<char, 
std::__1::char_traits<char>, std::__1::allocator<char> > const&) + 1009
    frame #4: 0x00000001201f4c7d 
libgdal.29.dylib`HDF5ImageDataset::Open(GDALOpenInfo*) + 829
    frame #5: 0x000000012087dd3c libgdal.29.dylib`GDALOpenEx + 2828
  * frame #6: 0x000000011614c09c 
qgis_core`QgsGdalProviderBase::gdalOpen(uri=0x0000700001364278, nOpenFlags=0) at qgsgdalproviderbase.cpp:300:22
    frame #7: 0x00000001161529d3 
qgis_core`QgsGdalProvider::initIfNeeded(this=0x0000000151cbe7c0) at qgsgdalprovider.cpp:3268:22
    frame #8: 0x0000hdf5000116158e18 
qgis_core`QgsGdalProvider::block(this=0x0000000151cbe7c0, bandNo=1, extent=0x00007000013653f0, width=240, height=241, feedback=0x0000000151cbe430) at qgsgdalprovider.cpp:661:9
    frame #9: 0x0000000116f6330d 
qgis_core`QgsSingleBandGrayRenderer::block(this=0x000060000b43cdc0, bandNo=1, extent=0x00007000013653f0, width=240, height=241, feedback=0x0000000151cbe430)at qgssinglebandgrayrenderer.cpp:109:63
    frame #10: 0x0000000116f1e0fa 
qgis_core`QgsBrightnessContrastFilter::block(this=0x0000600001b16480, bandNo=1, extent=0x00007000013653f0, width=240, height=241, feedback=0x0000000151cbe430) at qgsbrightnesscontrastfilter.cpp:119:57
    frame #11: 0x0000000116f20f3a 
qgis_core`QgsHueSaturationFilter::block(this=0x000060000a6bde60, bandNo=1, extent=0x00007000013653f0, width=240, height=241, feedback=0x0000000151cbe430) at qgshuesaturationfilter.cpp:124:57
    frame #12: 0x0000000116f5e969 
qgis_core`QgsRasterResampleFilter::block(this=0x0000600001b16440, bandNo=1, extent=0x00007000013653f0, width=240, height=241, feedback=0x0000000151cbe430) at qgsrasterresamplefilter.cpp:164:20
    frame #13: 0x0000000116f13a2e 
qgis_core`QgsRasterProjector::block(this=0x0000600002b6b560, bandNo=1, extent=0x00007000013653f0, width=240, height=241, feedback=0x0000000151cbe430) at qgsrasterprojector.cpp:824:20
    frame #14: 0x0000000116eb596b 
qgis_core`QgsRasterIterator::readNextRasterPartInternal(this=0x00007000013659b0, bandNumber=1, nCols=0x0000700001365744, nRows=0x0000700001365740, 
block=0x0000700001365730, topLeftCol=0x000070000136573c, 
topLeftRow=0x0000700001365738, blockExtent=0x0000000000000000, 
tileColumns=0x00007000013655b4, tileRows=0x00007000013655b0, 
tileTopLeftColumn=0x00007000013655ac, tileTopLeftRow=0x00007000013655a8) 
at qgsrasteriterator.cpp:201:27
    frame #15: 0x0000000116eb5c19 
qgis_core`QgsRasterIterator::readNextRasterPart(this=0x00007000013659b0, 
bandNumber=1, nCols=0x0000700001365744, nRows=0x0000700001365740, 
block=nullptr, topLeftCol=0x000070000136573c, 
topLeftRow=0x0000700001365738, blockExtent=0x0000000000000000, 
tileColumns=0x0000000000000000, tileRows=0x0000000000000000, 
tileTopLeftColumn=0x0000000000000000, tileTopLeftRow=0x0000000000000000) 
at qgsrasteriterator.cpp:126:20
    frame #16: 0x0000000116f3b649 
qgis_core`QgsRasterDrawer::draw(this=0x0000700001365990, 
p=0x00006000000a4bf0, viewPort=0x000060000b43caa0, 
qgsMapToPixel=0x0000000151cbe200, feedback=0x0000000151cbe430) at 
qgsrasterdrawer.cpp:78:22
    frame #17: 0x0000000116f3b3bb 
qgis_core`QgsRasterDrawer::draw(this=0x0000700001365990, 
context=0x0000000151cbe0f0, viewPort=0x000060000b43caa0, 
feedback=0x0000000151cbe430) at qgsrasterdrawer.cpp:46:3
    frame #18: 0x0000000116efc7af 
qgis_core`QgsRasterLayerRenderer::render(this=0x00006000031f3060) at 
qgsrasterlayerrenderer.cpp:399:10
    frame #19: 0x0000000116c04a86 
qgis_core`QgsMapRendererParallelJob::renderLayerStatic(job=0x00000001251f2950) 
at qgsmaprendererparalleljob.cpp:379:35
    frame #20: 0x0000000116c0aadd 
qgis_core`QtConcurrent::FunctionWrapper1<void, 
LayerRenderJob&>::operator(this=0x0000600002eeb0b8, 
u=0x00000001251f2950)(LayerRenderJob&) at 
qtconcurrentfunctionwrappers.h:80:16
    frame #21: 0x0000000116c09e48 
qgis_core`QtConcurrent::MapKernel<std::__1::__wrap_iter<LayerRenderJob*>, 
QtConcurrent::FunctionWrapper1<void, LayerRenderJob&> 
>::runIteration(this=0x0000600002eeb080, it=__wrap_iter<LayerRenderJob *> 
@ 0x0000700001365d08, (null)=2, (null)=0x0000000000000000) at 
qtconcurrentmapkernel.h:68:9
    frame #22: 0x0000000116c09ec3 
qgis_core`QtConcurrent::MapKernel<std::__1::__wrap_iter<LayerRenderJob*>, 
QtConcurrent::FunctionWrapper1<void, LayerRenderJob&> 
>::runIterations(this=0x0000600002eeb080, 
sequenceBeginIterator=__wrap_iter<LayerRenderJob *> @ 0x0000700001365d58, 
beginIndex=2, endIndex=3, (null)=0x0000000000000000) at 
qtconcurrentmapkernel.h:77:13
    frame #23: 0x0000000116c0a1de 
qgis_core`QtConcurrent::IterateKernel<std::__1::__wrap_iter<LayerRenderJob*>, 
void>::forThreadFunction(this=0x0000600002eeb080) at 
qtconcurrentiteratekernel.h:255:49
    frame #24: 0x0000000116c09cd7 
qgis_core`QtConcurrent::IterateKernel<std::__1::__wrap_iter<LayerRenderJob*>, 
void>::threadFunction(this=0x0000600002eeb080) at 
qtconcurrentiteratekernel.h:217:26
    frame #25: 0x0000000100275499 
QtConcurrent`QtConcurrent::ThreadEngineBase::run() + 185
    frame #26: 0x0000000104e7daec QtCore`QThreadPoolThread::run() + 124
    frame #27: 0x0000000104e79619 QtCore`QThreadPrivate::start(void*) + 
329
    frame #28: 0x00007ff81311d4e1 libsystem_pthread.dylib`_pthread_start + 
125
    frame #29: 0x00007ff813118f6b libsystem_pthread.dylib`thread_start + 
15
kittybwained commented 8 months ago

Getting the same thing on FreeBSD 14.0-RELEASE-p5 with both qgis and qgis-ltr (installed from pkg). Here's the backtrace from the coredump of qgis (3.34 Prizren):

#0  0x000000087450386a in thr_kill () at /lib/libc.so.7
#1  0x000000087447bf54 in raise () at /lib/libc.so.7
#2  0x000000085da5053f in  () at /lib/libthr.so.3
#3  0x000000085da4fafb in  () at /lib/libthr.so.3
#4  0x0000000820e132d3 in <signal handler called> ()
#5  0x000000087450386a in thr_kill () at /lib/libc.so.7
#6  0x000000087447bf54 in raise () at /lib/libc.so.7
#7  0x000000087452cfc9 in abort () at /lib/libc.so.7
#8  0x000000000024de37 in  ()
#9  0x000000085da5053f in  () at /lib/libthr.so.3
#10 0x000000085da4fafb in  () at /lib/libthr.so.3
#11 0x0000000820e132d3 in <signal handler called> ()
#12 0x000000088dd4a6c2 in H5FL_reg_malloc () at /usr/local/lib/libhdf5.so.200
#13 0x000000088de5dc3b in H5SL_create () at /usr/local/lib/libhdf5.so.200
#14 0x000000088de1a5ba in H5P_create_id () at /usr/local/lib/libhdf5.so.200
#15 0x000000088ddef674 in H5Pcreate () at /usr/local/lib/libhdf5.so.200
#16 0x000000083baadcf2 in  () at /usr/local/lib/libgdal.so.34
#17 0x000000083bab3ebd in  () at /usr/local/lib/libgdal.so.34
#18 0x000000083c100ce5 in GDALDriver::Open(GDALOpenInfo*, bool) () at /usr/local/lib/libgdal.so.34
#19 0x000000083c111204 in GDALOpenEx () at /usr/local/lib/libgdal.so.34
#20 0x0000000834896e4e in  () at /usr/local/lib/libqgis_core.so.3.34.3
#21 0x000000083489adac in  () at /usr/local/lib/libqgis_core.so.3.34.3
#22 0x000000083489f315 in  () at /usr/local/lib/libqgis_core.so.3.34.3
#23 0x0000000834f7bbb2 in QgsSingleBandGrayRenderer::block(int, QgsRectangle const&, int, int, QgsRasterBlockFeedback*) () at /usr/local/lib/libqgis_core.so.3.34.3
#24 0x0000000834f5a201 in QgsBrightnessContrastFilter::block(int, QgsRectangle const&, int, int, QgsRasterBlockFeedback*) () at /usr/local/lib/libqgis_core.so.3.34.3
#25 0x0000000834f5b28b in QgsHueSaturationFilter::block(int, QgsRectangle const&, int, int, QgsRasterBlockFeedback*) () at /usr/local/lib/libqgis_core.so.3.34.3
#26 0x0000000834f79e66 in QgsRasterResampleFilter::block(int, QgsRectangle const&, int, int, QgsRasterBlockFeedback*) () at /usr/local/lib/libqgis_core.so.3.34.3
#27 0x0000000834f33067 in QgsRasterIterator::readNextRasterPartInternal(int, int&, int&, std::__1::unique_ptr<QgsRasterBlock, std::__1::default_delete<QgsRasterBlock> >*, int&, int&, QgsRectangle*, int&, int&, int&, int&) () at /usr/local/lib/libqgis_core.so.3.34.3
#28 0x0000000834f33210 in QgsRasterIterator::readNextRasterPart(int, int&, int&, std::__1::unique_ptr<QgsRasterBlock, std::__1::default_delete<QgsRasterBlock> >&, int&, int&, QgsRectangle*, int*, int*, int*, int*) () at /usr/local/lib/libqgis_core.so.3.34.3
#29 0x0000000834f6a140 in QgsRasterDrawer::draw(QPainter*, QgsRasterViewPort*, QgsMapToPixel const*, QgsRasterBlockFeedback*) () at /usr/local/lib/libqgis_core.so.3.34.3
#30 0x0000000834f4a07c in QgsRasterLayerRenderer::render() () at /usr/local/lib/libqgis_core.so.3.34.3
#31 0x0000000834df92ab in QgsMapRendererParallelJob::renderLayerStatic(LayerRenderJob&) () at /usr/local/lib/libqgis_core.so.3.34.3
#32 0x0000000834dfa33d in  () at /usr/local/lib/libqgis_core.so.3.34.3
#33 0x0000000834dfa392 in  () at /usr/local/lib/libqgis_core.so.3.34.3
#34 0x0000000834dfa504 in  () at /usr/local/lib/libqgis_core.so.3.34.3
#35 0x0000000835fe8640 in QtConcurrent::ThreadEngineBase::run() () at /usr/local/lib/qt5/libQt5Concurrent.so.5
#36 0x000000082a472302 in  () at /usr/local/lib/qt5/libQt5Core.so.5
#37 0x000000082a46e0b9 in  () at /usr/local/lib/qt5/libQt5Core.so.5
#38 0x000000085da46a75 in  () at /lib/libthr.so.3
#39 0x0000000000000000 in  ()
vbrancat commented 7 months ago

It seems that this issue might be resolved by downloading the new version of the HDF5 files from the HDF Group page and replace them in the qGIS package. So it seems that the solution for this issue is just to update the HDF5 drivers on Mac and Linux platforms.

leonard-leo commented 5 months ago

@vbrancat Can you give us more details on what you did to solve this issue? QGIS v.3.34.7 crashes on Mac OS 14.4.1 when trying to open an h5 downloaded from the NASA Earth Explorer (similar to what the user g-studer-ellis had described).

g8sqh commented 2 months ago

in https://github.com/qgis/QGIS/issues/30054 I raised a similar issue. You need to build libhdf5 with threadsafe turned on. I'm not a Mac person, so don't know if this is how Mac builds it, but it's essential for QGIS to work well with multiple HDF5 (in my case BAG) files