Opening NetCDF/mesh file consumed all the memory

tschmetzer commented 2 years ago

What is the bug or the crash?

QGIS throws a bad allocation error when trying to load a layer of a NetCDF file from a public source

Steps to reproduce the issue

Download the gridded population file _PBL_IMAGE_SSP2_population_density5min.nc or _PBL_IMAGE_SSP2_SPA2_RCP_2.6_population_density5min.nc from https://dataplatform.knmi.nl/dataset/ssps-gpopd-v17 and drag it into the layers panel. I tried only with these two files but the other files might show the same behaviour.
Select (at least) the second entry to import
Get a bad allocation error

Versions

QGIS version 3.22.0-Białowieża QGIS code revision d9022691f1 Qt version 5.15.2 Python version 3.9.5 GDAL/OGR version 3.4.0 PROJ version 8.2.0 EPSG Registry database version v10.038 (2021-10-21) GEOS version 3.10.0-CAPI-1.16.0 SQLite version 3.35.2 PDAL version 2.3.0 PostgreSQL client version 13.0 SpatiaLite version 5.0.1 QWT version 6.1.3 QScintilla2 version 2.11.5 OS version Windows 10 Version 2009

Active Python plugins GroupStats 2.2.5 Qgis2threejs 2.6 QuickOSM 2.0.0 db_manager 0.1.20 grassprovider 2.12.99 MetaSearch 0.3.5 processing 2.12.99 sagaprovider 2.12.99

Supported QGIS version

[X] I'm running a supported QGIS version according to the roadmap.

New profile

[x] I tried with a new QGIS profile

Additional context

No response

gioman commented 2 years ago

1. from https://dataplatform.knmi.nl/dataset/ssps-gpopd-v17

@tschmetzer I can't see a download link, just an API address which is not very convenient for who wants to do a quick test. Can you attach here the dataset?

gioman commented 2 years ago

@tschmetzer I can't see a download link, just an API address which is not very convenient for who wants to do a quick test. Can you attach here the dataset?

@tschmetzer disregard, I see the download links now.

gioman commented 2 years ago

@tschmetzer here the behaviour seems different: adding the dataset with d&d and choosing the mesh layer (the middle one) it brings immediately QGIS to a freeze.

tschmetzer commented 2 years ago

disregard

I forgot to mention that you need to press "Toggle browser" underneath the "Broswe dataset" heading. Great you found it yourself :)

tschmetzer commented 2 years ago

Closing several other applications, closing other QGIS instances and restarting QGIS -> the layer is displayed. Yet even with only one instance but several other layers already loaded there is a crash or mem alloc exception. Same bahaviour with several instances but loading only the NetCDF file and no other layers into a new project. Seems like a memory issue to me.

vcloarec commented 2 years ago

Trying with 3.22.0 under ubuntu, I can open it but... consume a lot of memory. I think the freeze come from the size of the mesh (more than 9 millions faces) and the time needed to load it. And the eventual crash could come form not enough RAM for this mesh. On my side, I need nearly 10 GBytes to open it. @tschmetzer, can you monitor your RAM when loading the file with the Resource Monitor?

tschmetzer commented 2 years ago

Thank you @vcloarec. You seem to be right.

On the machine with a lot of memory it's not a problem to open several instances with one instance loading the mesh data. The mesh adds 7.1 GB memory which sums up to 8.5 GB for that mesh holding QGIS instance.
On the machine with only 8 GB and other applications and instances already consuming 5-7 GB of the memory GIS crashes and even makes several (!) other applications like Chrome crash at the same time.

My notions:

Why and how does a too large layer/mesh crash the hole application? Is there any chance to abort the loading before leading to a crash? That shouldn't happen. Or is there something else wrong? Even with several very large vector or raster layers I never encountered a crash.
And if there is a crash how is it possible other applications are involved as well? According to my understanding the OS shouldn't allow this to happen.
Should we make the "Bad allocation" informations more userfriendly like "Not enough memory available" or "What you want to do consumes more memory than currently available. Consider closing other applications. Need to abort."

qgis / QGIS