Opening NetCDF makes computer crash

[EmileSonneveld]

What is the bug or the crash?

When I open this file with QGIS, my whole user session crashes. I get a black screen and need to log in again. I think it is due to an over de top amount of RAM being allocated: https://emilesonneveld.be/misc/Sentinel2_makes_linux_user_session_crash.nc (Save any work before opening this NetCDF)

Steps to reproduce the issue

1. Open the attached file: https://emilesonneveld.be/misc/Sentinel2_makes_linux_user_session_crash.nc
2. Wait
3. Experience a crash

Versions

3.22.4-Białowieża

Supported QGIS version

• [X] I'm running a supported QGIS version according to the roadmap.

New profile

• [X] I tried with a new QGIS profile

Additional context

I made a new profile and added the standard OpenStreetMap as background. Still got the crash I run Ubuntu with kernel 6.2.0-39-generic My colleague runs Linux mint and has the same problem

[agiudiceandrea]

@EmileSonneveld, thanks for reporting. QGIS 3.22 is no longer supported. May you please actually try if the issue occurs also using a currently supported QGIS version (QGIS 3.34.2, QGIS 3.28.14) and provide the full version info ("In the QGIS Help menu -> About, click in the table, Ctrl+A and then Ctrl+C. Finally paste here. Do not make a screenshot.") as requested in the issue report form?

[rouault]

The huge RAM consumption arises when opening the file with "Add mesh layer" . It works fine with "Add raster layer" (using GDAL netCDF driver)

[rouault]

@PeterPetrik Debugging this, I see QGIS spends its time in MDAL::DriverGdal::initFaces(). On this dataset where width = 16831 and height = 12992, and my understanding of "typedef std::vector Faces;" and "typedef std::vector Face;", my estimate of the minimum amount of RAM required is 16830 12991 (sizeof(size_t) + sizeof(void) + 4 sizeof(size_t)) ~= 10 GB, and probably more due to memory fragmentation given we allocate lots of Face elements one at a time. As adding piece-wise loading of data is probably a difficult undertaking, it seems that MDAL should bail out on a GDAL dataset beyond some dimension, likely correlated to available RAM (there's a GDAL CPLGetUsablePhysicalRAM() function for that)

[EmileSonneveld]

Loading it without the mesh indeed avoids the problem! Can I disable mesh creation by default in QGIS? Or configure the NetCDF to avoid mesh creation? The ram usage could indeed be 10Gb+. That's about the amount that was free at time of crash.

[PeterPetrik]

Hi and thanks for report and debugging: created MDAL issue for this https://github.com/lutraconsulting/MDAL/issues/476

[EmileSonneveld]

In 3.34.2-Prizren, only QGIS crashes without error message. Which is quite better. It allows to try again without the mesh

[JaqucesT]

Opening this file in 3.34.3 memory consumption rises till 36 GB in steps and then it starts rising in a linear way as if there is a memory leak. When QGis reaches 50 GB of memory consumption i get a "Bad allocation" error, but the program stays alive. Removing this layer then takes a long time where you can see memory consumption drop to some normal 350 MB.

[rouault]

Another instance of that issue in https://github.com/qgis/QGIS/issues/56786#issuecomment-1987378784