search

SciTools / iris

A powerful, format-agnostic, and community-driven Python package for analysing and visualising Earth science data
https://scitools-iris.readthedocs.io/en/stable/
BSD 3-Clause "New" or "Revised" License
635 stars 283 forks source link

Crash (segfault) when loading specific files #6023

Closed jfrost-mo closed 4 months ago

jfrost-mo commented 4 months ago

🐛 Bug Report

I've got some multi-level data that crashes python when loaded with iris. The last file (lfric_struct_048.nc) appears to be corrupt, and raises an IO/OSError when loaded induvidully. When the files are loaded together with iris.load("*.nc") it segfaults.

How To Reproduce

Steps to reproduce the behaviour:

  1. Be within the Met Office linux estate
  2. module load scitools/default-next
  3. python3 -c 'import iris; print(iris.load("/data/users/raltrials/u-de155/20190725T0000Z/lfric_multilevel_slam_only/*.nc"))'
  4. Observe the segmentation fault.

Expected behaviour

It does not crash the python interpreter, and instead raises an OSError that can be handled appropriately.

Environment

Additional context

Tracebacks. Click to expand this section... Segfault when all are loaded: ``` ❯ python3 -c 'import iris; print(iris.load("/data/users/raltrials/u-de155/20190725T0000Z/lfric_multilevel_slam_only/*.nc"))' /net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_05_28/lib/python3.11/site-packages/iris/__init__.py:326: FutureWarning: Ignoring a datum in netCDF load for consistency with existing behaviour. In a future version of Iris, this datum will be applied. To apply the datum when loading, use the iris.FUTURE.datum_support flag. return _load_collection(uris, constraints, callback).merged().cubes() *** Error in `python3': double free or corruption (out): 0x0000556569651510 *** ======= Backtrace: ========= /usr/lib64/libc.so.6(+0x81329)[0x2aba05f09329] /net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_05_28/lib/python3.11/site-packages/fiona/../../.././libhdf5.so.310(H5MM_xfree+0xa)[0x2aba3100fcca] /net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_05_28/lib/python3.11/site-packages/fiona/../../.././libhdf5.so.310(+0x1d9dd6)[0x2aba31043dd6] /net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_05_28/lib/python3.11/site-packages/fiona/../../.././libhdf5.so.310(H5O_msg_reset+0x24)[0x2aba31045d34] /net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_05_28/lib/python3.11/site-packages/fiona/../../.././libhdf5.so.310(H5G__link_release_table+0x40)[0x2aba30fc8b10] /net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_05_28/lib/python3.11/site-packages/fiona/../../.././libhdf5.so.310(H5G__dense_iterate+0x1a5)[0x2aba30fbfca5] /net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_05_28/lib/python3.11/site-packages/fiona/../../.././libhdf5.so.310(H5G__obj_iterate+0xb7)[0x2aba30fcf437] /net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_05_28/lib/python3.11/site-packages/fiona/../../.././libhdf5.so.310(H5G_iterate+0xae)[0x2aba30fc75fe] /net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_05_28/lib/python3.11/site-packages/fiona/../../.././libhdf5.so.310(H5L_iterate+0x4e)[0x2aba31008c0e] /net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_05_28/lib/python3.11/site-packages/fiona/../../.././libhdf5.so.310(H5VL__native_link_specific+0x2a5)[0x2aba3116d1d5] /net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_05_28/lib/python3.11/site-packages/fiona/../../.././libhdf5.so.310(+0x2e8794)[0x2aba31152794] /net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_05_28/lib/python3.11/site-packages/fiona/../../.././libhdf5.so.310(H5VL_link_specific+0x47)[0x2aba3115d757] /net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_05_28/lib/python3.11/site-packages/fiona/../../.././libhdf5.so.310(+0x1936db)[0x2aba30ffd6db] /net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_05_28/lib/python3.11/site-packages/fiona/../../.././libhdf5.so.310(H5Literate2+0x8f)[0x2aba31001b7f] /net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_05_28/lib/python3.11/site-packages/fiona/../../.././libnetcdf.so.19(+0xaf20e)[0x2aba309aa20e] /net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_05_28/lib/python3.11/site-packages/fiona/../../.././libnetcdf.so.19(+0xb2238)[0x2aba309ad238] /net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_05_28/lib/python3.11/site-packages/fiona/../../.././libnetcdf.so.19(NC_open+0x2c2)[0x2aba3092e3f2] /net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_05_28/lib/python3.11/site-packages/fiona/../../.././libnetcdf.so.19(nc_open+0x15)[0x2aba3092e4d5] /net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_05_28/lib/python3.11/site-packages/netCDF4/_netCDF4.cpython-311-x86_64-linux-gnu.so(+0xf8836)[0x2aba36b71836] python3(+0x1e0651)[0x556565bac651] python3(PyObject_Call+0xbe)[0x556565be716e] python3(_PyEval_EvalFrameDefault+0x49f9)[0x556565bbe119] python3(_PyFunction_Vectorcall+0x181)[0x556565bdd4c1] python3(+0x218933)[0x556565be4933] python3(_PyObject_MakeTpCall+0x233)[0x556565bac303] python3(_PyEval_EvalFrameDefault+0x716)[0x556565bb9e36] python3(_PyFunction_Vectorcall+0x181)[0x556565bdd4c1] python3(+0x21886c)[0x556565be486c] python3(_PyObject_MakeTpCall+0x233)[0x556565bac303] python3(_PyEval_EvalFrameDefault+0x716)[0x556565bb9e36] python3(+0x22e5f2)[0x556565bfa5f2] python3(_PyEval_EvalFrameDefault+0x5a2)[0x556565bb9cc2] python3(+0x22e5f2)[0x556565bfa5f2] python3(_PyEval_EvalFrameDefault+0x5a2)[0x556565bb9cc2] python3(+0x22e5f2)[0x556565bfa5f2] python3(_PyEval_EvalFrameDefault+0x5a2)[0x556565bb9cc2] python3(+0x2a442d)[0x556565c7042d] python3(PyEval_EvalCode+0x9f)[0x556565c6fabf] python3(+0x2c2a1a)[0x556565c8ea1a] python3(+0x2be593)[0x556565c8a593] python3(PyRun_StringFlags+0x62)[0x556565c7ed62] python3(PyRun_SimpleStringFlags+0x3c)[0x556565c7eb1c] python3(Py_RunMain+0x23f)[0x556565c995bf] python3(Py_BytesMain+0x37)[0x556565c5fa77] /usr/lib64/libc.so.6(__libc_start_main+0xf5)[0x2aba05eaa555] python3(+0x29391d)[0x556565c5f91d] ======= Memory map: ======== 2aba0533f000-2aba05361000 r-xp 00000000 fd:00 8388717 /usr/lib64/ld-2.17.so 2aba05361000-2aba05363000 rw-p 00000000 00:00 0 2aba05363000-2aba0536a000 r--s 00000000 fd:00 27945249 /usr/lib64/gconv/gconv-modules.cache 2aba0536a000-2aba0536e000 rw-p 00000000 00:00 0 2aba0536e000-2aba05372000 r--p 00000000 00:32 13835060284556009420 /net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_05_28/lib/python3.11/lib-dynload/math.cpython-311-x86_64-linux-gnu.so 2aba05372000-2aba05379000 r-xp 00004000 00:32 13835060284556009420 /net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_05_28/lib/python3.11/lib-dynload/math.cpython-311-x86_64-linux-gnu.so 2aba05379000-2aba0537e000 r--p 0000b000 00:32 13835060284556009420 /net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_05_28/lib/python3.11/lib-dynload/math.cpython-311-x86_64-linux-gnu.so 2aba0537e000-2aba0537f000 r--p 0000f000 00:32 13835060284556009420 /net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_05_28/lib/python3.11/lib-dynload/math.cpython-311-x86_64-linux-gnu.so 2aba0537f000-2aba05380000 rw-p 00010000 00:32 13835060284556009420 /net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_05_28/lib/python3.11/lib-dynload/math.cpython-311-x86_64-linux-gnu.soAborted ``` Exception when one is loaded: ``` ❯ python3 -c 'import iris; print(iris.load("/data/users/raltrials/u-de155/20190725T0000Z/lfric_multilevel_slam_only/lfric_struct_048.nc"))' Traceback (most recent call last): File "", line 1, in File "/net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_05_28/lib/python3.11/site-packages/iris/__init__.py", line 326, in load return _load_collection(uris, constraints, callback).merged().cubes() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_05_28/lib/python3.11/site-packages/iris/__init__.py", line 294, in _load_collection result = _CubeFilterCollection.from_cubes(cubes, constraints) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_05_28/lib/python3.11/site-packages/iris/cube.py", line 97, in from_cubes for cube in cubes: File "/net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_05_28/lib/python3.11/site-packages/iris/__init__.py", line 275, in _generate_cubes for cube in iris.io.load_files(part_names, callback, constraints): File "/net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_05_28/lib/python3.11/site-packages/iris/io/__init__.py", line 206, in load_files all_file_paths = expand_filespecs(filenames) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_05_28/lib/python3.11/site-packages/iris/io/__init__.py", line 184, in expand_filespecs raise IOError(msg) OSError: One or more of the files specified did not exist: * "/data/users/raltrials/u-de155/20190725T0000Z/lfric_multilevel_slam_only/lfric_struct_048.nc" didn't match any files ```
jfrost-mo commented 4 months ago

OK, so I might have been a bit quick with the bug report. The data in question was only partially written, and has since been overwritten with the complete data, making this hard to reproduce. As such I'm going to close this issue.