Closed jfrost-mo closed 4 months ago
OK, so I might have been a bit quick with the bug report. The data in question was only partially written, and has since been overwritten with the complete data, making this hard to reproduce. As such I'm going to close this issue.
🐛 Bug Report
I've got some multi-level data that crashes python when loaded with iris. The last file (lfric_struct_048.nc) appears to be corrupt, and raises an IO/OSError when loaded induvidully. When the files are loaded together with
iris.load("*.nc")
it segfaults.How To Reproduce
Steps to reproduce the behaviour:
module load scitools/default-next
python3 -c 'import iris; print(iris.load("/data/users/raltrials/u-de155/20190725T0000Z/lfric_multilevel_slam_only/*.nc"))'
Expected behaviour
It does not crash the python interpreter, and instead raises an OSError that can be handled appropriately.
Environment
Additional context
Tracebacks. Click to expand this section...
Segfault when all are loaded: ``` ❯ python3 -c 'import iris; print(iris.load("/data/users/raltrials/u-de155/20190725T0000Z/lfric_multilevel_slam_only/*.nc"))' /net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_05_28/lib/python3.11/site-packages/iris/__init__.py:326: FutureWarning: Ignoring a datum in netCDF load for consistency with existing behaviour. In a future version of Iris, this datum will be applied. To apply the datum when loading, use the iris.FUTURE.datum_support flag. return _load_collection(uris, constraints, callback).merged().cubes() *** Error in `python3': double free or corruption (out): 0x0000556569651510 *** ======= Backtrace: ========= /usr/lib64/libc.so.6(+0x81329)[0x2aba05f09329] /net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_05_28/lib/python3.11/site-packages/fiona/../../.././libhdf5.so.310(H5MM_xfree+0xa)[0x2aba3100fcca] /net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_05_28/lib/python3.11/site-packages/fiona/../../.././libhdf5.so.310(+0x1d9dd6)[0x2aba31043dd6] /net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_05_28/lib/python3.11/site-packages/fiona/../../.././libhdf5.so.310(H5O_msg_reset+0x24)[0x2aba31045d34] /net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_05_28/lib/python3.11/site-packages/fiona/../../.././libhdf5.so.310(H5G__link_release_table+0x40)[0x2aba30fc8b10] /net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_05_28/lib/python3.11/site-packages/fiona/../../.././libhdf5.so.310(H5G__dense_iterate+0x1a5)[0x2aba30fbfca5] /net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_05_28/lib/python3.11/site-packages/fiona/../../.././libhdf5.so.310(H5G__obj_iterate+0xb7)[0x2aba30fcf437] /net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_05_28/lib/python3.11/site-packages/fiona/../../.././libhdf5.so.310(H5G_iterate+0xae)[0x2aba30fc75fe] /net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_05_28/lib/python3.11/site-packages/fiona/../../.././libhdf5.so.310(H5L_iterate+0x4e)[0x2aba31008c0e] /net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_05_28/lib/python3.11/site-packages/fiona/../../.././libhdf5.so.310(H5VL__native_link_specific+0x2a5)[0x2aba3116d1d5] /net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_05_28/lib/python3.11/site-packages/fiona/../../.././libhdf5.so.310(+0x2e8794)[0x2aba31152794] /net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_05_28/lib/python3.11/site-packages/fiona/../../.././libhdf5.so.310(H5VL_link_specific+0x47)[0x2aba3115d757] /net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_05_28/lib/python3.11/site-packages/fiona/../../.././libhdf5.so.310(+0x1936db)[0x2aba30ffd6db] /net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_05_28/lib/python3.11/site-packages/fiona/../../.././libhdf5.so.310(H5Literate2+0x8f)[0x2aba31001b7f] /net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_05_28/lib/python3.11/site-packages/fiona/../../.././libnetcdf.so.19(+0xaf20e)[0x2aba309aa20e] /net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_05_28/lib/python3.11/site-packages/fiona/../../.././libnetcdf.so.19(+0xb2238)[0x2aba309ad238] /net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_05_28/lib/python3.11/site-packages/fiona/../../.././libnetcdf.so.19(NC_open+0x2c2)[0x2aba3092e3f2] /net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_05_28/lib/python3.11/site-packages/fiona/../../.././libnetcdf.so.19(nc_open+0x15)[0x2aba3092e4d5] /net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_05_28/lib/python3.11/site-packages/netCDF4/_netCDF4.cpython-311-x86_64-linux-gnu.so(+0xf8836)[0x2aba36b71836] python3(+0x1e0651)[0x556565bac651] python3(PyObject_Call+0xbe)[0x556565be716e] python3(_PyEval_EvalFrameDefault+0x49f9)[0x556565bbe119] python3(_PyFunction_Vectorcall+0x181)[0x556565bdd4c1] python3(+0x218933)[0x556565be4933] python3(_PyObject_MakeTpCall+0x233)[0x556565bac303] python3(_PyEval_EvalFrameDefault+0x716)[0x556565bb9e36] python3(_PyFunction_Vectorcall+0x181)[0x556565bdd4c1] python3(+0x21886c)[0x556565be486c] python3(_PyObject_MakeTpCall+0x233)[0x556565bac303] python3(_PyEval_EvalFrameDefault+0x716)[0x556565bb9e36] python3(+0x22e5f2)[0x556565bfa5f2] python3(_PyEval_EvalFrameDefault+0x5a2)[0x556565bb9cc2] python3(+0x22e5f2)[0x556565bfa5f2] python3(_PyEval_EvalFrameDefault+0x5a2)[0x556565bb9cc2] python3(+0x22e5f2)[0x556565bfa5f2] python3(_PyEval_EvalFrameDefault+0x5a2)[0x556565bb9cc2] python3(+0x2a442d)[0x556565c7042d] python3(PyEval_EvalCode+0x9f)[0x556565c6fabf] python3(+0x2c2a1a)[0x556565c8ea1a] python3(+0x2be593)[0x556565c8a593] python3(PyRun_StringFlags+0x62)[0x556565c7ed62] python3(PyRun_SimpleStringFlags+0x3c)[0x556565c7eb1c] python3(Py_RunMain+0x23f)[0x556565c995bf] python3(Py_BytesMain+0x37)[0x556565c5fa77] /usr/lib64/libc.so.6(__libc_start_main+0xf5)[0x2aba05eaa555] python3(+0x29391d)[0x556565c5f91d] ======= Memory map: ======== 2aba0533f000-2aba05361000 r-xp 00000000 fd:00 8388717 /usr/lib64/ld-2.17.so 2aba05361000-2aba05363000 rw-p 00000000 00:00 0 2aba05363000-2aba0536a000 r--s 00000000 fd:00 27945249 /usr/lib64/gconv/gconv-modules.cache 2aba0536a000-2aba0536e000 rw-p 00000000 00:00 0 2aba0536e000-2aba05372000 r--p 00000000 00:32 13835060284556009420 /net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_05_28/lib/python3.11/lib-dynload/math.cpython-311-x86_64-linux-gnu.so 2aba05372000-2aba05379000 r-xp 00004000 00:32 13835060284556009420 /net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_05_28/lib/python3.11/lib-dynload/math.cpython-311-x86_64-linux-gnu.so 2aba05379000-2aba0537e000 r--p 0000b000 00:32 13835060284556009420 /net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_05_28/lib/python3.11/lib-dynload/math.cpython-311-x86_64-linux-gnu.so 2aba0537e000-2aba0537f000 r--p 0000f000 00:32 13835060284556009420 /net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_05_28/lib/python3.11/lib-dynload/math.cpython-311-x86_64-linux-gnu.so 2aba0537f000-2aba05380000 rw-p 00010000 00:32 13835060284556009420 /net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_05_28/lib/python3.11/lib-dynload/math.cpython-311-x86_64-linux-gnu.soAborted ``` Exception when one is loaded: ``` ❯ python3 -c 'import iris; print(iris.load("/data/users/raltrials/u-de155/20190725T0000Z/lfric_multilevel_slam_only/lfric_struct_048.nc"))' Traceback (most recent call last): File "