ImportError when compiling with gcc 14.1.1 and conda

cphyc commented 3 months ago

Bug report

Bug summary

The code compiles but fails when being imported.

Code for reproduction

python -c "import yt"

Actual outcome

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "[...]/yt/yt/__init__.py", line 14, in <module>
    from yt.data_objects.api import (
  File "[...]/yt/yt/data_objects/api.py", line 1, in <module>
    from . import construction_data_containers as __cdc, selection_objects as __sdc
  File "[...]/yt/yt/data_objects/construction_data_containers.py", line 17, in <module>
    from yt.data_objects.selection_objects.data_selection_objects import (
  File "[...]/yt/yt/data_objects/selection_objects/__init__.py", line 1, in <module>
    from .boolean_operations import (
  File "[...]/yt/yt/data_objects/selection_objects/boolean_operations.py", line 5, in <module>
    from yt.data_objects.selection_objects.data_selection_objects import (
  File "[...]/yt/yt/data_objects/selection_objects/data_selection_objects.py", line 14, in <module>
    from yt.data_objects.data_containers import YTDataContainer
  File "[...]/yt/yt/data_objects/data_containers.py", line 13, in <module>
    from yt.data_objects.profiles import create_profile
  File "[...]/yt/yt/data_objects/profiles.py", line 5, in <module>
    from yt.fields.derived_field import DerivedField
  File "[...]/yt/yt/fields/derived_field.py", line 18, in <module>
    from .field_detector import FieldDetector
  File "[...]/yt/yt/fields/field_detector.py", line 7, in <module>
    from yt.utilities.io_handler import io_registry
  File "[...]/yt/yt/utilities/io_handler.py", line 10, in <module>
    from yt.geometry.selection_routines import GridSelector
  File "yt/geometry/selection_routines.pyx", line 1, in init yt.geometry.selection_routines
    # distutils: include_dirs = LIB_DIR
ImportError: [...]/yt/yt/utilities/lib/image_samplers.cpython-312-x86_64-linux-gnu.so: undefined symbol: __cxa_call_terminate

Expected outcome

No error.

Version Information

Operating System: Linux
Python Version: 3.12.3
yt version: dev
Other Libraries (if applicable): python installed from conda environment with conda-forge channel.

Other pieces of information:

No issue when using the system-wide Python 3.12 installation
No issue if I compile with gcc-13 and g++-13 (CC=gcc-13 CXX=g++-13 python -m pip install -e yt).

The last point suggests some incompatibility between the conda-provided libstdcxx-ng (on version 13.2) install and the system-wide one (on version 14.1).

Details

**Conda environment** ``` # packages in environment at [...]/miniforge3/envs/py312: # # Name Version Build Channel _libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 2_gnu conda-forge asttokens 2.4.1 pyhd8ed1ab_0 conda-forge blosc 1.21.5 hc2324a3_1 conda-forge brotli 1.1.0 hd590300_1 conda-forge brotli-bin 1.1.0 hd590300_1 conda-forge brotli-python 1.1.0 py312h30efb56_1 conda-forge bzip2 1.0.8 hd590300_5 conda-forge c-ares 1.28.1 hd590300_0 conda-forge ca-certificates 2024.6.2 hbcca054_0 conda-forge certifi 2024.6.2 pyhd8ed1ab_0 conda-forge cftime 1.6.3 py312h085067d_1 conda-forge charset-normalizer 3.3.2 pyhd8ed1ab_0 conda-forge cmyt 2.0.0 pyhd8ed1ab_0 conda-forge colorama 0.4.6 pyhd8ed1ab_0 conda-forge colossus 1.3.5 pypi_0 pypi comm 0.2.2 pyhd8ed1ab_0 conda-forge contourpy 1.2.1 py312h8572e83_0 conda-forge cycler 0.12.1 pyhd8ed1ab_0 conda-forge cython 3.0.10 py312h30efb56_0 conda-forge debugpy 1.8.1 py312h30efb56_0 conda-forge decorator 5.1.1 pyhd8ed1ab_0 conda-forge et_xmlfile 1.1.0 pyhd8ed1ab_0 conda-forge ewah-bool-utils 1.2.1 py312h1d6d2e6_0 conda-forge exceptiongroup 1.2.0 pyhd8ed1ab_2 conda-forge executing 2.0.1 pyhd8ed1ab_0 conda-forge fonttools 4.53.0 py312h9a8786e_0 conda-forge freetype 2.12.1 h267a509_2 conda-forge gmp 6.3.0 h59595ed_1 conda-forge gmpy2 2.1.5 py312h1d5cde6_1 conda-forge hdf4 4.2.15 h2a13503_7 conda-forge hdf5 1.14.3 nompi_hdf9ad27_105 conda-forge icu 73.2 h59595ed_0 conda-forge idna 3.7 pyhd8ed1ab_0 conda-forge importlib-metadata 7.1.0 pyha770c72_0 conda-forge importlib_metadata 7.1.0 hd8ed1ab_0 conda-forge ipykernel 6.29.4 pyh3099207_0 conda-forge ipython 8.25.0 pyh707e725_0 conda-forge ipywidgets 8.1.3 pyhd8ed1ab_0 conda-forge jedi 0.19.1 pyhd8ed1ab_0 conda-forge jupyter_client 8.6.2 pyhd8ed1ab_0 conda-forge jupyter_core 5.7.2 py312h7900ff3_0 conda-forge jupyterlab_widgets 3.0.11 pyhd8ed1ab_0 conda-forge keyutils 1.6.1 h166bdaf_0 conda-forge kiwisolver 1.4.5 py312h8572e83_1 conda-forge krb5 1.21.2 h659d440_0 conda-forge lcms2 2.16 hb7c19ff_0 conda-forge ld_impl_linux-64 2.40 hf3520f5_3 conda-forge lerc 4.0.0 h27087fc_0 conda-forge libaec 1.1.3 h59595ed_0 conda-forge libblas 3.9.0 22_linux64_openblas conda-forge libbrotlicommon 1.1.0 hd590300_1 conda-forge libbrotlidec 1.1.0 hd590300_1 conda-forge libbrotlienc 1.1.0 hd590300_1 conda-forge libcblas 3.9.0 22_linux64_openblas conda-forge libcurl 8.8.0 hca28451_0 conda-forge libdeflate 1.20 hd590300_0 conda-forge libedit 3.1.20191231 he28a2e2_2 conda-forge libev 4.33 hd590300_2 conda-forge libexpat 2.6.2 h59595ed_0 conda-forge libffi 3.4.2 h7f98852_5 conda-forge libgcc-ng 13.2.0 h77fa898_7 conda-forge libgfortran-ng 13.2.0 h69a702a_7 conda-forge libgfortran5 13.2.0 hca663fb_7 conda-forge libgomp 13.2.0 h77fa898_7 conda-forge libiconv 1.17 hd590300_2 conda-forge libjpeg-turbo 3.0.0 hd590300_1 conda-forge liblapack 3.9.0 22_linux64_openblas conda-forge libnetcdf 4.9.2 nompi_h135f659_114 conda-forge libnghttp2 1.58.0 h47da74e_1 conda-forge libnsl 2.0.1 hd590300_0 conda-forge libopenblas 0.3.27 pthreads_h413a1c8_0 conda-forge libpng 1.6.43 h2797004_0 conda-forge libsodium 1.0.18 h36c2ea0_1 conda-forge libsqlite 3.45.3 h2797004_0 conda-forge libssh2 1.11.0 h0841786_0 conda-forge libstdcxx-ng 13.2.0 hc0a3c3a_7 conda-forge libtiff 4.6.0 h1dd3fc0_3 conda-forge libuuid 2.38.1 h0b41bf4_0 conda-forge libwebp-base 1.4.0 hd590300_0 conda-forge libxcb 1.15 h0b41bf4_0 conda-forge libxcrypt 4.4.36 hd590300_1 conda-forge libxml2 2.12.7 hc051c1a_1 conda-forge libzip 1.10.1 h2629f0a_3 conda-forge libzlib 1.3.1 h4ab18f5_1 conda-forge lz4-c 1.9.4 hcb278e6_0 conda-forge matplotlib-base 3.8.4 py312h20ab3a6_2 conda-forge matplotlib-inline 0.1.7 pyhd8ed1ab_0 conda-forge matplotlib-label-lines 0.7.0 pypi_0 pypi more-itertools 10.2.0 pyhd8ed1ab_0 conda-forge mpc 1.3.1 hfe3b2da_0 conda-forge mpfr 4.2.1 h9458935_1 conda-forge mpmath 1.3.0 pyhd8ed1ab_0 conda-forge munkres 1.1.4 pyh9f0ad1d_0 conda-forge ncurses 6.5 h59595ed_0 conda-forge nest-asyncio 1.6.0 pyhd8ed1ab_0 conda-forge netcdf4 1.6.5 nompi_py312h39d4375_102 conda-forge numpy 1.26.4 py312heda63a1_0 conda-forge openjpeg 2.5.2 h488ebb8_0 conda-forge openpyxl 3.1.2 py312h98912ed_1 conda-forge openssl 3.3.1 h4ab18f5_0 conda-forge packaging 24.0 pyhd8ed1ab_0 conda-forge pandas 2.2.2 py312h1d6d2e6_1 conda-forge parso 0.8.4 pyhd8ed1ab_0 conda-forge pexpect 4.9.0 pyhd8ed1ab_0 conda-forge pickleshare 0.7.5 py_1003 conda-forge pillow 10.3.0 py312hdcec9eb_0 conda-forge pip 24.0 pyhd8ed1ab_0 conda-forge platformdirs 4.2.2 pyhd8ed1ab_0 conda-forge pooch 1.8.2 pyhd8ed1ab_0 conda-forge prompt-toolkit 3.0.46 pyha770c72_0 conda-forge psutil 5.9.8 py312h98912ed_0 conda-forge pthread-stubs 0.4 h36c2ea0_1001 conda-forge ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge pure_eval 0.2.2 pyhd8ed1ab_0 conda-forge pygments 2.18.0 pyhd8ed1ab_0 conda-forge pyparsing 3.1.2 pyhd8ed1ab_0 conda-forge pysocks 1.7.1 pyha2e5f31_6 conda-forge python 3.12.3 hab00c5b_0_cpython conda-forge python-dateutil 2.9.0 pyhd8ed1ab_0 conda-forge python-tzdata 2024.1 pyhd8ed1ab_0 conda-forge python_abi 3.12 4_cp312 conda-forge pytz 2024.1 pyhd8ed1ab_0 conda-forge pyzmq 26.0.3 py312h8fd38d8_0 conda-forge readline 8.2 h8228510_1 conda-forge requests 2.32.3 pyhd8ed1ab_0 conda-forge scipy 1.13.1 py312hc2bc53b_0 conda-forge setuptools 70.0.0 pyhd8ed1ab_0 conda-forge sh 2.0.7 pypi_0 pypi six 1.16.0 pyh6c4a22f_0 conda-forge snappy 1.2.0 hdb0a2a9_1 conda-forge stack_data 0.6.2 pyhd8ed1ab_0 conda-forge sympy 1.12 pypyh9d50eac_103 conda-forge tk 8.6.13 noxft_h4845f30_101 conda-forge tomli-w 1.0.0 pyhd8ed1ab_0 conda-forge tornado 6.4.1 py312h9a8786e_0 conda-forge tqdm 4.66.4 pyhd8ed1ab_0 conda-forge traitlets 5.14.3 pyhd8ed1ab_0 conda-forge typing_extensions 4.12.2 pyha770c72_0 conda-forge tzdata 2024a h0c530f3_0 conda-forge unyt 2.9.5 pyhd8ed1ab_0 conda-forge urllib3 2.2.1 pyhd8ed1ab_0 conda-forge wcwidth 0.2.13 pyhd8ed1ab_0 conda-forge wheel 0.43.0 pyhd8ed1ab_1 conda-forge widgetsnbextension 4.0.11 pyhd8ed1ab_0 conda-forge xlrd 2.0.1 pyhd8ed1ab_3 conda-forge xorg-libxau 1.0.11 hd590300_0 conda-forge xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge xz 5.2.6 h166bdaf_0 conda-forge yt 4.4.dev0 pypi_0 pypi zeromq 4.3.5 h75354e8_4 conda-forge zipp 3.19.2 pyhd8ed1ab_0 conda-forge zlib 1.3.1 h4ab18f5_1 conda-forge zstd 1.5.6 ha6fb4c9_0 conda-forge ```

cphyc commented 3 months ago

After some digging, it seems like:

numpy, cython, etc. depend on libstdcxx-ng >= 12 (the C++ standard library),
conda resolves this to libstdcxx-ng == 13.2 at the time of writing (10/06/2024),
with a system with GCC 14, this seems to lead to incompatibilities between the expected C++ stdlib and the one conda provides.

I'm not sure how to fix it. In the meantime, compiling with GCC-13 works :)

cphyc commented 3 months ago

Gotcha! Here's more info:

$ python -c "import yt"
# fails with `ImportError: [...]/yt/utilities/lib/image_samplers.cpython-312-x86_64-linux-gnu.so
# undefined symbol: __cxa_call_terminate`

$ ldd yt/utilities/lib/image_samplers.cpython-312-x86_64-linux-gnu.so
[...]
    libstdc++.so.6 => ~/miniforge3/envs/py312/lib/libstdc++.so.6 (0x00007fd779101000)
[...]

# If we preload the system C++ std library this now works
$ LD_PRELOAD=/usr/lib64/libstdc++.so.6.0.33 python -c 'import yt'

matthewturk commented 3 months ago

Is it possible that you are compiling with the "wrong" CXX and this gets baked in? I don't see and gxx packages in your list.

cphyc commented 3 months ago

The gcc/g++ version that is being used by default is the system one. Or are you saying one should install gcc/g++ with conda?

I'm not sure there's any actionable solution for this issue, so I'm also fine to close it as "stalled".

matthewturk commented 3 months ago

Could you try installing the conda gcc/g++ and seeing if that fixes it?

zingale commented 2 months ago

probably not helpful, but I am running yt with GCC 14.1. I install from source and build with pip install ..

yt-project / yt

ImportError when compiling with gcc 14.1.1 and conda #4922

Bug report