search

conda-forge / casadi-feedstock

A conda-smithy repository for casadi.
BSD 3-Clause "New" or "Revised" License
8 stars 12 forks source link

Segfault with ipopt 3.14.12_1, casadi 3.6.3_0 and libgomp >= 13 on Linux #91

Closed traversaro closed 11 months ago

traversaro commented 1 year ago

TL;DR: A workaround is to pin libgomp to 12, i.e. mamba install libgomp=12.

Solution to issue cannot be found in the documentation.

Issue

traversaro@IITICUBLAP257:~/test$  mamba create -n casadi casadi
traversaro@IITICUBLAP257:~/test$   mamba activate casadi
(casadi) traversaro@IITICUBLAP257:~/test$  wget https://raw.githubusercontent.com/conda-forge/casadi-feedstock/main/recipe/ipopt_nl.py
(casadi) traversaro@IITICUBLAP257:~/test$ python ipopt_nl.py
Segmentation fault
(casadi) traversaro@IITICUBLAP257:~/test$ mamba install ipopt=3.14.12=*_0
(casadi) traversaro@IITICUBLAP257:~/test$ python ipopt_nl.py

******************************************************************************
This program contains Ipopt, a library for large-scale nonlinear optimization.
 Ipopt is released as open source code under the Eclipse Public License (EPL).
         For more information visit https://github.com/coin-or/Ipopt
******************************************************************************

This is Ipopt version 3.14.12, running with linear solver MUMPS 5.2.1.

Installed packages

(casadi) traversaro@IITICUBLAP257:~/test$ mamba list
# packages in environment at /home/traversaro/miniforge3/envs/casadi:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
ampl-mp                   3.1.0             h2cc385e_1006    conda-forge
brotli-python             1.1.0           py311hb755f60_0    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
ca-certificates           2023.7.22            hbcca054_0    conda-forge
casadi                    3.6.3           py311hff1730f_0    conda-forge
certifi                   2023.7.22          pyhd8ed1ab_0    conda-forge
charset-normalizer        3.2.0              pyhd8ed1ab_0    conda-forge
eigen                     3.4.0                h00ab1b0_0    conda-forge
icu                       73.2                 h59595ed_0    conda-forge
idna                      3.4                pyhd8ed1ab_0    conda-forge
ipopt                     3.14.12              he6d3896_1    conda-forge
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
libblas                   3.9.0           18_linux64_openblas    conda-forge
libcblas                  3.9.0           18_linux64_openblas    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libexpat                  2.5.0                hcb278e6_1    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 13.2.0               h807b86a_0    conda-forge
libgfortran-ng            13.2.0               h69a702a_0    conda-forge
libgfortran5              13.2.0               ha4646dd_0    conda-forge
libgomp                   13.2.0               h807b86a_0    conda-forge
libhwloc                  2.9.1           nocuda_h7313eea_6    conda-forge
libiconv                  1.17                 h166bdaf_0    conda-forge
liblapack                 3.9.0           18_linux64_openblas    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libopenblas               0.3.24          pthreads_h413a1c8_0    conda-forge
libosqp                   0.6.3                h59595ed_0    conda-forge
libqdldl                  0.1.5                h27087fc_1    conda-forge
libspral                  2023.08.02           h2baf039_0    conda-forge
libsqlite                 3.43.0               h2797004_0    conda-forge
libstdcxx-ng              13.2.0               h7e041cc_0    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libxml2                   2.11.5               h232c23b_1    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
metis                     5.1.0             h59595ed_1007    conda-forge
mumps-include             5.2.1               ha770c72_11    conda-forge
mumps-seq                 5.2.1               h2104b81_11    conda-forge
ncurses                   6.4                  hcb278e6_0    conda-forge
numpy                     1.25.2          py311h64a7726_0    conda-forge
openssl                   3.1.2                hd590300_0    conda-forge
packaging                 23.1               pyhd8ed1ab_0    conda-forge
pip                       23.2.1             pyhd8ed1ab_0    conda-forge
platformdirs              3.10.0             pyhd8ed1ab_0    conda-forge
pooch                     1.7.0              pyha770c72_3    conda-forge
proxsuite                 0.3.7           py311h9547e67_0    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
python                    3.11.5          hab00c5b_0_cpython    conda-forge
python_abi                3.11                    3_cp311    conda-forge
readline                  8.2                  h8228510_1    conda-forge
requests                  2.31.0             pyhd8ed1ab_0    conda-forge
scipy                     1.11.2          py311h64a7726_0    conda-forge
scotch                    6.0.9                hb2e6521_2    conda-forge
setuptools                68.1.2             pyhd8ed1ab_0    conda-forge
simde                     0.7.6                h00ab1b0_0    conda-forge
tinyxml2                  9.0.0                h9c3ff4c_2    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
typing-extensions         4.7.1                hd8ed1ab_0    conda-forge
typing_extensions         4.7.1              pyha770c72_0    conda-forge
tzdata                    2023c                h71feb2d_0    conda-forge
unixodbc                  2.3.12               h661eb56_0    conda-forge
urllib3                   2.0.4              pyhd8ed1ab_0    conda-forge
wheel                     0.41.2             pyhd8ed1ab_0    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
zlib                      1.2.13               hd590300_5    conda-forge

Environment info

(casadi) traversaro@IITICUBLAP257:~/test$ mamba info

                  __    __    __    __
                 /  \  /  \  /  \  /  \
                /    \/    \/    \/    \
███████████████/  /██/  /██/  /██/  /████████████████████████
              /  / \   / \   / \   / \  \____
             /  /   \_/   \_/   \_/   \    o \__,
            / _/                       \_____/  `
            |/
        ███╗   ███╗ █████╗ ███╗   ███╗██████╗  █████╗
        ████╗ ████║██╔══██╗████╗ ████║██╔══██╗██╔══██╗
        ██╔████╔██║███████║██╔████╔██║██████╔╝███████║
        ██║╚██╔╝██║██╔══██║██║╚██╔╝██║██╔══██╗██╔══██║
        ██║ ╚═╝ ██║██║  ██║██║ ╚═╝ ██║██████╔╝██║  ██║
        ╚═╝     ╚═╝╚═╝  ╚═╝╚═╝     ╚═╝╚═════╝ ╚═╝  ╚═╝

        mamba (1.4.2) supported by @QuantStack

        GitHub:  https://github.com/mamba-org/mamba
        Twitter: https://twitter.com/QuantStack

█████████████████████████████████████████████████████████████

     active environment : casadi
    active env location : /home/traversaro/miniforge3/envs/casadi
            shell level : 1
       user config file : /home/traversaro/.condarc
 populated config files : /home/traversaro/miniforge3/.condarc
                          /home/traversaro/.condarc
          conda version : 23.3.1
    conda-build version : not installed
         python version : 3.10.12.final.0
       virtual packages : __archspec=1=x86_64
                          __cuda=12.2=0
                          __glibc=2.35=0
                          __linux=5.15.90.1=0
                          __unix=0=0
       base environment : /home/traversaro/miniforge3  (writable)
      conda av data dir : /home/traversaro/miniforge3/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /home/traversaro/miniforge3/pkgs
                          /home/traversaro/.conda/pkgs
       envs directories : /home/traversaro/miniforge3/envs
                          /home/traversaro/.conda/envs
               platform : linux-64
             user-agent : conda/23.3.1 requests/2.31.0 CPython/3.10.12 Linux/5.15.90.1-microsoft-standard-WSL2 ubuntu/22.04.2 glibc/2.35
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False
traversaro commented 1 year ago

Originally reported in https://github.com/ami-iit/hippopt/issues/8#issuecomment-1715891086 .

traversaro commented 1 year ago

I reproduced the failure in CI in https://github.com/conda-forge/casadi-feedstock/pull/90 and https://github.com/conda-forge/casadi-feedstock/pull/91 (with and without re-render)

2023-09-12T15:21:27.8690455Z + python ipopt_nl.py
2023-09-12T15:21:27.8691351Z /home/conda/feedstock_root/build_artifacts/casadi_1694531457704/test_tmp/run_test.sh: line 7:  2965 Segmentation fault      (core dumped) python ipopt_nl.py
2023-09-12T15:21:28.3353626Z Tests failed for casadi-3.6.3-py310h528c296_1.conda - moving package to /home/conda/feedstock_root/build_artifacts/broken
traversaro commented 1 year ago

On C++, there is no segfault, the segfault is only on Python. I was able to obtain a really simple reproducer:

python -c "import ctypes; ctypes._dlopen('libcasadi_nlpsol_ipopt.so', 9)"

or even simpler:

python -c "import ctypes; ctypes._dlopen('libcasadi_nlpsol_ipopt.so', 8)"

The flags are:

RTLD_NOW: 2
RTLD_LAZY: 1
RTLD_GLOBAL: 256
RTLD_LOCAL: 0
RTLD_DEEPBIND: 8

So this seems some issue of Python with RTLD_DEEPBIND and ipopt linked with spral.

traversaro commented 12 months ago

Exploring via ldd which library is segfaulting when loaded via RTLD_DEEPBIND, it turns out that it is libgomp, see:

 python -c "import ctypes; ctypes._dlopen('/home/traversaro/miniforge3/envs/hippoptdev/lib/./libgomp.so.1', 8)"

None of the libraries linked by libgomp have this problem, so the issue is probably due to libgomp itself. The issue was introduced in https://github.com/conda-forge/ipopt-feedstock/pull/94 as since that PR ipopt links spral that links libgomp.

traversaro commented 12 months ago

Indeed, the backtrace indicate a failure inside a libgomp function:

(gdb) bt
#0  initialize_env () at ../../../libgomp/env.c:2062
#1  0x00007ffff7fc947e in call_init (l=<optimized out>, argc=argc@entry=3, argv=argv@entry=0x7fffffffc1f8, env=env@entry=0x7fffffffc218)
    at ./elf/dl-init.c:70
#2  0x00007ffff7fc9568 in call_init (env=0x7fffffffc218, argv=0x7fffffffc1f8, argc=3, l=<optimized out>) at ./elf/dl-init.c:33
#3  _dl_init (main_map=0x555555b8e620, argc=3, argv=0x7fffffffc1f8, env=0x7fffffffc218) at ./elf/dl-init.c:117
#4  0x00007ffff7e09c85 in __GI__dl_catch_exception (exception=<optimized out>, operate=<optimized out>, args=<optimized out>)
    at ./elf/dl-error-skeleton.c:182
#5  0x00007ffff7fd0ff6 in dl_open_worker (a=0x7fffffffb910) at ./elf/dl-open.c:808
traversaro commented 12 months ago

From https://conda-forge.org/feedstock-outputs/ I get that libgomp is produced by https://github.com/conda-forge/ctng-compilers-feedstock , and as there is no specific patch for that file (see https://github.com/conda-forge/ctng-compilers-feedstock/tree/845b9c62143c4c2b2af8e35736e64aa29d8c3e05/recipe/patches ), so the failure is in https://github.com/gcc-mirror/gcc/blob/releases/gcc-13.2.0/libgomp/env.c#L2062 .

traversaro commented 12 months ago

The issue happens with libgomp 13.2.0 and 13.1.0, but not with libgomp 12.* .

traversaro commented 12 months ago

@SDafarra a simple workaround is to install libgomp (if you want to use spral enabled in the ipopt 3.14.12_1 build).

traversaro commented 11 months ago

The crash is fixed, but a problem remains related to the use of ipopt+spral with CasADi Python bindings: https://github.com/conda-forge/casadi-feedstock/issues/93 .