isuruf
commented
2 years ago Can you send the output when LD_DEBUG=libs is set for both scenarios?
Closed 183amir closed 2 years ago
isuruf
commented
2 years ago Can you send the output when LD_DEBUG=libs is set for both scenarios?
183amir
commented
2 years ago Here are the steps to reproduce this issue:
$ conda create -n blitz \
-c https://www.idiap.ch/software/bob/conda/label/beta -c conda-forge \
python=3.8 bob.extension tensorflow=2.4 compilers pip click-plugins boost libblitz pkg-config --yes
# only bob.extension (a noarch package) will come from https://www.idiap.ch/software/bob/conda/label/beta
$ conda activate blitz
$ git clone https://github.com/bioidiap/bob.blitz.git
$ cd bob.blitz
$ python -m pip install -e .
$ python -c "import tensorflow, bob.blitz" # works
$ python -c "import bob.blitz, tensorflow" # breaks
free(): invalid pointerAnd here is the output of:
LD_DEBUG=libs python -c "import bob.blitz, tensorflow": libs_bob.blitz_tensorflow.txtLD_DEBUG=libs python -c "import tensorflow, bob.blitz": libs_tensorflow_bob.blitz.txt183amir
commented
2 years ago And here are the compile and link commands of bob.blitz:
Running setup.py develop for bob.blitz
Running command /home/amir/miniconda/envs/blitz/bin/python3.8 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/amir/idiap/git/bob/bob.blitz/setup.py'"'"'; __file__='"'"'/home/amir/idiap/git/bob/bob.blitz/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps
running develop
running egg_info
creating bob.blitz.egg-info
writing bob.blitz.egg-info/PKG-INFO
writing dependency_links to bob.blitz.egg-info/dependency_links.txt
writing requirements to bob.blitz.egg-info/requires.txt
writing top-level names to bob.blitz.egg-info/top_level.txt
writing manifest file 'bob.blitz.egg-info/SOURCES.txt'
reading manifest file 'bob.blitz.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
adding license file 'LICENSE'
writing manifest file 'bob.blitz.egg-info/SOURCES.txt'
running build_ext
building 'bob.blitz.version' extension
creating build
creating build/temp.linux-x86_64-3.8
creating build/temp.linux-x86_64-3.8/bob
creating build/temp.linux-x86_64-3.8/bob/blitz
/home/amir/miniconda/envs/blitz/bin/x86_64-conda-linux-gnu-cc -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/amir/miniconda/envs/blitz/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /home/amir/miniconda/envs/blitz/include -fPIC -Wno-strict-aliasing -DBOB_EXT_MODULE_PREFIX="bob.blitz" -DBOB_EXT_MODULE_NAME="version" -DBOB_EXT_ENTRY_NAME=PyInit_version -DBOB_EXT_MODULE_VERSION="2.0.24b0" -DHAVE_BOOST=1 -DHAVE_BLITZ=1 -DPY_ARRAY_UNIQUE_SYMBOL=BOB_BLITZ_NUMPY_C_API -DNO_IMPORT_ARRAY=1 -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -I/home/amir/idiap/git/bob/bob.blitz/bob/blitz/include -I/tmp/pip-build-env-by8_4g2l/overlay/lib/python3.8/site-packages/bob/extension/include -I/home/amir/miniconda/envs/blitz/include/python3.8 -c bob/blitz/version.cpp -o build/temp.linux-x86_64-3.8/bob/blitz/version.o -fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -pthread -isystem /tmp/pip-build-env-by8_4g2l/overlay/lib/python3.8/site-packages/numpy/core/include -isystem /home/amir/miniconda/envs/blitz/include
creating build/lib.linux-x86_64-3.8
creating build/lib.linux-x86_64-3.8/bob
creating build/lib.linux-x86_64-3.8/bob/blitz
/home/amir/miniconda/envs/blitz/bin/x86_64-conda-linux-gnu-c++ -pthread -shared -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,-rpath,/home/amir/miniconda/envs/blitz/lib -L/home/amir/miniconda/envs/blitz/lib -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,-rpath,/home/amir/miniconda/envs/blitz/lib -L/home/amir/miniconda/envs/blitz/lib -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,/home/amir/miniconda/envs/blitz/lib -Wl,-rpath-link,/home/amir/miniconda/envs/blitz/lib -L/home/amir/miniconda/envs/blitz/lib -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/amir/miniconda/envs/blitz/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /home/amir/miniconda/envs/blitz/include build/temp.linux-x86_64-3.8/bob/blitz/version.o -L/home/amir/miniconda/envs/blitz/lib -Wl,-R/home/amir/miniconda/envs/blitz/lib -lblitz -o build/lib.linux-x86_64-3.8/bob/blitz/version.cpython-38-x86_64-linux-gnu.so
building 'bob.blitz._library' extension
########## Below might be important ############
/home/amir/miniconda/envs/blitz/bin/x86_64-conda-linux-gnu-cc -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/amir/miniconda/envs/blitz/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /home/amir/miniconda/envs/blitz/include -fPIC -Wno-strict-aliasing -DBOB_EXT_MODULE_PREFIX="bob.blitz" -DBOB_EXT_MODULE_NAME="_library" -DBOB_EXT_ENTRY_NAME=PyInit__library -DBOB_EXT_MODULE_VERSION="2.0.24b0" -DHAVE_BOOST=1 -DHAVE_BLITZ=1 -DPY_ARRAY_UNIQUE_SYMBOL=BOB_BLITZ_NUMPY_C_API -DNO_IMPORT_ARRAY=1 -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -I/home/amir/idiap/git/bob/bob.blitz/bob/blitz/include -I/tmp/pip-build-env-by8_4g2l/overlay/lib/python3.8/site-packages/bob/extension/include -I/home/amir/miniconda/envs/blitz/include/python3.8 -c bob/blitz/api.cpp -o build/temp.linux-x86_64-3.8/bob/blitz/api.o -fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -pthread -isystem /tmp/pip-build-env-by8_4g2l/overlay/lib/python3.8/site-packages/numpy/core/include -isystem /home/amir/miniconda/envs/blitz/include
/home/amir/miniconda/envs/blitz/bin/x86_64-conda-linux-gnu-cc -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/amir/miniconda/envs/blitz/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /home/amir/miniconda/envs/blitz/include -fPIC -Wno-strict-aliasing -DBOB_EXT_MODULE_PREFIX="bob.blitz" -DBOB_EXT_MODULE_NAME="_library" -DBOB_EXT_ENTRY_NAME=PyInit__library -DBOB_EXT_MODULE_VERSION="2.0.24b0" -DHAVE_BOOST=1 -DHAVE_BLITZ=1 -DPY_ARRAY_UNIQUE_SYMBOL=BOB_BLITZ_NUMPY_C_API -DNO_IMPORT_ARRAY=1 -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -I/home/amir/idiap/git/bob/bob.blitz/bob/blitz/include -I/tmp/pip-build-env-by8_4g2l/overlay/lib/python3.8/site-packages/bob/extension/include -I/home/amir/miniconda/envs/blitz/include/python3.8 -c bob/blitz/array.cpp -o build/temp.linux-x86_64-3.8/bob/blitz/array.o -fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -pthread -isystem /tmp/pip-build-env-by8_4g2l/overlay/lib/python3.8/site-packages/numpy/core/include -isystem /home/amir/miniconda/envs/blitz/include
/home/amir/miniconda/envs/blitz/bin/x86_64-conda-linux-gnu-cc -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/amir/miniconda/envs/blitz/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /home/amir/miniconda/envs/blitz/include -fPIC -Wno-strict-aliasing -DBOB_EXT_MODULE_PREFIX="bob.blitz" -DBOB_EXT_MODULE_NAME="_library" -DBOB_EXT_ENTRY_NAME=PyInit__library -DBOB_EXT_MODULE_VERSION="2.0.24b0" -DHAVE_BOOST=1 -DHAVE_BLITZ=1 -DPY_ARRAY_UNIQUE_SYMBOL=BOB_BLITZ_NUMPY_C_API -DNO_IMPORT_ARRAY=1 -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -I/home/amir/idiap/git/bob/bob.blitz/bob/blitz/include -I/tmp/pip-build-env-by8_4g2l/overlay/lib/python3.8/site-packages/bob/extension/include -I/home/amir/miniconda/envs/blitz/include/python3.8 -c bob/blitz/main.cpp -o build/temp.linux-x86_64-3.8/bob/blitz/main.o -fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -pthread -isystem /tmp/pip-build-env-by8_4g2l/overlay/lib/python3.8/site-packages/numpy/core/include -isystem /home/amir/miniconda/envs/blitz/include
/home/amir/miniconda/envs/blitz/bin/x86_64-conda-linux-gnu-c++ -pthread -shared -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,-rpath,/home/amir/miniconda/envs/blitz/lib -L/home/amir/miniconda/envs/blitz/lib -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,-rpath,/home/amir/miniconda/envs/blitz/lib -L/home/amir/miniconda/envs/blitz/lib -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,/home/amir/miniconda/envs/blitz/lib -Wl,-rpath-link,/home/amir/miniconda/envs/blitz/lib -L/home/amir/miniconda/envs/blitz/lib -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/amir/miniconda/envs/blitz/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /home/amir/miniconda/envs/blitz/include build/temp.linux-x86_64-3.8/bob/blitz/api.o build/temp.linux-x86_64-3.8/bob/blitz/array.o build/temp.linux-x86_64-3.8/bob/blitz/main.o -L/home/amir/miniconda/envs/blitz/lib -Wl,-R/home/amir/miniconda/envs/blitz/lib -lblitz -o build/lib.linux-x86_64-3.8/bob/blitz/_library.cpython-38-x86_64-linux-gnu.so
########## Above might be important ############
copying build/lib.linux-x86_64-3.8/bob/blitz/version.cpython-38-x86_64-linux-gnu.so -> bob/blitz
copying build/lib.linux-x86_64-3.8/bob/blitz/_library.cpython-38-x86_64-linux-gnu.so -> bob/blitz
Creating /home/amir/miniconda/envs/blitz/lib/python3.8/site-packages/bob.blitz.egg-link (link to .)
Adding bob.blitz 2.0.24b0 to easy-install.pth file
Installed /home/amir/idiap/git/bob/bob.blitz
Successfully installed bob.blitz-2.0.24b0I have also narrowed down where in tensorflow the code breaks which is here: https://github.com/tensorflow/tensorflow/blob/v2.4.3/tensorflow/lite/experimental/microfrontend/python/ops/audio_microfrontend_op.py#L29 where the code breaks in a setting like:
def load_op_library(library_filename):
# loading with dlopen RTLD_LOCAL | RTLD_NOW
lib_handle = py_tf.TF_LoadLibrary(library_filename)
try:
tmp = py_tf.TF_GetOpList(lib_handle)
wrappers = _pywrap_python_op_gen.GetPythonWrappers(tmp) # breaks here
finally:
py_tf.TF_DeleteLibraryHandle(lib_handle)
_audio_microfrontend_op = load_op_library("_audio_microfrontend_op.so")It seems like libblitz is linked with -ffast-math. When a library is linked with -ffast-math (not compiled with -ffast-math) it links in a special runtime that would set some FPU registers at load time. When libstdc++.so is loaded after the FPU registers are changed, it must be changing some behaviour of libstdc++ that tensorflow is relying on. Still, libstdc++.so should work, but I guess there's an upstream bug in gcc. There's nothing we can do here though except removing -ffast-math.
At conda-forge, we strongly discourage linking with -ffast-math due to it changing the behaviour of all libraries loaded into the process. Compiling with -ffast-math is still okay if it is isolated and doesn't use header only libraries.
183amir
commented
2 years ago Thanks a lot for looking into this, you're right that this has to do with libstdc++
$ LD_PRELOAD=/home/amir/miniconda/envs/blitz/lib/libstdc++.so.6 python -c "import bob.blitz, tensorflow"
# or
$ LD_PRELOAD=/lib64/libstdc++.so.6 python -c "import bob.blitz, tensorflow"I tried recompiling libblitz using the changes in https://github.com/conda-forge/libblitz-feedstock/pull/17 and testing again, but, I still get the same error. Looking at the logs of libblitz building, I can see that -ffast-math is removed, so I am not sure if that is the problem or the only problem.
Also, I noticed that even doing:
$ LD_PRELOAD=/home/amir/miniconda/envs/blitz/lib/libblitz.so.0 python -c "import bob.blitz, tensorflow"
# or
$ LD_PRELOAD=/home/amir/idiap/git/bob/bob.blitz/bob/blitz/_library.cpython-38-x86_64-linux-gnu.so python -c "import bob.blitz, tensorflow"would fix the crash, where:
$ ldd -d /home/amir/miniconda/envs/blitz/lib/libblitz.so.0
linux-vdso.so.1 (0x00007fff6b1f3000)
libm.so.6 => /lib64/libm.so.6 (0x00007f91f36fd000)
libc.so.6 => /lib64/libc.so.6 (0x00007f91f352e000)
/lib64/ld-linux-x86-64.so.2 (0x00007f91f396e000)
libgcc_s.so.1 => /home/amir/miniconda/envs/blitz/lib/./libgcc_s.so.1 (0x00007f91f3519000)
Issue: I am trying to fix a segmentation fault (core dump) when importing my package before tensorflow. This is how the error happens:
but this error does not happen when I do:
or when I do:
and here is what I get from my env:
So my question is why does adding
LD_PRELOAD=/lib64/libstdc++.so.6fixe my issue? even though it looks like the packages are linked correctly. I will post instructions to reproduce this issue soon.Environment (
conda list):Details about
condaand system (conda info):