search

lczech / gappa

A toolkit for analyzing and visualizing phylogenetic (placement) data
GNU General Public License v3.0
56 stars 7 forks source link

gappa/0.8.3 :: core dump #22

Closed EricDeveaud closed 1 year ago

EricDeveaud commented 1 year ago

Hello

I tried to update gappa 0.8.2 to 0.8.3 using the release tagg archive: https://github.com/lczech/gappa/archive/refs/tags/v0.8.3.tar.gz

build went fine using gcc/9.2.0and cmake/3.19.4 on rhel-8.4 but when I tried to run the tests test/run.sh all tests failed and generated core files.

even gappa --version or gappa --help core dupms

NB 0.8.2 compiled the same way works fine.

regards

Eric

lczech commented 1 year ago

Hi @EricDeveaud,

thanks for reporting this! That's a weird issue - I've been testing on CentOS, which is a derivative of RHEL, and it works fine there... Let's try to figure this out!

Could you please build gappa in debug mode:

make clean
cmake -S . -B ./build -DCMAKE_BUILD_TYPE=DEBUG
cmake --build ./build -- -j 4

and post the full output of these commands here?

Then, run this to figure out what libraries gappa links to:

ldd bin/gappa

and post the output of that here.

Lastly, could you please run the debugging tool gdb (might need to install first) like this:

gdb -ex "set confirm off" -iex "set pagination off" -ex "run" -ex "thread apply all bt" -ex "bt" -ex "quit" --args bin/gappa --version

and post the output of that as well?! That should help to figure out what's going on there!

Cheers and so long Lucas

EricDeveaud commented 1 year ago

ok let's go

NB I added -DCMAKE_VERBOSE_MAKEFILE=ON in order to have a complete overview of the build process

rpm_maker:src/gappa > wget -O- https://github.com/lczech/gappa/archive/refs/tags/v0.8.3.tar.gz | tar xz
rpm_maker:src/gappa > cd gappa-0.8.3/
rpm_maker:gappa/gappa-0.8.3 > module load gcc/9.2.0 cmake/3.19.4
rpm_maker:gappa/gappa-0.8.3 > cmake -S . -B ./build -DCMAKE_BUILD_TYPE=DEBUG -DCMAKE_VERBOSE_MAKEFILE=ON
CMake Deprecation Warning at CMakeLists.txt:26 (cmake_minimum_required):
  Compatibility with CMake < 2.8.12 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.

-- The CXX compiler identification is GNU 9.2.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /opt/gensoft/exe/gcc/9.2.0/bin/g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Gappa build type: DEBUG
-- Static linking of system libraries: OFF
-- CLI11 not found
-- Downloading CLI11 from https://github.com/CLIUtils/CLI11/archive/13becaddb657eacd090537719a669d66d393b8b2.zip
CMake Deprecation Warning at CMakeLists.txt:45 (cmake_minimum_required):
  Compatibility with CMake < 2.8.12 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.

-- Configuring done
-- Generating done
-- Build files have been written to: /opt/gensoft/src/gappa/gappa-0.8.3/build/CLI11Download
Scanning dependencies of target CLI11
[ 11%] Creating directories for 'CLI11'
[ 22%] Performing download step (download, verify and extract) for 'CLI11'
-- Downloading...
   dst='/opt/gensoft/src/gappa/gappa-0.8.3/build/CLI11Download/CLI11-prefix/src/13becaddb657eacd090537719a669d66d393b8b2.zip'
   timeout='none'
   inactivity timeout='none'
-- Using src='https://github.com/CLIUtils/CLI11/archive/13becaddb657eacd090537719a669d66d393b8b2.zip'
-- Downloading... done
-- extracting...
     src='/opt/gensoft/src/gappa/gappa-0.8.3/build/CLI11Download/CLI11-prefix/src/13becaddb657eacd090537719a669d66d393b8b2.zip'
     dst='/opt/gensoft/src/gappa/gappa-0.8.3/libs/CLI11'
-- extracting... [tar xfz]
-- extracting... [analysis]
-- extracting... [rename]
-- extracting... [clean up]
-- extracting... done
[ 33%] No update step for 'CLI11'
[ 44%] No patch step for 'CLI11'
[ 55%] No configure step for 'CLI11'
[ 66%] No build step for 'CLI11'
[ 77%] No install step for 'CLI11'
[ 88%] No test step for 'CLI11'
[100%] Completed 'CLI11'
[100%] Built target CLI11
-- Finished downloading CLI11
-- genesis not found
-- Downloading genesis from https://github.com/lczech/genesis/archive/60a519711602f14d6c44765a477a104ac009e847.zip
CMake Deprecation Warning at CMakeLists.txt:45 (cmake_minimum_required):
  Compatibility with CMake < 2.8.12 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.

-- Configuring done
-- Generating done
-- Build files have been written to: /opt/gensoft/src/gappa/gappa-0.8.3/build/genesisDownload
Scanning dependencies of target genesis
[ 11%] Creating directories for 'genesis'
[ 22%] Performing download step (download, verify and extract) for 'genesis'
-- Downloading...
   dst='/opt/gensoft/src/gappa/gappa-0.8.3/build/genesisDownload/genesis-prefix/src/60a519711602f14d6c44765a477a104ac009e847.zip'
   timeout='none'
   inactivity timeout='none'
-- Using src='https://github.com/lczech/genesis/archive/60a519711602f14d6c44765a477a104ac009e847.zip'
-- Downloading... done
-- extracting...
     src='/opt/gensoft/src/gappa/gappa-0.8.3/build/genesisDownload/genesis-prefix/src/60a519711602f14d6c44765a477a104ac009e847.zip'
     dst='/opt/gensoft/src/gappa/gappa-0.8.3/libs/genesis'
-- extracting... [tar xfz]
-- extracting... [analysis]
-- extracting... [rename]
-- extracting... [clean up]
-- extracting... done
[ 33%] No update step for 'genesis'
[ 44%] No patch step for 'genesis'
[ 55%] No configure step for 'genesis'
[ 66%] No build step for 'genesis'
[ 77%] No install step for 'genesis'
[ 88%] No test step for 'genesis'
[100%] Completed 'genesis'
[100%] Built target genesis
-- Finished downloading genesis
-- sparsepp not found
-- Downloading sparsepp from https://github.com/greg7mdp/sparsepp/archive/6bfe3b4bdb364993e612d6bb729d680cf4c77649.zip
CMake Deprecation Warning at CMakeLists.txt:45 (cmake_minimum_required):
  Compatibility with CMake < 2.8.12 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.

-- Configuring done
-- Generating done
-- Build files have been written to: /opt/gensoft/src/gappa/gappa-0.8.3/build/sparseppDownload
Scanning dependencies of target sparsepp
[ 11%] Creating directories for 'sparsepp'
[ 22%] Performing download step (download, verify and extract) for 'sparsepp'
-- Downloading...
   dst='/opt/gensoft/src/gappa/gappa-0.8.3/build/sparseppDownload/sparsepp-prefix/src/6bfe3b4bdb364993e612d6bb729d680cf4c77649.zip'
   timeout='none'
   inactivity timeout='none'
-- Using src='https://github.com/greg7mdp/sparsepp/archive/6bfe3b4bdb364993e612d6bb729d680cf4c77649.zip'
-- Downloading... done
-- extracting...
     src='/opt/gensoft/src/gappa/gappa-0.8.3/build/sparseppDownload/sparsepp-prefix/src/6bfe3b4bdb364993e612d6bb729d680cf4c77649.zip'
     dst='/opt/gensoft/src/gappa/gappa-0.8.3/libs/sparsepp'
-- extracting... [tar xfz]
-- extracting... [analysis]
-- extracting... [rename]
-- extracting... [clean up]
-- extracting... done
[ 33%] No update step for 'sparsepp'
[ 44%] No patch step for 'sparsepp'
[ 55%] No configure step for 'sparsepp'
[ 66%] No build step for 'sparsepp'
[ 77%] No install step for 'sparsepp'
[ 88%] No test step for 'sparsepp'
[100%] Completed 'sparsepp'
[100%] Built target sparsepp
-- Finished downloading sparsepp
-- Configuring Genesis
-- CMake version 3.19.4
-- The C compiler identification is GNU 9.2.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /opt/gensoft/exe/gcc/9.2.0/bin/gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Building Genesis as a dependency
-- OS: Linux 4.18.0-305.49.1.el8_4.x86_64
-- Genesis version: v0.28.0 -- "Battery Life"
-- Genesis commit hash: 
-- Build type: DEBUG
-- Unity build: FULL
-- C++ compiler: GNU 9.2.0 at /opt/gensoft/exe/gcc/9.2.0/bin/g++
-- C compiler  : GNU 9.2.0 at /opt/gensoft/exe/gcc/9.2.0/bin/gcc
-- Building with LTO/IPO support
-- Check if the system is big endian
-- Searching 16 bit integer
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for stddef.h
-- Looking for stddef.h - found
-- Check size of unsigned short
-- Check size of unsigned short - done
-- Searching 16 bit integer - Using unsigned short
-- Check if the system is big endian - little endian
-- Looking for zlib
-- Found ZLIB: /usr/lib64/libz.so (found version "1.2.11") 
-- Found zlib: /usr/include 1.2.11
-- Using zlib
-- Looking for Threads
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - yes
-- Found Threads: TRUE  
-- Found Threads: -pthread
-- Using Threads
-- Looking for OpenMP
-- Found OpenMP_C: -fopenmp (found version "4.5") 
-- Found OpenMP_CXX: -fopenmp (found version "4.5") 
-- Found OpenMP: TRUE (found version "4.5")  
-- Found OpenMP: -fopenmp
-- Using OpenMP
-- Building static lib
-- Final build flags:
-- CMAKE_CXX_FLAGS         :  -std=c++11 -Wall -Wextra -Wno-unknown-pragmas -pedantic -pedantic-errors -Werror -std=c++11 -Wall -Wextra -Wno-unknown-pragmas -pedantic -pedantic-errors -Werror -fopenmp 
-- CMAKE_CXX_FLAGS_DEBUG   :  -O2 -DDEBUG -g -ggdb3 -D_GLIBCXX_DEBUG -D_GLIBCXX_DEBUG_PEDANTIC -O2 -DDEBUG -g -ggdb3 -D_GLIBCXX_DEBUG -D_GLIBCXX_DEBUG_PEDANTIC 
-- GENESIS_CXX_FLAGS       :  -fopenmp 
-- GENESIS_EXE_LINKER_FLAGS:  
-- GENESIS_LINK_LIBRARIES  :  /usr/lib64/libz.so;-pthread;genesis_lib_static 
-- Finished configuring Genesis
-- Building with LTO/IPO support
-- CMAKE_EXE_LINKER_FLAGS -L/opt/gensoft/exe/gcc/9.2.0/lib64 -L/opt/gensoft/exe/gcc/9.2.0/lib  
-- GENESIS_LINK_LIBRARIES /usr/lib64/libz.so;-pthread;genesis_lib_static
-- Configuring done
-- Generating done
-- Build files have been written to: /opt/gensoft/src/gappa/gappa-0.8.3/build

then make 

rpm_maker:gappa/gappa-0.8.3 > cmake --build ./build -- -j 2>&1 | tee log

log is attached. log.txt

now the fun part 

rpm_maker:gappa/gappa-0.8.3 > ldd bin/gappa 
    linux-vdso.so.1 (0x00007ffc29f79000)
    libz.so.1 => /lib64/libz.so.1 (0x00007f95ef393000)
    libgomp.so.1 => /opt/gensoft/exe/gcc/9.2.0/lib64/libgomp.so.1 (0x00007f95ef15d000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f95eef3d000)
    libstdc++.so.6 => /opt/gensoft/exe/gcc/9.2.0/lib64/libstdc++.so.6 (0x00007f95eeb5e000)
    libm.so.6 => /lib64/libm.so.6 (0x00007f95ee7dc000)
    libgcc_s.so.1 => /opt/gensoft/exe/gcc/9.2.0/lib64/libgcc_s.so.1 (0x00007f95ee5c4000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f95ee1ff000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007f95edffb000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f95ef5ab000)

seems OK

just to be sure:

rpm_maker:gappa/gappa-0.8.3 > bin/gappa --version 
Segmentation fault (core dumped)

here's the output of gdb 

rpm_maker:gappa/gappa-0.8.3 > gdb -ex "set confirm off" -iex "set pagination off" -ex "run" -ex "thread apply all bt" -ex "bt" -ex "quit" --args bin/gappa --version
GNU gdb (GDB) Red Hat Enterprise Linux 8.2-15.el8
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from bin/gappa...done.
Starting program: /opt/gensoft/src/gappa/gappa-0.8.3/bin/gappa --version
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
warning: File "/opt/gensoft/exe/gcc/9.2.0/lib64/libstdc++.so.6.0.27-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
To enable execution of this file add
    add-auto-load-safe-path /opt/gensoft/exe/gcc/9.2.0/lib64/libstdc++.so.6.0.27-gdb.py
line to your configuration file "/home/gensoft/.gdbinit".
To completely disable this security protection add
    set auto-load safe-path /
line to your configuration file "/home/gensoft/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
    info "(gdb)Auto-loading safe path"

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7ffff7fed0c0 (LWP 203890)):
#0  0x0000000000000000 in ?? ()
#1  0x00000000007c65d1 in std::ctype<char>::widen (__c=32 ' ', this=0x7ffff77606e0 <(anonymous namespace)::ctype_c>) at /opt/gensoft/exe/gcc/9.2.0/include/c++/9.2.0/bits/locale_facets.h:877
#2  std::ctype<char>::widen (__c=32 ' ', this=0x7ffff77606e0 <(anonymous namespace)::ctype_c>) at /opt/gensoft/exe/gcc/9.2.0/include/c++/9.2.0/bits/locale_facets.h:872
#3  std::basic_ios<char, std::char_traits<char> >::widen (__c=32 ' ', this=<optimized out>) at /opt/gensoft/exe/gcc/9.2.0/include/c++/9.2.0/bits/basic_ios.h:450
#4  std::basic_ios<char, std::char_traits<char> >::fill (this=<optimized out>) at /opt/gensoft/exe/gcc/9.2.0/include/c++/9.2.0/bits/basic_ios.h:374
#5  std::basic_ios<char, std::char_traits<char> >::fill (__ch=48 '0', this=<optimized out>) at /opt/gensoft/exe/gcc/9.2.0/include/c++/9.2.0/bits/basic_ios.h:392
#6  std::operator<< <char, std::char_traits<char> > (__f=..., __os=...) at /opt/gensoft/exe/gcc/9.2.0/include/c++/9.2.0/iomanip:180
#7  genesis::utils::<lambda(unsigned char)>::operator()(unsigned char) const (__closure=__closure@entry=0x7fffffffcdb0, v=<optimized out>) at /opt/gensoft/src/gappa/gappa-0.8.3/build/genesis_unity_sources/lib/all.cpp:59026
#8  0x00000000007cb4bc in genesis::utils::color_to_hex (c=..., prefix=..., uppercase=<optimized out>, with_alpha=<optimized out>) at /opt/gensoft/src/gappa/gappa-0.8.3/libs/genesis/lib/genesis/utils/tools/color.hpp:131
#9  0x00000000009112f5 in ColorMapOptions::ColorMapOptions (this=<optimized out>, this=<optimized out>) at /opt/gensoft/src/gappa/gappa-0.8.3/libs/genesis/lib/genesis/utils/tools/color/map.hpp:113
#10 0x0000000000a03092 in CorrelationOptions::CorrelationOptions (this=0xe59c40) at /opt/gensoft/exe/gcc/9.2.0/include/c++/9.2.0/ext/new_allocator.h:89
#11 __gnu_cxx::new_allocator<CorrelationOptions>::construct<CorrelationOptions> (this=<optimized out>, __p=0xe59c40) at /opt/gensoft/src/gappa/gappa-0.8.3/src/commands/analyze/correlation.hpp:43
#12 std::allocator_traits<std::allocator<CorrelationOptions> >::construct<CorrelationOptions> (__a=..., __p=0xe59c40) at /opt/gensoft/exe/gcc/9.2.0/include/c++/9.2.0/bits/alloc_traits.h:484
#13 std::_Sp_counted_ptr_inplace<CorrelationOptions, std::allocator<CorrelationOptions>, (__gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace<>(std::allocator<CorrelationOptions>) (__a=..., this=0xe59c30) at /opt/gensoft/exe/gcc/9.2.0/include/c++/9.2.0/bits/shared_ptr_base.h:548
#14 std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<CorrelationOptions, std::allocator<CorrelationOptions>>(CorrelationOptions*&, std::_Sp_alloc_shared_tag<std::allocator<CorrelationOptions> >) (__a=..., __p=<synthetic pointer>: <optimized out>, this=<synthetic pointer>) at /opt/gensoft/exe/gcc/9.2.0/include/c++/9.2.0/bits/shared_ptr_base.h:679
#15 std::__shared_ptr<CorrelationOptions, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator<CorrelationOptions>>(std::_Sp_alloc_shared_tag<std::allocator<CorrelationOptions> >) (__tag=..., this=<synthetic pointer>) at /opt/gensoft/exe/gcc/9.2.0/include/c++/9.2.0/bits/shared_ptr_base.h:1344
#16 std::shared_ptr<CorrelationOptions>::shared_ptr<std::allocator<CorrelationOptions>>(std::_Sp_alloc_shared_tag<std::allocator<CorrelationOptions> >) (__tag=..., this=<synthetic pointer>) at /opt/gensoft/exe/gcc/9.2.0/include/c++/9.2.0/bits/shared_ptr.h:359
#17 std::allocate_shared<CorrelationOptions, std::allocator<CorrelationOptions>>(std::allocator<CorrelationOptions> const&) (__a=...) at /opt/gensoft/exe/gcc/9.2.0/include/c++/9.2.0/bits/shared_ptr.h:702
#18 std::make_shared<CorrelationOptions> () at /opt/gensoft/exe/gcc/9.2.0/include/c++/9.2.0/bits/shared_ptr.h:718
#19 setup_correlation (app=...) at /opt/gensoft/src/gappa/gappa-0.8.3/src/commands/analyze/correlation.cpp:96
#20 0x00000000006cebf1 in setup_analyze (app=...) at /opt/gensoft/src/gappa/gappa-0.8.3/src/commands/analyze.hpp:60
#21 main (argc=2, argv=0x7fffffffdab8) at /opt/gensoft/src/gappa/gappa-0.8.3/src/main.cpp:109
#0  0x0000000000000000 in ?? ()
#1  0x00000000007c65d1 in std::ctype<char>::widen (__c=32 ' ', this=0x7ffff77606e0 <(anonymous namespace)::ctype_c>) at /opt/gensoft/exe/gcc/9.2.0/include/c++/9.2.0/bits/locale_facets.h:877
#2  std::ctype<char>::widen (__c=32 ' ', this=0x7ffff77606e0 <(anonymous namespace)::ctype_c>) at /opt/gensoft/exe/gcc/9.2.0/include/c++/9.2.0/bits/locale_facets.h:872
#3  std::basic_ios<char, std::char_traits<char> >::widen (__c=32 ' ', this=<optimized out>) at /opt/gensoft/exe/gcc/9.2.0/include/c++/9.2.0/bits/basic_ios.h:450
#4  std::basic_ios<char, std::char_traits<char> >::fill (this=<optimized out>) at /opt/gensoft/exe/gcc/9.2.0/include/c++/9.2.0/bits/basic_ios.h:374
#5  std::basic_ios<char, std::char_traits<char> >::fill (__ch=48 '0', this=<optimized out>) at /opt/gensoft/exe/gcc/9.2.0/include/c++/9.2.0/bits/basic_ios.h:392
#6  std::operator<< <char, std::char_traits<char> > (__f=..., __os=...) at /opt/gensoft/exe/gcc/9.2.0/include/c++/9.2.0/iomanip:180
#7  genesis::utils::<lambda(unsigned char)>::operator()(unsigned char) const (__closure=__closure@entry=0x7fffffffcdb0, v=<optimized out>) at /opt/gensoft/src/gappa/gappa-0.8.3/build/genesis_unity_sources/lib/all.cpp:59026
#8  0x00000000007cb4bc in genesis::utils::color_to_hex (c=..., prefix=..., uppercase=<optimized out>, with_alpha=<optimized out>) at /opt/gensoft/src/gappa/gappa-0.8.3/libs/genesis/lib/genesis/utils/tools/color.hpp:131
#9  0x00000000009112f5 in ColorMapOptions::ColorMapOptions (this=<optimized out>, this=<optimized out>) at /opt/gensoft/src/gappa/gappa-0.8.3/libs/genesis/lib/genesis/utils/tools/color/map.hpp:113
#10 0x0000000000a03092 in CorrelationOptions::CorrelationOptions (this=0xe59c40) at /opt/gensoft/exe/gcc/9.2.0/include/c++/9.2.0/ext/new_allocator.h:89
#11 __gnu_cxx::new_allocator<CorrelationOptions>::construct<CorrelationOptions> (this=<optimized out>, __p=0xe59c40) at /opt/gensoft/src/gappa/gappa-0.8.3/src/commands/analyze/correlation.hpp:43
#12 std::allocator_traits<std::allocator<CorrelationOptions> >::construct<CorrelationOptions> (__a=..., __p=0xe59c40) at /opt/gensoft/exe/gcc/9.2.0/include/c++/9.2.0/bits/alloc_traits.h:484
#13 std::_Sp_counted_ptr_inplace<CorrelationOptions, std::allocator<CorrelationOptions>, (__gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace<>(std::allocator<CorrelationOptions>) (__a=..., this=0xe59c30) at /opt/gensoft/exe/gcc/9.2.0/include/c++/9.2.0/bits/shared_ptr_base.h:548
#14 std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<CorrelationOptions, std::allocator<CorrelationOptions>>(CorrelationOptions*&, std::_Sp_alloc_shared_tag<std::allocator<CorrelationOptions> >) (__a=..., __p=<synthetic pointer>: <optimized out>, this=<synthetic pointer>) at /opt/gensoft/exe/gcc/9.2.0/include/c++/9.2.0/bits/shared_ptr_base.h:679
#15 std::__shared_ptr<CorrelationOptions, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator<CorrelationOptions>>(std::_Sp_alloc_shared_tag<std::allocator<CorrelationOptions> >) (__tag=..., this=<synthetic pointer>) at /opt/gensoft/exe/gcc/9.2.0/include/c++/9.2.0/bits/shared_ptr_base.h:1344
#16 std::shared_ptr<CorrelationOptions>::shared_ptr<std::allocator<CorrelationOptions>>(std::_Sp_alloc_shared_tag<std::allocator<CorrelationOptions> >) (__tag=..., this=<synthetic pointer>) at /opt/gensoft/exe/gcc/9.2.0/include/c++/9.2.0/bits/shared_ptr.h:359
#17 std::allocate_shared<CorrelationOptions, std::allocator<CorrelationOptions>>(std::allocator<CorrelationOptions> const&) (__a=...) at /opt/gensoft/exe/gcc/9.2.0/include/c++/9.2.0/bits/shared_ptr.h:702
#18 std::make_shared<CorrelationOptions> () at /opt/gensoft/exe/gcc/9.2.0/include/c++/9.2.0/bits/shared_ptr.h:718
#19 setup_correlation (app=...) at /opt/gensoft/src/gappa/gappa-0.8.3/src/commands/analyze/correlation.cpp:96
#20 0x00000000006cebf1 in setup_analyze (app=...) at /opt/gensoft/src/gappa/gappa-0.8.3/src/commands/analyze.hpp:60
#21 main (argc=2, argv=0x7fffffffdab8) at /opt/gensoft/src/gappa/gappa-0.8.3/src/main.cpp:109

as I'm not familliar nor used to C++ I can't help in this output decipher ;-)

regards

Eric

lczech commented 1 year ago

Okay wow, this is a tricky one, and given that I cannot reproduce locally, I'll leave it up to you how much time and energy you want to invest fixing this - I'm all in to get this figured out if you are, and very happy if you want to help me solve this riddle ;-)

So, first of all, thanks for the thorough test! This revealed a few possible things to try.

  1. Given that you are on Red Hat, and use a module system, I assume that you are on some cluster. Could you then maybe try to load a different compiler or different version of gcc, and try compiling with that? Currently, you are using module load gcc/9.2.0, but module avail should give you a list of other gcc versions, or maybe clang. There is a possibility that this is a bug in the version of the C++ standard library that comes with your specific gcc version.

  2. From the output of gdb, it seems that the seg fault appears in code related to locale settings (where the program determines how to print certain text according to your system language settings). Your GitHub handle says you are in Paris? So that might be the cause here - or more precisely, some code not handling this correctly. On the command line (of your Red Hat system), could you please run locale -a and locale and print the output here? That will tell us what languages you have available, and what your system language is set to.

  3. Next, it could be that something goes wrong with the dynamic linking to the C++ standard library. So let's try to link statically. Re-compile like this:

    make clean
cmake -S . -B ./build -DCMAKE_BUILD_TYPE=DEBUG -DGAPPA_BUILD_STATIC=ON -DGENESIS_USE_OPENMP=OFF
cmake --build ./build -- -j 12

    This will print a red warning about deactivating OpenMP, but that's okay, it's just to see if this is the culprit here. Please again post the output of that here, as well as the ldd and gdb commands as before.

  4. It could also be that some of the compilation settings mess things up. If none of the above yield any success, can you compile with this as well:

    make clean
cmake -S . -B ./build -DCMAKE_BUILD_TYPE=DEBUG -DGENESIS_USE_LTO=OFF
cmake --build ./build -- -j 12

    including the other steps as before?

Cheers and so long Lucas

EricDeveaud commented 1 year ago

Hello,

will be happy to perform any tests withvarious compilation settings thta may help.

I must say that gappa is built in a docker container. if that may have some imprtance. just a note, we already built near 1000 packages, libraries, interpreters, softs and so on using this environnement. we use a this setup to beabbleto have a strict isolation from the system libs adn to generate .rpms//.debs that we then install to our cluster.

docker environnement isset to use LC_ALL=C LANG=C let me know if you want me to build gappa with any other settings and I will

built gappa with gcc/10.1.0, gcc/11.3.0 using either LANG=C L_ALL=C or LANG=en_US.utf-8 LC_ALL=en_US.utf-8 and behaviour is the same gappa --version segfault in all the cases

now I'm starting with next steps sattic build, openmp off stay tunned

Eric

EricDeveaud commented 1 year ago

here we go.

EDIT * sorry confused gcc version it is 10.0.1 not 11.3.0that I used LANG=C LC_ALL=C gcc/11.3.0 GAPPA_BUILD_STATIC=ON GENESIS_USE_OPENMP=OFF

rpm_maker:src/gappa > export LANG=C LC_ALL=C
rpm_maker:src/gappa > rm -rf gappa-0.8.3
rpm_maker:src/gappa > tar xf v0.8.3.tar.gz 
rpm_maker:src/gappa > cd gappa-0.8.3
rpm_maker:gappa/gappa-0.8.3 > cmake -S . -B ./build -DCMAKE_BUILD_TYPE=DEBUG -DGAPPA_BUILD_STATIC=ON -DGENESIS_USE_OPENMP=OFF -DCMAKE_VERBOSE_MAKEFILE=ON 2>&1 | tee build.C.gcc_11.3.0_static_openmp_off.txt
rpm_maker:gappa/gappa-0.8.3 > cmake --build ./build -- -j 2>&1 | tee -a build.C.gcc_11.3.0_static_openmp_off.txt

build.C.gcc_11.3.0_static_openmp_off.txt

then there is some progress ;-) but. not as expected.

rpm_maker:gappa/gappa-0.8.3 > ./bin/gappa  --version 
v0.8.3
Aborted (core dumped)

gdb says: 

rpm_maker:gappa/gappa-0.8.3 > gdb -ex "set confirm off" -iex "set pagination off" -ex "run" -ex "thread apply all bt" -ex "bt" -ex "quit" --args bin/gappa --version  2>&1 | tee -a  gdb.C.gcc_11.3.0_static_openmp_off.txt

gdb.C.gcc_11.3.0_static_openmp_off.txt

try to build with other LANG setting

LANG=en_US.utf-8 LC_ALL=en_US.utf-8 gcc/11.3.0 GAPPA_BUILD_STATIC=ON GENESIS_USE_OPENMP=OFF

rpm_maker:src/gappa > LANG=en_US.utf-8 LC_ALL=en_US.utf-8
rpm_maker:src/gappa > rm -rf gappa-0.8.3
rpm_maker:src/gappa > tar xf v0.8.3.tar.gz 
rpm_maker:src/gappa > cd gappa-0.8.3
rpm_maker:gappa/gappa-0.8.3 > cmake -S . -B ./build -DCMAKE_BUILD_TYPE=DEBUG -DGAPPA_BUILD_STATIC=ON -DGENESIS_USE_OPENMP=OFF -DCMAKE_VERBOSE_MAKEFILE=ON 2>&1 | tee build.en_US.utf8.gcc_11.3.0_static_openmp_off.txt
rpm_maker:gappa/gappa-0.8.3 > cmake --build ./build -- -j 2>&1 | tee -a  build.en_US.utf8.gcc_11.3.0_static_openmp_off.txt

build.en_US.utf8.gcc_11.3.0_static_openmp_off.txt

gappa --version exact same behaviour. see log
gdb.en_US.utf8.gcc_11.3.0_static_openmp_off.txt

next step LTO_OFF

EricDeveaud commented 1 year ago

playing with LTO=OFF

rpm_maker:src/gappa > rm -rf gappa-0.8.3/
rpm_maker:src/gappa > export LC_ALL=C
rpm_maker:src/gappa > tar  xf v0.8.3.tar.gz 
rpm_maker:src/gappa > cd gappa-0.8.3/
rpm_maker:gappa/gappa-0.8.3 > cmake -S . -B ./build -DCMAKE_BUILD_TYPE=DEBUG -DGENESIS_USE_LTO=OFF -DCMAKE_VERBOSE_MAKEFILE=ON 2>&1 | tee build.C.gcc-10.l.0.lto_off.txt 
rpm_maker:gappa/gappa-0.8.3 > cmake --build ./build -- -j 2>&1 | tee -a  build.C.gcc-10.l.0.lto_off.txt 
rpm_maker:gappa/gappa-0.8.3 > ldd bin/gappa 
    linux-vdso.so.1 (0x00007ffe5a1ed000)
    libz.so.1 => /lib64/libz.so.1 (0x00007f7195292000)
    libgomp.so.1 => /opt/gensoft/exe/gcc/10.1.0/lib64/libgomp.so.1 (0x00007f7195053000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f7194e33000)
    libstdc++.so.6 => /opt/gensoft/exe/gcc/10.1.0/lib64/libstdc++.so.6 (0x00007f7194a60000)
    libm.so.6 => /lib64/libm.so.6 (0x00007f71946de000)
    libgcc_s.so.1 => /opt/gensoft/exe/gcc/10.1.0/lib64/libgcc_s.so.1 (0x00007f71944c6000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f7194101000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f71954aa000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007f7193efd000)

now it works 

rpm_maker:gappa/gappa-0.8.3 > ./bin/gappa  --version 
v0.8.3
rpm_maker:gappa/gappa-0.8.3 > echo $?
0

also test suite passes all tests

[==========] 61.585s
[   PASSED ] 20 tests

hope this. helps you to understand where the problem is hidding ;-)

regards

Eric

lczech commented 1 year ago

Awesome, thank you @EricDeveaud for the thorough testing! So, somehow LTO causes the issue. That sent me down a rabbit whole of googling... and it seems you are not the only one having issues with gcc's LTO... (see, e.g., here), but it still is a very specific problem.

Could you please try two more things, to narrow it down even more:

  1. Disable strict-aliasing:

    make clean
cmake -S . -B ./build -DCMAKE_CXX_FLAGS="-fno-strict-aliasing"
cmake --build ./build -- -j 12

  2. Disable the lifetime-dse optimization:

    make clean
cmake -S . -B ./build -DCMAKE_CXX_FLAGS="-flifetime-dse=1"
cmake --build ./build -- -j 12

and for both, try running gappa again, or if failing, run the gdb command again on both. My gut feeling is that the second one is more likely to be the problem, but hard to tell.

If either of them are the problem, I can add the respective flag to the gappa build instructions. If neither of them are the cause, we might have encountered a new bug in gcc, or in some other code. None of the segfaults that you have encountered above are in gappa code itself, so it might be really hard to track it down from here, and probably not worth it...

So, if either of the two trials above work, we are good. If not, you can simply deactivate LTO (-DGENESIS_USE_LTO=OFF, as above) for your build, and that should solve the problem. Would that be a solution that works for you?

So long, and thanks again Lucas

EricDeveaud commented 1 year ago

here we are.

first one no-strict-aliasing

rpm_maker:gappa/gappa-0.8.3 > cmake -S . -B ./build -DCMAKE_CXX_FLAGS="-fno-strict-aliasing" -DCMAKE_VERBOSE_MAKEFILE=ON 2>&1 | tee build.C.gcc-10.1.0.txt
rpm_maker:gappa/gappa-0.8.3 > cmake --build ./build -- -j  2>&1 | tee -a build.C.gcc-10.1.0.txt

build.C.gcc-10.1.0-no-strict-aliasing.txt

it still segfault here's the gdb log:

gdb.C.gcc-10.1.0-no-strict-aliasing.txt

let's try lifetime-dse

rpm_maker:gappa/gappa-0.8.3 > cmake -S . -B ./build -DCMAKE_CXX_FLAGS="-flifetime-dse=1" -DCMAKE_VERBOSE_MAKEFILE=ON 2>&1 | tee log build.C.gcc-10.1.0-lifetime-dse.txt
rpm_maker:gappa/gappa-0.8.3 > cmake --build ./build -- -j 2>&1 | tee log -a build.C.gcc-10.1.0-lifetime-dse.txt

build log build.C.gcc-10.1.0-lifetime-dse.txt : again segfault.

gdb log: gdb.C.gcc-10.1.0-lifetime-dse.txt

hope this new tests helps youto figure the problem, in the mean time disabling LTO is not a problem for us.

thanks for your valuable help.

regards

Eric

lczech commented 1 year ago

Hey Eric,

thank you again for the new tests! Unfortunately, as both of them still yield a segfault (and again, both in external code that is not written by me), it seems that the issue is somewhere deep, caused by some weirdness (or even a bug?) of the gcc LTO process... So, no idea how to further proceed from here. I could try with a Red Hat VM or something, but not sure that it's worth it for now. LTO was not enabled before this release, so at least, gappa won't be slower than before when you deactivate it again ;-)

So, if that's all right, please just disable LTO as tested above. Let me know if there are any further issues, or if we can consider this closed (for now).

Cheers Lucas

EricDeveaud commented 1 year ago

Lucas,

I think we can consider that this one is closed ;-) as said earlier, I can live without Link Time Optimisation. again thanks for your help.

Eric