munmap_chunk(): invalid pointer - failures in build compiled by GCC 11

[pemensik]

Fedora Rawhide recently upgraded to GCC 11 and rebuilt kyua with it. Because stacktrace_test is always failing on Fedora, unit tests were disabled. That is unfortunate, because kyua became unusable on rawhide. It fails on platforms: armv7hl i686 ppc64le s390x, ppc64le only few test fail. Others fail completely.

Check last rawhide build build.log for alternative architectures. Every single unit test failed, but it was silently ignored. If you check build log

My backtraces in bug report hint, it might be related to compile-time warning:

g++ -DHAVE_CONFIG_H -I.         -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection  -D_FORTIFY_SOURCE=2 -Wall -Wcast-qual -Wextra -Wpointer-arith -Wredundant-decls -Wreturn-type -Wshadow -Wsign-compare -Wswitch -Wwrite-strings -DNDEBUG -Wctor-dtor-privacy -Wno-deprecated -Wno-non-template-friend -Wno-pmf-conversions -Wnon-virtual-dtor -Woverloaded-virtual -Wreorder -Wsign-promo -Wsynth -c -o utils/utils_auto_array_test-auto_array_test.o `test -f 'utils/auto_array_test.cpp' || echo './'`utils/auto_array_test.cpp
In file included from utils/auto_array_test.cpp:29:
In member function 'utils::auto_array<(anonymous namespace)::test_array>::~auto_array()',
    inlined from '(anonymous namespace)::atfu_tc_release::body() const' at utils/auto_array_test.cpp:203:39:
./utils/auto_array.ipp:95:9: warning: '(anonymous namespace)::test_array::operator delete[](void*)' called on pointer returned from a mismatched allocation function [-Wmismatched-new-delete]
   95 |         delete [] _ptr;
      |         ^~~~~~~~~~~~~~
./utils/auto_array.ipp: In member function '(anonymous namespace)::atfu_tc_release::body() const':
utils/auto_array_test.cpp:78:35: note: returned from 'operator new(unsigned long)'
   78 |         void* mem = ::operator new(size);
      |                     ~~~~~~~~~~~~~~^~~~~~
In file included from utils/auto_array_test.cpp:29:
In member function 'utils::auto_array<(anonymous namespace)::test_array>::~auto_array()',
    inlined from '(anonymous namespace)::atfu_tc_scope::body() const' at utils/auto_array_test.cpp:126:54:
./utils/auto_array.ipp:95:9: warning: '(anonymous namespace)::test_array::operator delete[](void*)' called on pointer returned from a mismatched allocation function [-Wmismatched-new-delete]
   95 |         delete [] _ptr;
      |         ^~~~~~~~~~~~~~
./utils/auto_array.ipp: In member function '(anonymous namespace)::atfu_tc_scope::body() const':
utils/auto_array_test.cpp:78:35: note: returned from 'operator new(unsigned long)'
   78 |         void* mem = ::operator new(size);
      |                     ~~~~~~~~~~~~~~^~~~~~
In file included from utils/auto_array_test.cpp:29:
In member function 'utils::auto_array<(anonymous namespace)::test_array>::~auto_array()',
    inlined from '(anonymous namespace)::atfu_tc_copy::body() const' at utils/auto_array_test.cpp:143:55:
./utils/auto_array.ipp:95:9: warning: '(anonymous namespace)::test_array::operator delete[](void*)' called on pointer returned from a mismatched allocation function [-Wmismatched-new-delete]
   95 |         delete [] _ptr;
      |         ^~~~~~~~~~~~~~
./utils/auto_array.ipp: In member function '(anonymous namespace)::atfu_tc_copy::body() const':
utils/auto_array_test.cpp:78:35: note: returned from 'operator new(unsigned long)'
   78 |         void* mem = ::operator new(size);
      |                     ~~~~~~~~~~~~~~^~~~~~
In file included from utils/auto_array_test.cpp:29:
In member function 'utils::auto_array<(anonymous namespace)::test_array>::~auto_array()',
    inlined from '(anonymous namespace)::atfu_tc_copy_ref::body() const' at utils/auto_array_test.cpp:166:55:
./utils/auto_array.ipp:95:9: warning: '(anonymous namespace)::test_array::operator delete[](void*)' called on pointer returned from a mismatched allocation function [-Wmismatched-new-delete]
...

Backtrace from the bug:

#0  0xf7fca559 in __kernel_vsyscall ()
#1  0xf7a9dbb8 in __libc_signal_restore_set (set=0xffffc2cc) at ../sysdeps/unix/sysv/linux/internal-signals.h:105
#2  __GI_raise (sig=<optimized out>) at ../sysdeps/unix/sysv/linux/raise.c:47
#3  0xf7a863c9 in __GI_abort () at abort.c:79
#4  0xf7ae449c in __libc_message (action=<optimized out>, fmt=<optimized out>) at ../sysdeps/posix/libc_fatal.c:155
#5  0xf7aecdaf in malloc_printerr (str=str@entry=0xf7bd373c "munmap_chunk(): invalid pointer") at malloc.c:5626
#6  0xf7aed1fb in munmap_chunk (p=p@entry=0xffffc784) at malloc.c:2995
#7  0xf7af1e00 in __GI___libc_free (mem=0xffffc78c) at malloc.c:3302
#8  0xf7cd3dac in operator delete (ptr=0xffffc78c) at ../../../../libstdc++-v3/libsupc++/del_op.cc:49
#9  0xf7cd3ddc in operator delete (ptr=0xffffc78c) at ../../../../libstdc++-v3/libsupc++/del_ops.cc:33
#10 0x565d53e4 in __gnu_cxx::new_allocator<char>::deallocate (__t=<optimized out>, __p=<optimized out>, this=0xffffc7d8)
    at /usr/include/c++/11/ext/new_allocator.h:139
#11 std::allocator_traits<std::allocator<char> >::deallocate (__n=<optimized out>, __p=<optimized out>, __a=...) at /usr/include/c++/11/bits/alloc_traits.h:492
#12 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_destroy (__size=<optimized out>, this=0xffffc7d8)
    at /usr/include/c++/11/bits/basic_string.h:237
#13 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_dispose (this=0xffffc7d8)
    at /usr/include/c++/11/bits/basic_string.h:232
#14 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string (this=<optimized out>, this=<optimized out>)
    at /usr/include/c++/11/bits/basic_string.h:658
#15 utils::fs::path::~path (this=<optimized out>, this=<optimized out>) at ./utils/fs/path.hpp:55
#16 utils::process::child::fork_files_aux (stdout_file=..., stderr_file=...) at utils/process/child.cpp:273
#17 0x565b2afb in utils::process::child::fork_files<utils::process::executor::detail::run_child<(anonymous namespace)::list_test_cases> > (stderr_file=..., 
    stdout_file=..., hook=...) at ./utils/process/child.ipp:56
#18 utils::process::executor::executor_handle::spawn<(anonymous namespace)::list_test_cases> (timeout=..., stderr_target=..., stdout_target=..., 
    unprivileged_user=..., hook=..., this=0x56625854) at ./utils/process/executor.ipp:132
#19 engine::scheduler::scheduler_handle::list_tests[abi:cxx11](model::test_program const*, utils::config::tree const&) (this=0xffffcd18, 
    test_program=<optimized out>, user_config=...) at engine/scheduler.cpp:1037
#20 0x565b34ba in engine::scheduler::lazy_test_program::test_cases[abi:cxx11]() const (this=0x56628e70) at engine/scheduler.cpp:575
#21 0x565a96cf in engine::scanner::impl::advance (this=<optimized out>) at engine/scanner.cpp:128
#22 0x565aa08f in engine::scanner::yield[abi:cxx11]() (this=0xffffcd30) at engine/scanner.cpp:190
#23 0x5659e2c8 in drivers::run_tests::drive (hooks=..., user_config=..., filters=std::set with 1 element = {...}, store_path=..., build_root=..., 
    kyuafile_path=...) at drivers/run_tests.cpp:289
#24 cli::cmd_test::run (this=0x566218e0, ui=0xffffd46c, cmdline=..., user_config=...) at cli/cmd_test.cpp:160
#25 0x56589fab in utils::cmdline::base_command<utils::config::tree>::main (data=..., args=std::vector of length 2, capacity 2 = {...}, ui=0xffffd46c, 
    this=0x566218e0) at utils/cmdline/base_command.cpp:114
#26 (anonymous namespace)::run_subcommand (user_config=..., args=std::vector of length 2, capacity 2 = {...}, command=0x566218e0, ui=0xffffd46c)
    at cli/main.cpp:139
#27 (anonymous namespace)::safe_main (ui=<optimized out>, argc=<optimized out>, argv=<optimized out>, mock_command=...) at cli/main.cpp:228
#28 0x5657fe55 in cli::main (mock_command=..., argv=0xffffd694, argc=3, ui=0xffffd46c) at cli/main.cpp:280
#29 cli::main (argc=3, argv=0xffffd694) at cli/main.cpp:353
#30 0x565807f5 in main (argc=3, argv=0xffffd694) at /usr/src/debug/kyua-0.13-7.fc34.i386/main.cpp:49

Was kyua tested on any different distribution with the recent GCC? Do you have any ideas, how to fix this issue? I use kyua to ensure ISC BIND package is good. Unfortunately only my bind build. failed hard and revealed the problem. But would have to turn off unit tests, unless kyua is fixed soon. Any ideas, how to fix this?

[pemensik]

Every single test run prints:

*** Please report this problem to kyua-discuss@googlegroups.com detailing what you were doing before the crash happened; if possible, include the log file mentioned above
munmap_chunk(): invalid pointer
*** Fatal signal 6 received

[kaniini]

Can you reproduce this in valgrind?

[pemensik]

Sure, 100% reliable. What options would you like?

[pemensik]

Okay, maybe I was wrong. I might be fixed on rawhide already, new kyua rebuild seems to pass tests on all architectures again. Not sure which part was fixed, but it passes on test machine with gcc-c++-11.0.0-0.19.fc35.ppc64le.

It still fails on stacktrace test, but passes fine otherwise on all platforms:

stacktrace_test:dump_stacktrace__cannot_find_core  ->  passed  [0.410s]
stacktrace_test:dump_stacktrace__cannot_find_gdb  ->  passed  [0.828s]
stacktrace_test:dump_stacktrace__gdb_fail  ->  passed  [0.879s]
stacktrace_test:dump_stacktrace__gdb_timeout  ->  passed  [2.140s]
stacktrace_test:dump_stacktrace__integration  ->  failed: Line 391: atf::utils::grep_file("#0", exit_handle.stderr_file().str()) not met  [0.649s]
stacktrace_test:dump_stacktrace__ok  ->  failed: Line 419: atf::utils::grep_file("^frame 1$", exit_handle.stderr_file().str()) not met  [0.689s]
stacktrace_test:dump_stacktrace_if_available__append  ->  failed: Line 559: atf::utils::grep_file("frame 1", exit_handle.stderr_file().str()) not met  [0.699s]
stacktrace_test:dump_stacktrace_if_available__no_coredump  ->  passed  [0.011s]
stacktrace_test:dump_stacktrace_if_available__no_status  ->  passed  [0.110s]
stacktrace_test:find_core__found__long  ->  failed: Core dumped, but no candidates found  [0.729s]
stacktrace_test:find_core__found__short  ->  failed: Core dumped, but no candidates found  [0.701s]
stacktrace_test:find_core__not_found  ->  passed  [0.010s]
stacktrace_test:find_gdb__bogus_value  ->  passed  [0.009s]
stacktrace_test:find_gdb__search_builtin__fail  ->  passed  [0.002s]
stacktrace_test:find_gdb__search_builtin__ok  ->  passed  [0.002s]
stacktrace_test:find_gdb__use_builtin  ->  passed  [0.009s]
stacktrace_test:unlimit_core_size  ->  passed  [0.730s]
stacktrace_test:unlimit_core_size__hard_is_zero  ->  failed: Line 281: !status.coredump() not met  [0.359s]

I think default integration with systemd is responsible:

# cat /proc/sys/kernel/core_pattern
|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h

Could failing tests be skipped if core_pattern starts with '|'?

[pemensik]

@jmmv, could you please rebuild the package on Fedora? I have filled bug #1926779, but no response from any kyua maintainers were received.