search

redpanda-data / redpanda

Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
https://redpanda.com
9.61k stars 584 forks source link

CI Failure (heap-use-after-free) in `NodesDecommissioningTest.test_recommissioning_node` #8426

Closed rystsov closed 1 year ago

rystsov commented 1 year ago

https://buildkite.com/redpanda/redpanda/builds/21824#0185e922-b387-41ff-9071-8a603553ee88

Module: rptest.tests.nodes_decommissioning_test
Class:  NodesDecommissioningTest
Method: test_recommissioning_node
test_id:    rptest.tests.nodes_decommissioning_test.NodesDecommissioningTest.test_recommissioning_node
status:     FAIL
run time:   2 minutes 44.019 seconds

    <BadLogLines nodes=docker-rp-5(1) example="==1491==ERROR: AddressSanitizer: heap-use-after-free on address 0x61800003bbb8 at pc 0x5593f2ee9e5e bp 0x7ffc4bbb90b0 sp 0x7ffc4bbb90a8">
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/root/tests/rptest/services/cluster.py", line 67, in wrapped
    self.redpanda.raise_on_bad_logs(allow_list=log_allow_list)
  File "/root/tests/rptest/services/redpanda.py", line 1738, in raise_on_bad_logs
    raise BadLogLines(bad_lines)
rptest.services.utils.BadLogLines: <BadLogLines nodes=docker-rp-5(1) example="==1491==ERROR: AddressSanitizer: heap-use-after-free on address 0x61800003bbb8 at pc 0x5593f2ee9e5e bp 0x7ffc4bbb90b0 sp 0x7ffc4bbb90a8">
mmaslankaprv commented 1 year ago 
[Backtrace #0]
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x38bc0e5d: std::exception_ptr::operator bool() const at /vectorized/llvm/bin/../include/c++/v1/exception:155
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x38e0af17: seastar::basic_semaphore<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>::signal(unsigned long) at /vectorized/include/seastar/core/semaphore.hh:323
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x38e0ae4e: seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>::return_all() at /vectorized/include/seastar/core/semaphore.hh:469
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x38e0acf8: seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>::~semaphore_units() at /vectorized/include/seastar/core/semaphore.hh:449
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x39c3a100: void std::__1::__destroy_at<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>, 0>(seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>*) at /vectorized/llvm/bin/../include/c++/v1/__memory/construct_at.h:56
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x39c3a09c: void std::__1::destroy_at<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>, 0>(seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>*) at /vectorized/llvm/bin/../include/c++/v1/__memory/construct_at.h:81
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x39c3d5b2: seastar::future_state<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> >::clear() at /vectorized/include/seastar/core/future.hh:621
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x39c34208: seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> >::~future() at /vectorized/include/seastar/core/future.hh:628
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x3b54c7e0: void seastar::futurize<seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > >::satisfy_with_result_of<seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > seastar::future<void>::then_impl_nrvo<seastar::noncopyable_function<seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > ()>, seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > >(seastar::noncopyable_function<seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > ()>&&)::'lambda'(seastar::internal::promise_base_with_type<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> >&&, seastar::noncopyable_function<seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > ()>&, seastar::future_state<seastar::internal::monostate>&&)::operator()(seastar::internal::promise_base_with_type<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> >&&, seastar::noncopyable_function<seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > ()>&, seastar::future_state<seastar::internal::monostate>&&) const::'lambda'()>(seastar::internal::promise_base_with_type<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> >&&, seastar::noncopyable_function<seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > ()>&&) at /vectorized/include/seastar/core/future.hh:2134
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x3b54c4c6: seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > seastar::future<void>::then_impl_nrvo<seastar::noncopyable_function<seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > ()>, seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > >(seastar::noncopyable_function<seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > ()>&&)::'lambda'(seastar::internal::promise_base_with_type<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> >&&, seastar::noncopyable_function<seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > ()>&, seastar::future_state<seastar::internal::monostate>&&)::operator()(seastar::internal::promise_base_with_type<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> >&&, seastar::noncopyable_function<seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > ()>&, seastar::future_state<seastar::internal::monostate>&&) const at /vectorized/include/seastar/core/future.hh:1589
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x3b54be3a: seastar::continuation<seastar::internal::promise_base_with_type<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> >, seastar::noncopyable_function<seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > ()>, seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > seastar::future<void>::then_impl_nrvo<seastar::noncopyable_function<seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > ()>, seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > >(seastar::noncopyable_function<seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > ()>&&)::'lambda'(seastar::internal::promise_base_with_type<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> >&&, seastar::noncopyable_function<seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> > ()>&, seastar::future_state<seastar::internal::monostate>&&), void>::run_and_dispose() at /vectorized/include/seastar/core/future.hh:781
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x4b1ff5bd: seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2330
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x4b20541e: seastar::reactor::run_some_tasks() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2737
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x4b209f7f: seastar::reactor::do_run() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2906
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x4b207b08: seastar::reactor::run() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2789
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x4af1e16c: seastar::app_template::run_deprecated(int, char**, std::__1::function<void ()>&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/app-template.cc:265
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x4af1b825: seastar::app_template::run(int, char**, std::__1::function<seastar::future<int> ()>&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/app-template.cc:156
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x38828174: application::run(int, char**) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0ea2c1729d5f88308-1/redpanda/redpanda/src/v/redpanda/application.cc:322
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x387e57ee: main at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0ea2c1729d5f88308-1/redpanda/redpanda/src/v/redpanda/main.cc:22

freed by thread T0 here:[Backtrace #1]
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x387e389d: operator delete(void*) at /v/llvm/llvm/src/compiler-rt/lib/asan/asan_new_delete.cpp:152
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x478b10e1: seastar::shared_ptr_count_for<storage::disk_log_impl>::~shared_ptr_count_for() at /vectorized/include/seastar/core/shared_ptr.hh:465
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x38d63bd2: seastar::shared_ptr<storage::log::impl>::~shared_ptr() at /vectorized/include/seastar/core/shared_ptr.hh:538
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x38d639b8: storage::log::~log() at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0ea2c1729d5f88308-1/redpanda/redpanda/src/v/storage/log.h:36
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x38e039f9: raft::consensus::~consensus() at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0ea2c1729d5f88308-1/redpanda/redpanda/src/v/raft/consensus.h:55
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x38e0385c: seastar::lw_shared_ptr_no_esft<raft::consensus>::~lw_shared_ptr_no_esft() at /vectorized/include/seastar/core/shared_ptr.hh:158
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x38e037f3: seastar::internal::lw_shared_ptr_accessors_no_esft<raft::consensus>::dispose(seastar::lw_shared_ptr_counter_base*) at /vectorized/include/seastar/core/shared_ptr.hh:214
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x44dbcc0a: cluster::partition::~partition() at /vectorized/include/seastar/core/shared_ptr.hh:302
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x3942becc: seastar::lw_shared_ptr_no_esft<cluster::partition>::~lw_shared_ptr_no_esft() at /vectorized/include/seastar/core/shared_ptr.hh:158
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x3942be63: seastar::internal::lw_shared_ptr_accessors_no_esft<cluster::partition>::dispose(seastar::lw_shared_ptr_counter_base*) at /vectorized/include/seastar/core/shared_ptr.hh:214
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x44c544d6: cluster::controller_backend::delete_partition(model::ntp, detail::base_named_type<long, model::revision_id_model_type, std::__1::integral_constant<bool, true> >, cluster::partition_removal_mode) (.resume) at /vectorized/include/seastar/core/shared_ptr.hh:302
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x38ccd900: std::__1::coroutine_handle<seastar::internal::coroutine_traits_base<void>::promise_type>::resume() const at /vectorized/llvm/bin/../include/c++/v1/__coroutine/coroutine_handle.h:168
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x38ccd425: seastar::internal::coroutine_traits_base<void>::promise_type::run_and_dispose() at /vectorized/include/seastar/core/coroutine.hh:120
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x4b1ff5bd: seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2330
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x4b20541e: seastar::reactor::run_some_tasks() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2737
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x4b209f7f: seastar::reactor::do_run() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2906
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x4b207b08: seastar::reactor::run() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2789
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x4af1e16c: seastar::app_template::run_deprecated(int, char**, std::__1::function<void ()>&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/app-template.cc:265
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x4af1b825: seastar::app_template::run(int, char**, std::__1::function<seastar::future<int> ()>&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/app-template.cc:156
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x38828174: application::run(int, char**) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0ea2c1729d5f88308-1/redpanda/redpanda/src/v/redpanda/application.cc:322
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x387e57ee: main at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0ea2c1729d5f88308-1/redpanda/redpanda/src/v/redpanda/main.cc:22

previously allocated by thread T0 here:[Backtrace #2]
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x387e303d: operator new(unsigned long) at /v/llvm/llvm/src/compiler-rt/lib/asan/asan_new_delete.cpp:95
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x478b056b: seastar::shared_ptr<storage::disk_log_impl> seastar::shared_ptr_make_helper<storage::disk_log_impl, false>::make<storage::ntp_config, storage::log_manager&, storage::segment_set, storage::kvstore&>(storage::ntp_config&&, storage::log_manager&, storage::segment_set&&, storage::kvstore&) at /vectorized/include/seastar/core/shared_ptr.hh:632
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x477d0d5a: seastar::shared_ptr<storage::disk_log_impl> seastar::make_shared<storage::disk_log_impl, storage::ntp_config, storage::log_manager&, storage::segment_set, storage::kvstore&>(storage::ntp_config&&, storage::log_manager&, storage::segment_set&&, storage::kvstore&) at /vectorized/include/seastar/core/shared_ptr.hh:650
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x4772aab9: storage::make_disk_backed_log(storage::ntp_config, storage::log_manager&, storage::segment_set, storage::kvstore&) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0ea2c1729d5f88308-1/redpanda/redpanda/src/v/storage/disk_log_impl.cc:1656
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x4763850b: storage::log_manager::do_manage(storage::ntp_config) (.resume) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0ea2c1729d5f88308-1/redpanda/redpanda/src/v/storage/log_manager.cc:363
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x4765f370: std::__1::coroutine_handle<seastar::internal::coroutine_traits_base<storage::log>::promise_type>::resume() const at /vectorized/llvm/bin/../include/c++/v1/__coroutine/coroutine_handle.h:168
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x4765ee55: seastar::internal::coroutine_traits_base<storage::log>::promise_type::run_and_dispose() at /vectorized/include/seastar/core/coroutine.hh:78
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x4b1ff5bd: seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2330
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x4b20541e: seastar::reactor::run_some_tasks() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2737
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x4b209f7f: seastar::reactor::do_run() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2906
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x4b207b08: seastar::reactor::run() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2789
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x4af1e16c: seastar::app_template::run_deprecated(int, char**, std::__1::function<void ()>&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/app-template.cc:265
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x4af1b825: seastar::app_template::run(int, char**, std::__1::function<seastar::future<int> ()>&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/app-template.cc:156
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x38828174: application::run(int, char**) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0ea2c1729d5f88308-1/redpanda/redpanda/src/v/redpanda/application.cc:322
{/home/mmaslanka/dev/ducktape-remote/redpanda} 0x387e57ee: main at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0ea2c1729d5f88308-1/redpanda/redpanda/src/v/redpanda/main.cc:22
andijcr commented 1 year ago

closing it for now

dotnwat commented 1 year ago

closing it for now

@andijcr i've reopened this issue. please add information about why it is being closed.

andijcr commented 1 year ago

The underlying case is the same as #8384 , a segfault caused by a mismanaged semaphore in disk_log_impl. Fix is in https://github.com/redpanda-data/redpanda/pull/8431 . Couldn't reproduce locally, so I'm awaiting for a new ci failure

dotnwat commented 1 year ago

thanks @andijcr