search

scylladb / scylladb

NoSQL data store using the seastar framework, compatible with Apache Cassandra
http://scylladb.com
GNU Affero General Public License v3.0
13.58k stars 1.29k forks source link

coredump after nodetool cleanup failure #19033

Closed yarongilor closed 5 months ago

yarongilor commented 5 months ago

Packages

Scylla version: 6.1.0~dev-20240531.5b4e688668bf with build-id 39014653044c4f256aa6a74e012aa1cc1fd7d8b6

Kernel Version: 5.15.0-1060-gcp

Issue description

Describe your issue in detail and steps it took to produce it. node-4 nodetool cleanup command failed with:

2024-06-01 04:40:40.178
end
longevity-large-partitions-200k-pks-db-node-74d1069e-0-4
2024-06-01 04:40:40.178: (NodetoolEvent Severity.ERROR) period_type=end event_id=81c50f86-b97e-497e-8fba-4f3b7cd2907a during_nemesis=NodetoolCleanup duration=1m1s: nodetool_command=cleanup node=longevity-large-partitions-200k-pks-db-node-74d1069e-0-4 errors=["Encountered a bad command exit code!\n\nCommand: '/usr/bin/nodetool  cleanup '\n\nExit code: 4\n\nStdout:\n\n\n\nStderr:\n\nerror executing POST request to http://localhost:10000/storage_service/cleanup_all with parameters {}: remote replied with status code 500 Internal Server Error:\nservice::raft_operation_timeout_error (group [fd066d91-1fc5-11ef-804d-d6f28457f2d5] raft operation [read_barrier] timed out)\n\n", 'Traceback (most recent call last):\n  File "/home/ubuntu/scylla-cluster-tests/sdcm/cluster.py", line 2581, in run_nodetool\n    self.remoter.run(cmd, timeout=timeout, ignore_status=ignore_status, verbose=verbose, retry=retry)\n  File "/home/ubuntu/scylla-cluster-tests/sdcm/remote/remote_base.py", line 614, in run\n    result = _run()\n  File "/home/ubuntu/scylla-cluster-tests/sdcm/utils/decorators.py", line 65, in inner\n    return func(*args, **kwargs)\n  File "/home/ubuntu/scylla-cluster-tests/sdcm/remote/remote_base.py", line 605, in _run\n    return self._run_execute(cmd, timeout, ignore_status, verbose, new_session, watchers)\n  File "/home/ubuntu/scylla-cluster-tests/sdcm/remote/remote_base.py", line 538, in _run_execute\n    result = connection.run(**command_kwargs)\n  File "/home/ubuntu/scylla-cluster-tests/sdcm/remote/libssh2_client/__init__.py", line 620, in run\n    return self._complete_run(channel, exception, timeout_reached, timeout, result, warn, stdout, stderr)\n  File "/home/ubuntu/scylla-cluster-tests/sdcm/remote/libssh2_client/__init__.py", line 655, in _complete_run\n    raise UnexpectedExit(result)\nsdcm.remote.libssh2_client.exceptions.UnexpectedExit: Encountered a bad command exit code!\n\nCommand: \'/usr/bin/nodetool  cleanup \'\n\nExit code: 4\n\nStdout:\n\n\n\nStderr:\n\nerror executing POST request to http://localhost:10000/storage_service/cleanup_all with parameters {}: remote replied with status code 500 Internal Server Error:\nservice::raft_operation_timeout_error (group [fd066d91-1fc5-11ef-804d-d6f28457f2d5] raft operation [read_barrier] timed out)\n\n\n']
Traceback (most recent call last):
File "/home/ubuntu/scylla-cluster-tests/sdcm/cluster.py", line 2581, in run_nodetool
self.remoter.run(cmd, timeout=timeout, ignore_status=ignore_status, verbose=verbose, retry=retry)
File "/home/ubuntu/scylla-cluster-tests/sdcm/remote/remote_base.py", line 614, in run
result = _run()
File "/home/ubuntu/scylla-cluster-tests/sdcm/utils/decorators.py", line 65, in inner
return func(*args, **kwargs)
File "/home/ubuntu/scylla-cluster-tests/sdcm/remote/remote_base.py", line 605, in _run
return self._run_execute(cmd, timeout, ignore_status, verbose, new_session, watchers)
File "/home/ubuntu/scylla-cluster-tests/sdcm/remote/remote_base.py", line 538, in _run_execute
result = connection.run(**command_kwargs)
File "/home/ubuntu/scylla-cluster-tests/sdcm/remote/libssh2_client/__init__.py", line 620, in run
return self._complete_run(channel, exception, timeout_reached, timeout, result, warn, stdout, stderr)
File "/home/ubuntu/scylla-cluster-tests/sdcm/remote/libssh2_client/__init__.py", line 655, in _complete_run
raise UnexpectedExit(result)
sdcm.remote.libssh2_client.exceptions.UnexpectedExit: Encountered a bad command exit code!
Command: '/usr/bin/nodetool  cleanup '
Exit code: 4
Stdout:
Stderr:
error executing POST request to http://localhost:10000/storage_service/cleanup_all with parameters {}: remote replied with status code 500 Internal Server Error:
service::raft_operation_timeout_error (group [fd066d91-1fc5-11ef-804d-d6f28457f2d5] raft operation [read_barrier] timed out)

(see also in: https://github.com/scylladb/scylladb/issues/19031) node-4 got an error of: tablets - SSTable set wasn't found for tablet 103:

2024-06-01 05:05:05.469
Received: 2024-06-01 05:05:05.420
one-time
longevity-large-partitions-200k-pks-db-node-74d1069e-0-4
2024-06-01 05:05:05.469 <2024-06-01 05:05:05.420>: (DatabaseLogEvent Severity.ERROR) period_type=one-time event_id=878e487f-c159-4fc8-9c43-16788e0cb3f2: type=DATABASE_ERROR regex=(^ERROR|!\s*?ERR).*\[shard.*\] line_number=11491 node=longevity-large-partitions-200k-pks-db-node-74d1069e-0-4
2024-06-01T05:05:05.420+00:00 longevity-large-partitions-200k-pks-db-node-74d1069e-0-4      !ERR | scylla[7745]:  [shard 11:stmt] tablets - SSTable set wasn't found for tablet 103 of table scylla_bench.test, at: 0x63d89fe 0x63d9010 0x63d92f8 0x5e90af7 0x1d3bfbc 0x24b53f1 0x1cbe575 0x20ae2eb 0x20ad549 0x20c5f81 0x20c4ab0 0x3c0baac 0x3c0c8ae 0x3c9b4e3 0x1c64502 0x1c62409 0x1ac5677 0x1b5cd31 0x1b7d46b 0x1b7d038 0x5e67502 0x5ecfbdf 0x5ed11da 0x5ed2457 0x5ef6310 0x5e9182a /opt/scylladb/libreloc/libc.so.6+0x8c946 /opt/scylladb/libreloc/libc.so.6+0x11296f

node-4 got aborting on shard coredump:

2024-06-01 05:05:05.470 <2024-06-01 05:05:05.420>: (DatabaseLogEvent Severity.ERROR) period_type=one-time event_id=5f6f823a-4d82-428b-8ac2-6f9879a0606b: type=ABORTING_ON_SHARD regex=Aborting on shard line_number=11494 node=longevity-large-partitions-200k-pks-db-node-74d1069e-0-4
2024-06-01T05:05:05.420+00:00 longevity-large-partitions-200k-pks-db-node-74d1069e-0-4     !INFO | scylla[7745]: Aborting on shard 11, in scheduling group statement.
void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) at ./build/release/seastar/./seastar/include/seastar/util/backtrace.hh:68
(inlined by) seastar::backtrace_buffer::append_backtrace() at ./build/release/seastar/./seastar/src/core/reactor.cc:826
(inlined by) seastar::print_with_backtrace(seastar::backtrace_buffer&, bool) at ./build/release/seastar/./seastar/src/core/reactor.cc:859
seastar::print_with_backtrace(char const*, bool) at ./build/release/seastar/./seastar/src/core/reactor.cc:871
(inlined by) seastar::sigabrt_action() at ./build/release/seastar/./seastar/src/core/reactor.cc:4076
(inlined by) seastar::install_oneshot_signal_handler<6, (void (*)())(&seastar::sigabrt_action)>()::{lambda(int, siginfo_t*, void*)#1}::operator()(int, siginfo_t*, void*) const at ./build/release/seastar/./seastar/src/core/reactor.cc:4052
(inlined by) seastar::install_oneshot_signal_handler<6, (void (*)())(&seastar::sigabrt_action)>()::{lambda(int, siginfo_t*, void*)#1}::__invoke(int, siginfo_t*, void*) at ./build/release/seastar/./seastar/src/core/reactor.cc:4048
?? ??:0
?? ??:0
?? ??:0
?? ??:0
seastar::on_internal_error(seastar::logger&, std::basic_string_view<char, std::char_traits<char> >) at ./build/release/seastar/./seastar/src/core/on_internal_error.cc:57
replica::tablet_sstable_set::find_sstable_set(unsigned long) const at ./replica/tablets.cc:429
(inlined by) replica::tablet_sstable_set::create_single_key_sstable_reader(replica::table*, seastar::lw_shared_ptr<schema const>, reader_permit, utils::estimated_histogram&, interval<dht::ring_position> const&, query::partition_slice const&, tracing::trace_state_ptr, seastar::bool_class<streamed_mutation::forwarding_tag>, seastar::bool_class<mutation_reader::partition_range_forwarding_tag>, seastar::noncopyable_function<bool (sstables::sstable const&)> const&) const at ./replica/tablets.cc:647
sstables::sstable_set::create_single_key_sstable_reader(replica::table*, seastar::lw_shared_ptr<schema const>, reader_permit, utils::estimated_histogram&, interval<dht::ring_position> const&, query::partition_slice const&, tracing::trace_state_ptr, seastar::bool_class<streamed_mutation::forwarding_tag>, seastar::bool_class<mutation_reader::partition_range_forwarding_tag>, seastar::noncopyable_function<bool (sstables::sstable const&)> const&) const at ./sstables/sstable_set.cc:1268
replica::table::make_sstable_reader(seastar::lw_shared_ptr<schema const>, reader_permit, seastar::lw_shared_ptr<sstables::sstable_set const>, interval<dht::ring_position> const&, query::partition_slice const&, tracing::trace_state_ptr, seastar::bool_class<streamed_mutation::forwarding_tag>, seastar::bool_class<mutation_reader::partition_range_forwarding_tag>, seastar::noncopyable_function<bool (sstables::sstable const&)> const&) const at ./replica/table.cc:109
(inlined by) replica::table::sstables_as_snapshot_source()::$_0::operator()() const::{lambda(seastar::lw_shared_ptr<schema const>, reader_permit, interval<dht::ring_position> const&, query::partition_slice const&, tracing::trace_state_ptr, seastar::bool_class<streamed_mutation::forwarding_tag>, seastar::bool_class<mutation_reader::partition_range_forwarding_tag>)#1}::operator()(seastar::lw_shared_ptr<schema const>, reader_permit, interval<dht::ring_position> const&, query::partition_slice const&, tracing::trace_state_ptr, seastar::bool_class<streamed_mutation::forwarding_tag>, seastar::bool_class<mutation_reader::partition_range_forwarding_tag>) const at ./replica/table.cc:2252
(inlined by) flat_mutation_reader_v2 std::__invoke_impl<flat_mutation_reader_v2, replica::table::sstables_as_snapshot_source()::$_0::operator()() const::{lambda(seastar::lw_shared_ptr<schema const>, reader_permit, interval<dht::ring_position> const&, query::partition_slice const&, tracing::trace_state_ptr, seastar::bool_class<streamed_mutation::forwarding_tag>, seastar::bool_class<mutation_reader::partition_range_forwarding_tag>)#1}&, seastar::lw_shared_ptr<schema const>, reader_permit, interval<dht::ring_position> const&, query::partition_slice const&, tracing::trace_state_ptr, seastar::bool_class<streamed_mutation::forwarding_tag>, seastar::bool_class<mutation_reader::partition_range_forwarding_tag> >(std::__invoke_other, replica::table::sstables_as_snapshot_source()::$_0::operator()() const::{lambda(seastar::lw_shared_ptr<schema const>, reader_permit, interval<dht::ring_position> const&, query::partition_slice const&, tracing::trace_state_ptr, seastar::bool_class<streamed_mutation::forwarding_tag>, seastar::bool_class<mutation_reader::partition_range_forwarding_tag>)#1}&, seastar::lw_shared_ptr<schema const>&&, reader_permit&&, interval<dht::ring_position> const&, query::partition_slice const&, tracing::trace_state_ptr&&, seastar::bool_class<streamed_mutation::forwarding_tag>&&, seastar::bool_class<mutation_reader::partition_range_forwarding_tag>&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/bits/invoke.h:61
(inlined by) std::enable_if<is_invocable_r_v<flat_mutation_reader_v2, replica::table::sstables_as_snapshot_source()::$_0::operator()() const::{lambda(seastar::lw_shared_ptr<schema const>, reader_permit, interval<dht::ring_position> const&, query::partition_slice const&, tracing::trace_state_ptr, seastar::bool_class<streamed_mutation::forwarding_tag>, seastar::bool_class<mutation_reader::partition_range_forwarding_tag>)#1}&, seastar::lw_shared_ptr<schema const>, reader_permit, interval<dht::ring_position> const&, query::partition_slice const&, tracing::trace_state_ptr, seastar::bool_class<streamed_mutation::forwarding_tag>, seastar::bool_class<mutation_reader::partition_range_forwarding_tag> >, flat_mutation_reader_v2>::type std::__invoke_r<flat_mutation_reader_v2, replica::table::sstables_as_snapshot_source()::$_0::operator()() const::{lambda(seastar::lw_shared_ptr<schema const>, reader_permit, interval<dht::ring_position> const&, query::partition_slice const&, tracing::trace_state_ptr, seastar::bool_class<streamed_mutation::forwarding_tag>, seastar::bool_class<mutation_reader::partition_range_forwarding_tag>)#1}&, seastar::lw_shared_ptr<schema const>, reader_permit, interval<dht::ring_position> const&, query::partition_slice const&, tracing::trace_state_ptr, seastar::bool_class<streamed_mutation::forwarding_tag>, seastar::bool_class<mutation_reader::partition_range_forwarding_tag> >(replica::table::sstables_as_snapshot_source()::$_0::operator()() const::{lambda(seastar::lw_shared_ptr<schema const>, reader_permit, interval<dht::ring_position> const&, query::partition_slice const&, tracing::trace_state_ptr, seastar::bool_class<streamed_mutation::forwarding_tag>, seastar::bool_class<mutation_reader::partition_range_forwarding_tag>)#1}&, seastar::lw_shared_ptr<schema const>&&, reader_permit&&, interval<dht::ring_position> const&, query::partition_slice const&, tracing::trace_state_ptr&&, seastar::bool_class<streamed_mutation::forwarding_tag>&&, seastar::bool_class<mutation_reader::partition_range_forwarding_tag>&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/bits/invoke.h:114
(inlined by) std::_Function_handler<flat_mutation_reader_v2 (seastar::lw_shared_ptr<schema const>, reader_permit, interval<dht::ring_position> const&, query::partition_slice const&, tracing::trace_state_ptr, seastar::bool_class<streamed_mutation::forwarding_tag>, seastar::bool_class<mutation_reader::partition_range_forwarding_tag>), replica::table::sstables_as_snapshot_source()::$_0::operator()() const::{lambda(seastar::lw_shared_ptr<schema const>, reader_permit, interval<dht::ring_position> const&, query::partition_slice const&, tracing::trace_state_ptr, seastar::bool_class<streamed_mutation::forwarding_tag>, seastar::bool_class<mutation_reader::partition_range_forwarding_tag>)#1}>::_M_invoke(std::_Any_data const&, seastar::lw_shared_ptr<schema const>&&, reader_permit&&, interval<dht::ring_position> const&, query::partition_slice const&, tracing::trace_state_ptr&&, seastar::bool_class<streamed_mutation::forwarding_tag>&&, seastar::bool_class<mutation_reader::partition_range_forwarding_tag>&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/bits/std_function.h:290
std::function<flat_mutation_reader_v2 (seastar::lw_shared_ptr<schema const>, reader_permit, interval<dht::ring_position> const&, query::partition_slice const&, tracing::trace_state_ptr, seastar::bool_class<streamed_mutation::forwarding_tag>, seastar::bool_class<mutation_reader::partition_range_forwarding_tag>)>::operator()(seastar::lw_shared_ptr<schema const>, reader_permit, interval<dht::ring_position> const&, query::partition_slice const&, tracing::trace_state_ptr, seastar::bool_class<streamed_mutation::forwarding_tag>, seastar::bool_class<mutation_reader::partition_range_forwarding_tag>) const at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/bits/std_function.h:591
(inlined by) mutation_source::make_reader_v2(seastar::lw_shared_ptr<schema const>, reader_permit, interval<dht::ring_position> const&, query::partition_slice const&, tracing::trace_state_ptr, seastar::bool_class<streamed_mutation::forwarding_tag>, seastar::bool_class<mutation_reader::partition_range_forwarding_tag>) const at ././readers/mutation_source.hh:142
(inlined by) row_cache::create_underlying_reader(cache::read_context&, mutation_source&, interval<dht::ring_position> const&) at ./row_cache.cc:49
(inlined by) cache::autoupdating_underlying_reader::fast_forward_to(interval<dht::ring_position>&&, mutation_source&, unsigned long)::{lambda()#1}::operator()() at ././read_context.hh:98
(inlined by) seastar::future<void> seastar::futurize<void>::invoke<cache::autoupdating_underlying_reader::fast_forward_to(interval<dht::ring_position>&&, mutation_source&, unsigned long)::{lambda()#1}>(cache::autoupdating_underlying_reader::fast_forward_to(interval<dht::ring_position>&&, mutation_source&, unsigned long)::{lambda()#1}&&) at ././seastar/include/seastar/core/future.hh:2029
(inlined by) seastar::future<void> seastar::futurize<void>::invoke<cache::autoupdating_underlying_reader::fast_forward_to(interval<dht::ring_position>&&, mutation_source&, unsigned long)::{lambda()#1}>(cache::autoupdating_underlying_reader::fast_forward_to(interval<dht::ring_position>&&, mutation_source&, unsigned long)::{lambda()#1}&&, seastar::internal::monostate) at ././seastar/include/seastar/core/future.hh:1876
(inlined by) seastar::future<void> seastar::future<void>::then_impl<cache::autoupdating_underlying_reader::fast_forward_to(interval<dht::ring_position>&&, mutation_source&, unsigned long)::{lambda()#1}, seastar::future<void> >(cache::autoupdating_underlying_reader::fast_forward_to(interval<dht::ring_position>&&, mutation_source&, unsigned long)::{lambda()#1}&&) at ././seastar/include/seastar/core/future.hh:1501
(inlined by) seastar::future<void> seastar::future<void>::then<cache::autoupdating_underlying_reader::fast_forward_to(interval<dht::ring_position>&&, mutation_source&, unsigned long)::{lambda()#1}, seastar::future<void> >(cache::autoupdating_underlying_reader::fast_forward_to(interval<dht::ring_position>&&, mutation_source&, unsigned long)::{lambda()#1}&&) at ././seastar/include/seastar/core/future.hh:1427
(inlined by) cache::autoupdating_underlying_reader::fast_forward_to(interval<dht::ring_position>&&, mutation_source&, unsigned long) at ././read_context.hh:97
cache::read_context::create_underlying() at ./row_cache.cc:399
single_partition_populating_reader::create_reader() at ./row_cache.cc:421
single_partition_populating_reader::fill_buffer() at ./row_cache.cc:454
flat_mutation_reader_v2::impl::operator()() at ././readers/flat_mutation_reader_v2.hh:194
(inlined by) flat_mutation_reader_v2::operator()() at ././readers/flat_mutation_reader_v2.hh:410
(inlined by) db::view::view_update_builder::advance_all() at ./db/view/view.cc:1240
db::view::view_update_builder::build_some() at ./db/view/view.cc:1270
db::view::view_update_generator::generate_and_propagate_view_updates(replica::table const&, seastar::lw_shared_ptr<schema const> const&, reader_permit, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, mutation&&, seastar::optimized_optional<flat_mutation_reader_v2>, tracing::trace_state_ptr, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >) at ./db/view/view_update_generator.cc:412
replica::table::do_push_view_replica_updates(seastar::shared_ptr<db::view::view_update_generator>, seastar::lw_shared_ptr<schema const>, mutation, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, mutation_source, tracing::trace_state_ptr, reader_concurrency_semaphore&, enum_set<super_enum<query::partition_slice::option, (query::partition_slice::option)0, (query::partition_slice::option)1, (query::partition_slice::option)2, (query::partition_slice::option)3, (query::partition_slice::option)4, (query::partition_slice::option)5, (query::partition_slice::option)6, (query::partition_slice::option)7, (query::partition_slice::option)8, (query::partition_slice::option)9, (query::partition_slice::option)10, (query::partition_slice::option)11, (query::partition_slice::option)12, (query::partition_slice::option)13> >) const at ./replica/table.cc:3253
replica::table::push_view_replica_updates(seastar::shared_ptr<db::view::view_update_generator>, seastar::lw_shared_ptr<schema const> const&, mutation&&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, reader_concurrency_semaphore&) const at ./replica/table.cc:3264
(inlined by) replica::table::push_view_replica_updates(seastar::shared_ptr<db::view::view_update_generator>, seastar::lw_shared_ptr<schema const> const&, frozen_mutation const&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, reader_concurrency_semaphore&) const at ./replica/table.cc:3198
replica::database::do_apply(seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>) at ./replica/database.cc:2010
seastar::future<void> std::__invoke_impl<seastar::future<void>, seastar::future<void> (replica::database::* const&)(seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>), replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce> >(std::__invoke_memfun_deref, seastar::future<void> (replica::database::* const&)(seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>), replica::database*&&, seastar::lw_shared_ptr<schema const>&&, frozen_mutation const&, tracing::trace_state_ptr&&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >&&, seastar::bool_class<db::force_sync_tag>&&, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/bits/invoke.h:74
(inlined by) std::__invoke_result<seastar::future<void> (replica::database::* const&)(seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>), replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce> >::type std::__invoke<seastar::future<void> (replica::database::* const&)(seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>), replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce> >(seastar::future<void> (replica::database::* const&)(seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>), replica::database*&&, seastar::lw_shared_ptr<schema const>&&, frozen_mutation const&, tracing::trace_state_ptr&&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >&&, seastar::bool_class<db::force_sync_tag>&&, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/bits/invoke.h:96
(inlined by) decltype (std::__invoke((*this)._M_pmf, std::forward<replica::database*>({parm#1}), std::forward<seastar::lw_shared_ptr<schema const> >({parm#1}), std::forward<frozen_mutation const&>({parm#1}), std::forward<tracing::trace_state_ptr>({parm#1}), std::forward<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >({parm#1}), std::forward<seastar::bool_class<db::force_sync_tag> >({parm#1}), std::forward<std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce> >({parm#1}))) std::_Mem_fn_base<seastar::future<void> (replica::database::*)(seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>), true>::operator()<replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce> >(replica::database*&&, seastar::lw_shared_ptr<schema const>&&, frozen_mutation const&, tracing::trace_state_ptr&&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >&&, seastar::bool_class<db::force_sync_tag>&&, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>&&) const at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/functional:170
(inlined by) seastar::noncopyable_function<seastar::future<void> (replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)>::direct_vtable_for<std::_Mem_fn<seastar::future<void> (replica::database::*)(seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)> >::call(seastar::noncopyable_function<seastar::future<void> (replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)> const*, replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>) at ././seastar/include/seastar/util/noncopyable_function.hh:129
seastar::noncopyable_function<seastar::future<void> (replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)>::operator()(replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>) const at ././seastar/include/seastar/util/noncopyable_function.hh:215
(inlined by) seastar::inheriting_concrete_execution_stage<seastar::future<void>, replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce> >::make_stage_for_group(seastar::scheduling_group)::{lambda(replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)#1}::operator()(replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>) const at ././seastar/include/seastar/core/execution_stage.hh:340
(inlined by) seastar::noncopyable_function<seastar::future<void> (replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)>::direct_vtable_for<seastar::inheriting_concrete_execution_stage<seastar::future<void>, replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce> >::make_stage_for_group(seastar::scheduling_group)::{lambda(replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)#1}>::call(seastar::noncopyable_function<seastar::future<void> (replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)> const*, replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>) at ././seastar/include/seastar/util/noncopyable_function.hh:129
seastar::noncopyable_function<seastar::future<void> (replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)>::operator()(replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>) const at ././seastar/include/seastar/util/noncopyable_function.hh:215
(inlined by) seastar::future<void> std::__invoke_impl<seastar::future<void>, seastar::noncopyable_function<seastar::future<void> (replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)>&, replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce> >(std::__invoke_other, seastar::noncopyable_function<seastar::future<void> (replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)>&, replica::database*&&, seastar::lw_shared_ptr<schema const>&&, frozen_mutation const&, tracing::trace_state_ptr&&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >&&, seastar::bool_class<db::force_sync_tag>&&, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/bits/invoke.h:61
(inlined by) std::__invoke_result<seastar::noncopyable_function<seastar::future<void> (replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)>&, replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce> >::type std::__invoke<seastar::noncopyable_function<seastar::future<void> (replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)>&, replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce> >(seastar::noncopyable_function<seastar::future<void> (replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)>&, replica::database*&&, seastar::lw_shared_ptr<schema const>&&, frozen_mutation const&, tracing::trace_state_ptr&&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >&&, seastar::bool_class<db::force_sync_tag>&&, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/bits/invoke.h:96
(inlined by) decltype(auto) std::__apply_impl<seastar::noncopyable_function<seastar::future<void> (replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)>&, std::tuple<replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce> >, 0ul, 1ul, 2ul, 3ul, 4ul, 5ul, 6ul>(seastar::noncopyable_function<seastar::future<void> (replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)>&, std::tuple<replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce> >&&, std::integer_sequence<unsigned long, 0ul, 1ul, 2ul, 3ul, 4ul, 5ul, 6ul>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/tuple:2288
(inlined by) decltype(auto) std::apply<seastar::noncopyable_function<seastar::future<void> (replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)>&, std::tuple<replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce> > >(seastar::noncopyable_function<seastar::future<void> (replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)>&, std::tuple<replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce> >&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/tuple:2299
(inlined by) seastar::future<void> seastar::futurize<seastar::future<void> >::apply<seastar::noncopyable_function<seastar::future<void> (replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)>&, replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce> >(seastar::noncopyable_function<seastar::future<void> (replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)>&, std::tuple<replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce> >&&) at ././seastar/include/seastar/core/future.hh:2000
(inlined by) seastar::concrete_execution_stage<seastar::future<void>, replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce> >::do_flush() at ././seastar/include/seastar/core/execution_stage.hh:249
seastar::execution_stage::flush()::$_0::operator()() const at ./build/release/seastar/./seastar/src/core/execution_stage.cc:149
(inlined by) seastar::future<void> seastar::futurize<void>::invoke<seastar::execution_stage::flush()::$_0&>(seastar::execution_stage::flush()::$_0&) at ./build/release/seastar/./seastar/include/seastar/core/future.hh:2029
(inlined by) seastar::lambda_task<seastar::execution_stage::flush()::$_0>::run_and_dispose() at ./build/release/seastar/./seastar/include/seastar/core/make_task.hh:44
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at ./build/release/seastar/./seastar/src/core/reactor.cc:2694
seastar::reactor::run_some_tasks() at ./build/release/seastar/./seastar/src/core/reactor.cc:3157
seastar::reactor::do_run() at ./build/release/seastar/./seastar/src/core/reactor.cc:3325
seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_0::operator()() const at ./build/release/seastar/./seastar/src/core/reactor.cc:4568
(inlined by) void std::__invoke_impl<void, seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_0&>(std::__invoke_other, seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_0&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/bits/invoke.h:61
(inlined by) std::enable_if<is_invocable_r_v<void, seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_0&>, void>::type std::__invoke_r<void, seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_0&>(seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_0&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/bits/invoke.h:111
(inlined by) std::_Function_handler<void (), seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_0>::_M_invoke(std::_Any_data const&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/bits/std_function.h:290
std::function<void ()>::operator()() const at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/bits/std_function.h:591
(inlined by) seastar::posix_thread::start_routine(void*) at ./build/release/seastar/./seastar/src/core/posix.cc:90
?? ??:0
?? ??:0
2024-06-01 05:15:26.377 <2024-06-01 05:05:05.000>: (CoreDumpEvent Severity.ERROR) period_type=one-time event_id=9cc1d173-616a-4056-bdcf-12a8c955233f node=Node longevity-large-partitions-200k-pks-db-node-74d1069e-0-4 [34.73.45.90 | 10.142.15.246]
corefile_url=https://storage.cloud.google.com/upload.scylladb.com/core.scylla.113.23b5b930df264c819a308f973f22996a.7745.1717218305000000/core.scylla.113.23b5b930df264c819a308f973f22996a.7745.1717218305000000.gz
backtrace=           PID: 7745 (scylla)
UID: 113 (scylla)
GID: 120 (scylla)
Signal: 6 (ABRT)
Timestamp: Sat 2024-06-01 05:05:05 UTC (2min 6s ago)
Command Line: /usr/bin/scylla --blocked-reactor-notify-ms 25 --abort-on-lsa-bad-alloc 1 --abort-on-seastar-bad-alloc --abort-on-internal-error 1 --abort-on-ebadf 1 --enable-sstable-key-validation 1 --log-to-syslog 1 --log-to-stdout 0 --default-log-level info --network-stack posix --io-properties-file=/etc/scylla.d/io_properties.yaml --cpuset 1-7,9-15 --lock-memory=1
Executable: /opt/scylladb/libexec/scylla
Control Group: /scylla.slice/scylla-server.slice/scylla-server.service
Unit: scylla-server.service
Slice: scylla-server.slice
Boot ID: 23b5b930df264c819a308f973f22996a
Machine ID: 11ea5f0ed42c01c66e513cfd46f4e7ba
Hostname: longevity-large-partitions-200k-pks-db-node-74d1069e-0-4
Storage: /var/lib/systemd/coredump/core.scylla.113.23b5b930df264c819a308f973f22996a.7745.1717218305000000 (present)
Disk Size: 116.3G
Message: Process 7745 (scylla) of user 113 dumped core.
Stack trace of thread 7757:
#0  0x00007f9781583884 __pthread_kill_implementation (libc.so.6 + 0x8e884)
#1  0x00007f9781532afe raise (libc.so.6 + 0x3dafe)
#2  0x00007f978151b87f abort (libc.so.6 + 0x2687f)
#3  0x0000000005e90b78 _ZN7seastar17on_internal_errorERNS_6loggerESt17basic_string_viewIcSt11char_traitsIcEE (scylla + 0x5c90b78)
#4  0x0000000001d3bfbd _ZNK7replica18tablet_sstable_set32create_single_key_sstable_readerEPNS_5tableEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRN5utils19estimated_histogramERK8intervalIN3dht13ring_positionEERKN5query15partition_sliceEN7tracing15trace_state_ptrENS3_10bool_classIN17streamed_mutation14forwarding_tagEEENSO_IN15mutation_reader30partition_range_forwarding_tagEEERKNS3_20noncopyable_functionIFbRKN8sstables7sstableEEEE (scylla + 0x1b3bfbd)
#5  0x00000000024b53f2 _ZNK8sstables11sstable_set32create_single_key_sstable_readerEPN7replica5tableEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRN5utils19estimated_histogramERK8intervalIN3dht13ring_positionEERKN5query15partition_sliceEN7tracing15trace_state_ptrENS4_10bool_classIN17streamed_mutation14forwarding_tagEEENSP_IN15mutation_reader30partition_range_forwarding_tagEEERKNS4_20noncopyable_functionIFbRKNS_7sstableEEEE (scylla + 0x22b53f2)
#6  0x0000000001cbe576 _ZNSt17_Function_handlerIF23flat_mutation_reader_v2N7seastar13lw_shared_ptrIK6schemaEE13reader_permitRK8intervalIN3dht13ring_positionEERKN5query15partition_sliceEN7tracing15trace_state_ptrENS1_10bool_classIN17streamed_mutation14forwarding_tagEEENSJ_IN15mutation_reader30partition_range_forwarding_tagEEEEZZN7replica5table27sstables_as_snapshot_sourceEvENK3$_0clEvEUlS5_S6_SC_SG_SI_SM_SP_E_E9_M_invokeERKSt9_Any_dataOS5_OS6_SC_SG_OSI_OSM_OSP_ (scylla + 0x1abe576)
#7  0x00000000020ae2ec _ZN5cache30autoupdating_underlying_reader15fast_forward_toEO8intervalIN3dht13ring_positionEER15mutation_sourcem (scylla + 0x1eae2ec)
#8  0x00000000020ad54a _ZN5cache12read_context17create_underlyingEv (scylla + 0x1ead54a)
#9  0x00000000020c5f82 _ZN34single_partition_populating_reader13create_readerEv (scylla + 0x1ec5f82)
#10 0x00000000020c4ab1 _ZN34single_partition_populating_reader11fill_bufferEv (scylla + 0x1ec4ab1)
#11 0x0000000003c0baad _ZN2db4view19view_update_builder11advance_allEv (scylla + 0x3a0baad)
#12 0x0000000003c0c8af _ZN2db4view19view_update_builder10build_someEv (scylla + 0x3a0c8af)
#13 0x0000000003c9b4e4 _ZN2db4view21view_update_generator35generate_and_propagate_view_updatesERKN7replica5tableERKN7seastar13lw_shared_ptrIK6schemaEE13reader_permitOSt6vectorINS0_13view_and_baseESaISF_EEO8mutationNS6_18optimized_optionalI23flat_mutation_reader_v2EEN7tracing15trace_state_ptrENSt6chrono10time_pointI8gc_clockNSQ_8durationIlSt5ratioILl1ELl1EEEEEENSR_INS6_12lowres_clockENST_IlSU_ILl1ELl1000000000EEEEEE (scylla + 0x3a9b4e4)
#14 0x0000000001c64503 _ZNK7replica5table28do_push_view_replica_updatesEN7seastar10shared_ptrIN2db4view21view_update_generatorEEENS1_13lw_shared_ptrIK6schemaEE8mutationNSt6chrono10time_pointINS1_12lowres_clockENSC_8durationIlSt5ratioILl1ELl1000000000EEEEEE15mutation_sourceN7tracing15trace_state_ptrER28reader_concurrency_semaphore8enum_setI10super_enumIN5query15partition_slice6optionEJLST_0ELST_1ELST_2ELST_3ELST_4ELST_5ELST_6ELST_7ELST_8ELST_9ELST_10ELST_11ELST_12ELST_13EEEE (scylla + 0x1a64503)
#15 0x0000000001c6240a _ZNK7replica5table25push_view_replica_updatesEN7seastar10shared_ptrIN2db4view21view_update_generatorEEERKNS1_13lw_shared_ptrIK6schemaEERK15frozen_mutationNSt6chrono10time_pointINS1_12lowres_clockENSG_8durationIlSt5ratioILl1ELl1000000000EEEEEEN7tracing15trace_state_ptrER28reader_concurrency_semaphore (scylla + 0x1a6240a)
#16 0x0000000001ac5678 _ZN7replica8database8do_applyEN7seastar13lw_shared_ptrIK6schemaEERK15frozen_mutationN7tracing15trace_state_ptrENSt6chrono10time_pointINS1_12lowres_clockENSB_8durationIlSt5ratioILl1ELl1000000000EEEEEENS1_10bool_classIN2db14force_sync_tagEEESt7variantIJSt9monostateNSK_24per_partition_rate_limit12account_onlyENSP_19account_and_enforceEEE (scylla + 0x18c5678)
#17 0x0000000001b5cd32 _ZN7seastar20noncopyable_functionIFNS_6futureIvEEPN7replica8databaseENS_13lw_shared_ptrIK6schemaEERK15frozen_mutationN7tracing15trace_state_ptrENSt6chrono10time_pointINS_12lowres_clockENSF_8durationIlSt5ratioILl1ELl1000000000EEEEEENS_10bool_classIN2db14force_sync_tagEEESt7variantIJSt9monostateNSO_24per_partition_rate_limit12account_onlyENST_19account_and_enforceEEEEE17direct_vtable_forISt7_Mem_fnIMS4_FS2_S9_SC_SE_SM_SQ_SW_EEE4callEPKSY_S5_S9_SC_SE_SM_SQ_SW_ (scylla + 0x195cd32)
#18 0x0000000001b7d46c _ZN7seastar20noncopyable_functionIFNS_6futureIvEEPN7replica8databaseENS_13lw_shared_ptrIK6schemaEERK15frozen_mutationN7tracing15trace_state_ptrENSt6chrono10time_pointINS_12lowres_clockENSF_8durationIlSt5ratioILl1ELl1000000000EEEEEENS_10bool_classIN2db14force_sync_tagEEESt7variantIJSt9monostateNSO_24per_partition_rate_limit12account_onlyENST_19account_and_enforceEEEEE17direct_vtable_forIZNS_35inheriting_concrete_execution_stageIS2_JS5_S9_SC_SE_SM_SQ_SW_EE20make_stage_for_groupENS_16scheduling_groupEEUlS5_S9_SC_SE_SM_SQ_SW_E_E4callEPKSY_S5_S9_SC_SE_SM_SQ_SW_ (scylla + 0x197d46c)
#19 0x0000000001b7d039 _ZN7seastar24concrete_execution_stageINS_6futureIvEEJPN7replica8databaseENS_13lw_shared_ptrIK6schemaEERK15frozen_mutationN7tracing15trace_state_ptrENSt6chrono10time_pointINS_12lowres_clockENSF_8durationIlSt5ratioILl1ELl1000000000EEEEEENS_10bool_classIN2db14force_sync_tagEEESt7variantIJSt9monostateNSO_24per_partition_rate_limit12account_onlyENST_19account_and_enforceEEEEE8do_flushEv (scylla + 0x197d039)
#20 0x0000000005e67503 _ZN7seastar11lambda_taskIZNS_15execution_stage5flushEvE3$_0E15run_and_disposeEv (scylla + 0x5c67503)
#21 0x0000000005ecfbe0 _ZN7seastar7reactor9run_tasksERNS0_10task_queueE (scylla + 0x5ccfbe0)
#22 0x0000000005ed11db _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x5cd11db)
#23 0x0000000005ed2458 _ZN7seastar7reactor6do_runEv (scylla + 0x5cd2458)
#24 0x0000000005ef6311 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureERKNS1_11smp_optionsERKNS1_15reactor_optionsEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x5cf6311)
#25 0x0000000005e9182b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x5c9182b)
#26 0x00007f9781581947 start_thread (libc.so.6 + 0x8c947)
#27 0x00007f9781607970 __clone3 (libc.so.6 + 0x112970)
Stack trace of thread 7772:
#0  0x00007f97815f610a read (libc.so.6 + 0x10110a)
#1  0x0000000005f18d65 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x5d18d65)
#2  0x0000000005f19073 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1ERNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x5d19073)
#3  0x0000000005e9182b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x5c9182b)
#4  0x00007f9781581947 start_thread (libc.so.6 + 0x8c947)
#5  0x00007f9781607970 __clone3 (libc.so.6 + 0x112970)
Stack trace of thread 7762:
#0  0x00007f97815f610a read (libc.so.6 + 0x10110a)
#1  0x0000000005f18d65 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x5d18d65)
#2  0x0000000005f19073 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1ERNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x5d19073)
#3  0x0000000005e9182b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x5c9182b)
#4  0x00007f9781581947 start_thread (libc.so.6 + 0x8c947)
#5  0x00007f9781607970 __clone3 (libc.so.6 + 0x112970)
Stack trace of thread 7761:
#0  0x00007f97815f610a read (libc.so.6 + 0x10110a)
#1  0x0000000005f18d65 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x5d18d65)
#2  0x0000000005f19073 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1ERNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x5d19073)
#3  0x0000000005e9182b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x5c9182b)
#4  0x00007f9781581947 start_thread (libc.so.6 + 0x8c947)
#5  0x00007f9781607970 __clone3 (libc.so.6 + 0x112970)
Stack trace of thread 7760:
#0  0x00007f97815f610a read (libc.so.6 + 0x10110a)
#1  0x0000000005f18d65 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x5d18d65)
#2  0x0000000005f19073 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1ERNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x5d19073)
#3  0x0000000005e9182b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x5c9182b)
#4  0x00007f9781581947 start_thread (libc.so.6 + 0x8c947)
#5  0x00007f9781607970 __clone3 (libc.so.6 + 0x112970)
Stack trace of thread 7769:
#0  0x00007f97815f610a read (libc.so.6 + 0x10110a)
#1  0x0000000005f18d65 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x5d18d65)
#2  0x0000000005f19073 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1ERNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x5d19073)
#3  0x0000000005e9182b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x5c9182b)
#4  0x00007f9781581947 start_thread (libc.so.6 + 0x8c947)
#5  0x00007f9781607970 __clone3 (libc.so.6 + 0x112970)
Stack trace of thread 7768:
#0  0x00007f97815f610a read (libc.so.6 + 0x10110a)
#1  0x0000000005f18d65 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x5d18d65)
#2  0x0000000005f19073 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1ERNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x5d19073)
#3  0x0000000005e9182b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x5c9182b)
#4  0x00007f9781581947 start_thread (libc.so.6 + 0x8c947)
#5  0x00007f9781607970 __clone3 (libc.so.6 + 0x112970)
Stack trace of thread 7763:
#0  0x00007f97815f610a read (libc.so.6 + 0x10110a)
#1  0x0000000005f18d65 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x5d18d65)
#2  0x0000000005f19073 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1ERNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x5d19073)
#3  0x0000000005e9182b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x5c9182b)
#4  0x00007f9781581947 start_thread (libc.so.6 + 0x8c947)
#5  0x00007f9781607970 __clone3 (libc.so.6 + 0x112970)
Stack trace of thread 7764:
#0  0x00007f97815f610a read (libc.so.6 + 0x10110a)
#1  0x0000000005f18d65 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x5d18d65)
#2  0x0000000005f19073 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1ERNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x5d19073)
#3  0x0000000005e9182b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x5c9182b)
#4  0x00007f9781581947 start_thread (libc.so.6 + 0x8c947)
#5  0x00007f9781607970 __clone3 (libc.so.6 + 0x112970)
Stack trace of thread 7770:
#0  0x00007f97815f610a read (libc.so.6 + 0x10110a)
#1  0x0000000005f18d65 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x5d18d65)
#2  0x0000000005f19073 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1ERNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x5d19073)
#3  0x0000000005e9182b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x5c9182b)
#4  0x00007f9781581947 start_thread (libc.so.6 + 0x8c947)
#5  0x00007f9781607970 __clone3 (libc.so.6 + 0x112970)
Stack trace of thread 7771:
#0  0x00007f97815f610a read (libc.so.6 + 0x10110a)
#1  0x0000000005f18d65 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x5d18d65)
#2  0x0000000005f19073 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1ERNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x5d19073)
#3  0x0000000005e9182b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x5c9182b)
#4  0x00007f9781581947 start_thread (libc.so.6 + 0x8c947)
#5  0x00007f9781607970 __clone3 (libc.so.6 + 0x112970)
Stack trace of thread 7773:
#0  0x00007f97815f610a read (libc.so.6 + 0x10110a)
#1  0x0000000005f18d65 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x5d18d65)
#2  0x0000000005f19073 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1ERNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x5d19073)
#3  0x0000000005e9182b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x5c9182b)
#4  0x00007f9781581947 start_thread (libc.so.6 + 0x8c947)
#5  0x00007f9781607970 __clone3 (libc.so.6 + 0x112970)
Stack trace of thread 7765:
#0  0x00007f97815f610a read (libc.so.6 + 0x10110a)
#1  0x0000000005f18d65 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x5d18d65)
#2  0x0000000005f19073 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1ERNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x5d19073)
#3  0x0000000005e9182b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x5c9182b)
#4  0x00007f9781581947 start_thread (libc.so.6 + 0x8c947)
#5  0x00007f9781607970 __clone3 (libc.so.6 + 0x112970)
Stack trace of thread 7767:
#0  0x00007f97815f610a read (libc.so.6 + 0x10110a)
#1  0x0000000005f18d65 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x5d18d65)
#2  0x0000000005f19073 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1ERNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x5d19073)
#3  0x0000000005e9182b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x5c9182b)
#4  0x00007f9781581947 start_thread (libc.so.6 + 0x8c947)
#5  0x00007f9781607970 __clone3 (libc.so.6 + 0x112970)
Stack trace of thread 7756:
#0  0x000000000163a681 _ZN13compound_typeIL14allow_prefixes1EE8iteratorC2ERK24managed_bytes_basic_viewIL12mutable_view0EE (scylla + 0x143a681)
#1  0x0000000001dadc44 _ZNK11intrusive_b4treeI10rows_entryXadL_ZNS1_5_linkEEENS1_11tri_compareELm12ELm20ELNS_10key_searchE0ELNS_10with_debugE0EE11lower_boundI21position_in_partitionEENS5_14const_iteratorERKT_RbS2_ (scylla + 0x1badc44)
#2  0x0000000001dac39f _ZN29partition_snapshot_row_cursor13maybe_refreshEv (scylla + 0x1bac39f)
#3  0x0000000001db6bc5 _ZZN30partition_snapshot_flat_readerILb0EN7replica33partition_snapshot_read_accounterEE20lsa_partition_reader13next_intervalERK8intervalI21clustering_key_prefixEENKUlvE_clEv (scylla + 0x1bb6bc5)
#4  0x0000000001db6399 _ZN30partition_snapshot_flat_readerILb0EN7replica33partition_snapshot_read_accounterEE18emit_next_intervalEv (scylla + 0x1bb6399)
#5  0x0000000001db55c2 _ZN8logalloc18allocating_section12with_reserveIZZN30partition_snapshot_flat_readerILb0EN7replica33partition_snapshot_read_accounterEE11fill_bufferEvENKUlvE0_clEvEUlvE_EEDcRNS_6regionEOT_ (scylla + 0x1bb55c2)
#6  0x0000000001db4e5f _ZN30partition_snapshot_flat_readerILb0EN7replica33partition_snapshot_read_accounterEE11fill_bufferEv (scylla + 0x1bb4e5f)
#7  0x00000000021a80ed _ZN22mutation_reader_merger12prepare_nextEv (scylla + 0x1fa80ed)
#8  0x00000000021a8fb0 _ZN22mutation_reader_merger19maybe_produce_batchEv (scylla + 0x1fa8fb0)
#9  0x00000000021b591b _ZN24mutation_fragment_mergerI22mutation_reader_mergerEclEv (scylla + 0x1fb591b)
#10 0x00000000021b518a _ZN7seastar6repeatIZN14merging_readerI22mutation_reader_mergerE11fill_bufferEvEUlvE_EENS_6futureIvEEOT_ (scylla + 0x1fb518a)
#11 0x00000000021b3ee7 _ZN14merging_readerI22mutation_reader_mergerE11fill_bufferEv (scylla + 0x1fb3ee7)
#12 0x0000000003c0baad _ZN2db4view19view_update_builder11advance_allEv (scylla + 0x3a0baad)
#13 0x0000000003c0c8af _ZN2db4view19view_update_builder10build_someEv (scylla + 0x3a0c8af)
#14 0x0000000003c9b4e4 _ZN2db4view21view_update_generator35generate_and_propagate_view_updatesERKN7replica5tableERKN7seastar13lw_shared_ptrIK6schemaEE13reader_permitOSt6vectorINS0_13view_and_baseESaISF_EEO8mutationNS6_18optimized_optionalI23flat_mutation_reader_v2EEN7tracing15trace_state_ptrENSt6chrono10time_pointI8gc_clockNSQ_8durationIlSt5ratioILl1ELl1EEEEEENSR_INS6_12lowres_clockENST_IlSU_ILl1ELl1000000000EEEEEE (scylla + 0x3a9b4e4)
#15 0x0000000001c64503 _ZNK7replica5table28do_push_view_replica_updatesEN7seastar10shared_ptrIN2db4view21view_update_generatorEEENS1_13lw_shared_ptrIK6schemaEE8mutationNSt6chrono10time_pointINS1_12lowres_clockENSC_8durationIlSt5ratioILl1ELl1000000000EEEEEE15mutation_sourceN7tracing15trace_state_ptrER28reader_concurrency_semaphore8enum_setI10super_enumIN5query15partition_slice6optionEJLST_0ELST_1ELST_2ELST_3ELST_4ELST_5ELST_6ELST_7ELST_8ELST_9ELST_10ELST_11ELST_12ELST_13EEEE (scylla + 0x1a64503)
#16 0x0000000001c6240a _ZNK7replica5table25push_view_replica_updatesEN7seastar10shared_ptrIN2db4view21view_update_generatorEEERKNS1_13lw_shared_ptrIK6schemaEERK15frozen_mutationNSt6chrono10time_pointINS1_12lowres_clockENSG_8durationIlSt5ratioILl1ELl1000000000EEEEEEN7tracing15trace_state_ptrER28reader_concurrency_semaphore (scylla + 0x1a6240a)
#17 0x0000000001ac5678 _ZN7replica8database8do_applyEN7seastar13lw_shared_ptrIK6schemaEERK15frozen_mutationN7tracing15trace_state_ptrENSt6chrono10time_pointINS1_12lowres_clockENSB_8durationIlSt5ratioILl1ELl1000000000EEEEEENS1_10bool_classIN2db14force_sync_tagEEESt7variantIJSt9monostateNSK_24per_partition_rate_limit12account_onlyENSP_19account_and_enforceEEE (scylla + 0x18c5678)
#18 0x0000000001b5cd32 _ZN7seastar20noncopyable_functionIFNS_6futureIvEEPN7replica8databaseENS_13lw_shared_ptrIK6schemaEERK15frozen_mutationN7tracing15trace_state_ptrENSt6chrono10time_pointINS_12lowres_clockENSF_8durationIlSt5ratioILl1ELl1000000000EEEEEENS_10bool_classIN2db14force_sync_tagEEESt7variantIJSt9monostateNSO_24per_partition_rate_limit12account_onlyENST_19account_and_enforceEEEEE17direct_vtable_forISt7_Mem_fnIMS4_FS2_S9_SC_SE_SM_SQ_SW_EEE4callEPKSY_S5_S9_SC_SE_SM_SQ_SW_ (scylla + 0x195cd32)
#19 0x0000000001b7d46c _ZN7seastar20noncopyable_functionIFNS_6futureIvEEPN7replica8databaseENS_13lw_shared_ptrIK6schemaEERK15frozen_mutationN7tracing15trace_state_ptrENSt6chrono10time_pointINS_12lowres_clockENSF_8durationIlSt5ratioILl1ELl1000000000EEEEEENS_10bool_classIN2db14force_sync_tagEEESt7variantIJSt9monostateNSO_24per_partition_rate_limit12account_onlyENST_19account_and_enforceEEEEE17direct_vtable_forIZNS_35inheriting_concrete_execution_stageIS2_JS5_S9_SC_SE_SM_SQ_SW_EE20make_stage_for_groupENS_16scheduling_groupEEUlS5_S9_SC_SE_SM_SQ_SW_E_E4callEPKSY_S5_S9_SC_SE_SM_SQ_SW_ (scylla + 0x197d46c)
#20 0x0000000001b7d039 _ZN7seastar24concrete_execution_stageINS_6futureIvEEJPN7replica8databaseENS_13lw_shared_ptrIK6schemaEERK15frozen_mutationN7tracing15trace_state_ptrENSt6chrono10time_pointINS_12lowres_clockENSF_8durationIlSt5ratioILl1ELl1000000000EEEEEENS_10bool_classIN2db14force_sync_tagEEESt7variantIJSt9monostateNSO_24per_partition_rate_limit12account_onlyENST_19account_and_enforceEEEEE8do_flushEv (scylla + 0x197d039)
#21 0x0000000005e67503 _ZN7seastar11lambda_taskIZNS_15execution_stage5flushEvE3$_0E15run_and_disposeEv (scylla + 0x5c67503)
#22 0x0000000005ecfbe0 _ZN7seastar7reactor9run_tasksERNS0_10task_queueE (scylla + 0x5ccfbe0)
#23 0x0000000005ed11db _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x5cd11db)
#24 0x0000000005ed2458 _ZN7seastar7reactor6do_runEv (scylla + 0x5cd2458)
#25 0x0000000005ef6311 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureERKNS1_11smp_optionsERKNS1_15reactor_optionsEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x5cf6311)
#26 0x0000000005e9182b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x5c9182b)
#27 0x00007f9781581947 start_thread (libc.so.6 + 0x8c947)
#28 0x00007f9781607970 __clone3 (libc.so.6 + 0x112970)
Stack trace of thread 7745:
#0  0x0000000002f5f785 _ZN5utils11murmur_hash13hash3_x64_128ESt17basic_string_viewIaSt11char_traitsIaEEmRSt5arrayImLm2EE (scylla + 0x2d5f785)
#1  0x0000000003d0cd50 _ZN5utils15make_hashed_keyESt17basic_string_viewIaSt11char_traitsIaEE (scylla + 0x3b0cd50)
#2  0x0000000003d0d510 _ZN5utils6filter12bloom_filter3addERKSt17basic_string_viewIaSt11char_traitsIaEE (scylla + 0x3b0d510)
#3  0x00000000025a8eb7 _ZN8sstables2mc6writer21consume_new_partitionERKN3dht13decorated_keyE (scylla + 0x23a8eb7)
#4  0x0000000002610a47 _ZN8sstables26compacted_fragments_writer21consume_new_partitionERKN3dht13decorated_keyE (scylla + 0x2410a47)
#5  0x0000000002658dcd _ZN22compact_mutation_stateIL20compact_for_sstables1EE7consumeIN8sstables26compacted_fragments_writerE33noop_compacted_fragments_consumerEEN7seastar10bool_classINS6_18stop_iteration_tagEEEO14clustering_rowRT_RT0_ (scylla + 0x2458dcd)
#6  0x0000000002654908 _ZNO20mutation_fragment_v27consumeIN23flat_mutation_reader_v24impl16consumer_adapterI25compact_for_compaction_v2IN8sstables26compacted_fragments_writerE33noop_compacted_fragments_consumerEEEEEDcRT_ (scylla + 0x2454908)
#7  0x0000000002653bd5 _ZN23flat_mutation_reader_v24impl17consume_in_threadI25compact_for_compaction_v2IN8sstables26compacted_fragments_writerE33noop_compacted_fragments_consumerENS_9no_filterEEEDaT_T0_ (scylla + 0x2453bd5)
#8  0x0000000002649c3a _ZN23flat_mutation_reader_v217consume_in_threadI25compact_for_compaction_v2IN8sstables26compacted_fragments_writerE33noop_compacted_fragments_consumerEEEDaT_ (scylla + 0x2449c3a)
#9  0x00000000026478b2 _ZZZN8sstables10compaction7consumeEvENUl23flat_mutation_reader_v2E_clES1_ENUlvE_clEv (scylla + 0x24478b2)
#10 0x00000000026467d6 _ZN7seastar20noncopyable_functionIFvvEE17direct_vtable_forIZNS_5asyncIZZN8sstables10compaction7consumeEvENUl23flat_mutation_reader_v2E_clES7_EUlvE_JEEENS_8futurizeINSt13invoke_resultIT_JDpT0_EE4typeEE4typeENS_17thread_attributesEOSC_DpOSD_EUlvE_E4callEPKS2_ (scylla + 0x24467d6)
#11 0x00000000062aff47 _ZN7seastar14thread_context4mainEv (scylla + 0x60aff47)
Stack trace of thread 7747:
#0  0x00007ffe5c74e6f3 n/a (linux-vdso.so.1 + 0x6f3)
#1  0x00007ffe5c74e80a n/a (linux-vdso.so.1 + 0x80a)
#2  0x00007f97815cb2fd clock_gettime@@GLIBC_2.17 (libc.so.6 + 0xd62fd)
#3  0x00007f97817d05f5 _ZNSt6chrono3_V212steady_clock3nowEv (libstdc++.so.6 + 0xd95f5)
#4  0x0000000005ed24c3 _ZN7seastar7reactor6do_runEv (scylla + 0x5cd24c3)
#5  0x0000000005ef6311 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureERKNS1_11smp_optionsERKNS1_15reactor_optionsEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x5cf6311)
#6  0x0000000005e9182b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x5c9182b)
#7  0x00007f9781581947 start_thread (libc.so.6 + 0x8c947)
#8  0x00007f9781607970 __clone3 (libc.so.6 + 0x112970)
Stack trace of thread 7766:
#0  0x00007f97815f610a read (libc.so.6 + 0x10110a)
#1  0x0000000005f18d65 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x5d18d65)
#2  0x0000000005f19073 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1ERNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x5d19073)
#3  0x0000000005e9182b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x5c9182b)
#4  0x00007f9781581947 start_thread (libc.so.6 + 0x8c947)
#5  0x00007f9781607970 __clone3 (libc.so.6 + 0x112970)
Stack trace of thread 7748:
#0  0x0000000003ed55b6 _ZNK3dht19murmur3_partitioner9get_tokenERK6schema18partition_key_view (scylla + 0x3cd55b6)
#1  0x0000000003c5fbde _ZZN2db4view21view_update_generator9mutate_MVEN7seastar13lw_shared_ptrIK6schemaEEN3dht5tokenEN5utils14chunked_vectorI26frozen_mutation_and_schemaLm131072EEERNS0_5statsERN7replica8cf_statsEN7tracing15trace_state_ptrENS2_15semaphore_unitsINS2_33default_timeout_exception_factoryENS2_12lowres_clockEEENS2_10bool_classIN7service15allow_hints_tagEEENSO_INS0_24wait_for_all_updates_tagEEEEN3$_0clESB_ (scylla + 0x3a5fbde)
#2  0x0000000003c5f035 _ZZZZN7seastar23max_concurrent_for_eachIN5utils14chunked_vectorI26frozen_mutation_and_schemaLm131072EE13iterator_typeIS3_EES6_ZN2db4view21view_update_generator9mutate_MVENS_13lw_shared_ptrIK6schemaEEN3dht5tokenES4_RNS8_5statsERN7replica8cf_statsEN7tracing15trace_state_ptrENS_15semaphore_unitsINS_33default_timeout_exception_factoryENS_12lowres_clockEEENS_10bool_classIN7service15allow_hints_tagEEENSR_INS8_24wait_for_all_updates_tagEEEE3$_0EENS_6futureIvEET_T0_mOT1_ENKUlRZNS0_IS6_S6_SX_EESZ_S10_S11_mS13_E5stateE_clES15_ENKUlvE0_clEvENUlvE_clEv (scylla + 0x3a5f035)
#3  0x0000000003c5e6cb _ZZZN7seastar23max_concurrent_for_eachIN5utils14chunked_vectorI26frozen_mutation_and_schemaLm131072EE13iterator_typeIS3_EES6_ZN2db4view21view_update_generator9mutate_MVENS_13lw_shared_ptrIK6schemaEEN3dht5tokenES4_RNS8_5statsERN7replica8cf_statsEN7tracing15trace_state_ptrENS_15semaphore_unitsINS_33default_timeout_exception_factoryENS_12lowres_clockEEENS_10bool_classIN7service15allow_hints_tagEEENSR_INS8_24wait_for_all_updates_tagEEEE3$_0EENS_6futureIvEET_T0_mOT1_ENKUlRZNS0_IS6_S6_SX_EESZ_S10_S11_mS13_E5stateE_clES15_ENKUlvE0_clEv (scylla + 0x3a5e6cb)
#4  0x0000000003c1bad2 _ZN2db4view21view_update_generator9mutate_MVEN7seastar13lw_shared_ptrIK6schemaEEN3dht5tokenEN5utils14chunked_vectorI26frozen_mutation_and_schemaLm131072EEERNS0_5statsERN7replica8cf_statsEN7tracing15trace_state_ptrENS2_15semaphore_unitsINS2_33default_timeout_exception_factoryENS2_12lowres_clockEEENS2_10bool_classIN7service15allow_hints_tagEEENSO_INS0_24wait_for_all_updates_tagEEE (scylla + 0x3a1bad2)
#5  0x0000000003c9d340 _ZN2db4view21view_update_generator35generate_and_propagate_view_updatesERKN7replica5tableERKN7seastar13lw_shared_ptrIK6schemaEE13reader_permitOSt6vectorINS0_13view_and_baseESaISF_EEO8mutationNS6_18optimized_optionalI23flat_mutation_reader_v2EEN7tracing15trace_state_ptrENSt6chrono10time_pointI8gc_clockNSQ_8durationIlSt5ratioILl1ELl1EEEEEENSR_INS6_12lowres_clockENST_IlSU_ILl1ELl1000000000EEEEEE (scylla + 0x3a9d340)
#6  0x0000000001c64503 _ZNK7replica5table28do_push_view_replica_updatesEN7seastar10shared_ptrIN2db4view21view_update_generatorEEENS1_13lw_shared_ptrIK6schemaEE8mutationNSt6chrono10time_pointINS1_12lowres_clockENSC_8durationIlSt5ratioILl1ELl1000000000EEEEEE15mutation_sourceN7tracing15trace_state_ptrER28reader_concurrency_semaphore8enum_setI10super_enumIN5query15partition_slice6optionEJLST_0ELST_1ELST_2ELST_3ELST_4ELST_5ELST_6ELST_7ELST_8ELST_9ELST_10ELST_11ELST_12ELST_13EEEE (scylla + 0x1a64503)
#7  0x0000000001c6240a _ZNK7replica5table25push_view_replica_updatesEN7seastar10shared_ptrIN2db4view21view_update_generatorEEERKNS1_13lw_shared_ptrIK6schemaEERK15frozen_mutationNSt6chrono10time_pointINS1_12lowres_clockENSG_8durationIlSt5ratioILl1ELl1000000000EEEEEEN7tracing15trace_state_ptrER28reader_concurrency_semaphore (scylla + 0x1a6240a)
#8  0x0000000001ac5678 _ZN7replica8database8do_applyEN7seastar13lw_shared_ptrIK6schemaEERK15frozen_mutationN7tracing15trace_state_ptrENSt6chrono10time_pointINS1_12lowres_clockENSB_8durationIlSt5ratioILl1ELl1000000000EEEEEENS1_10bool_classIN2db14force_sync_tagEEESt7variantIJSt9monostateNSK_24per_partition_rate_limit12account_onlyENSP_19account_and_enforceEEE (scylla + 0x18c5678)
#9  0x0000000001b5cd32 _ZN7seastar20noncopyable_functionIFNS_6futureIvEEPN7replica8databaseENS_13lw_shared_ptrIK6schemaEERK15frozen_mutationN7tracing15trace_state_ptrENSt6chrono10time_pointINS_12lowres_clockENSF_8durationIlSt5ratioILl1ELl1000000000EEEEEENS_10bool_classIN2db14force_sync_tagEEESt7variantIJSt9monostateNSO_24per_partition_rate_limit12account_onlyENST_19account_and_enforceEEEEE17direct_vtable_forISt7_Mem_fnIMS4_FS2_S9_SC_SE_SM_SQ_SW_EEE4callEPKSY_S5_S9_SC_SE_SM_SQ_SW_ (scylla + 0x195cd32)
#10 0x0000000001b7d46c _ZN7seastar20noncopyable_functionIFNS_6futureIvEEPN7replica8databaseENS_13lw_shared_ptrIK6schemaEERK15frozen_mutationN7tracing15trace_state_ptrENSt6chrono10time_pointINS_12lowres_clockENSF_8durationIlSt5ratioILl1ELl1000000000EEEEEENS_10bool_classIN2db14force_sync_tagEEESt7variantIJSt9monostateNSO_24per_partition_rate_limit12account_onlyENST_19account_and_enforceEEEEE17direct_vtable_forIZNS_35inheriting_concrete_execution_stageIS2_JS5_S9_SC_SE_SM_SQ_SW_EE20make_stage_for_groupENS_16scheduling_groupEEUlS5_S9_SC_SE_SM_SQ_SW_E_E4callEPKSY_S5_S9_SC_SE_SM_SQ_SW_ (scylla + 0x197d46c)
#11 0x0000000001b7d039 _ZN7seastar24concrete_execution_stageINS_6futureIvEEJPN7replica8databaseENS_13lw_shared_ptrIK6schemaEERK15frozen_mutationN7tracing15trace_state_ptrENSt6chrono10time_pointINS_12lowres_clockENSF_8durationIlSt5ratioILl1ELl1000000000EEEEEENS_10bool_classIN2db14force_sync_tagEEESt7variantIJSt9monostateNSO_24per_partition_rate_limit12account_onlyENST_19account_and_enforceEEEEE8do_flushEv (scylla + 0x197d039)
#12 0x0000000005e67503 _ZN7seastar11lambda_taskIZNS_15execution_stage5flushEvE3$_0E15run_and_disposeEv (scylla + 0x5c67503)
#13 0x0000000005ecfbe0 _ZN7seastar7reactor9run_tasksERNS0_10task_queueE (scylla + 0x5ccfbe0)
#14 0x0000000005ed11db _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x5cd11db)
#15 0x0000000005ed2458 _ZN7seastar7reactor6do_runEv (scylla + 0x5cd2458)
#16 0x0000000005ef6311 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureERKNS1_11smp_optionsERKNS1_15reactor_optionsEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x5cf6311)
#17 0x0000000005e9182b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x5c9182b)
#18 0x00007f9781581947 start_thread (libc.so.6 + 0x8c947)
#19 0x00007f9781607970 __clone3 (libc.so.6 + 0x112970)
Stack trace of thread 7758:
#0  0x0000000005ed43f5 _ZN7seastar17smp_message_queue20flush_response_batchEv (scylla + 0x5cd43f5)
#1  0x0000000005ede5e2 _ZN7seastar3smp11poll_queuesEv (scylla + 0x5cde5e2)
#2  0x0000000005f13f4b _ZN7seastar7reactor10smp_pollfn4pollEv (scylla + 0x5d13f4b)
#3  0x0000000005ef52e9 _ZNSt17_Function_handlerIFbvEZN7seastar7reactor6do_runEvE3$_5E9_M_invokeERKSt9_Any_data (scylla + 0x5cf52e9)
#4  0x0000000005ed2496 _ZN7seastar7reactor6do_runEv (scylla + 0x5cd2496)
#5  0x0000000005ef6311 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureERKNS1_11smp_optionsERKNS1_15reactor_optionsEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x5cf6311)
#6  0x0000000005e9182b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x5c9182b)
#7  0x00007f9781581947 start_thread (libc.so.6 + 0x8c947)
#8  0x00007f9781607970 __clone3 (libc.so.6 + 0x112970)
Stack trace of thread 7749:
#0  0x0000000005ed10a7 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x5cd10a7)
#1  0x0000000005ed2458 _ZN7seastar7reactor6do_runEv (scylla + 0x5cd2458)
#2  0x0000000005ef6311 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureERKNS1_11smp_optionsERKNS1_15reactor_optionsEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x5cf6311)
#3  0x0000000005e9182b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x5c9182b)
#4  0x00007f9781581947 start_thread (libc.so.6 + 0x8c947)
#5  0x00007f9781607970 __clone3 (libc.so.6 + 0x112970)
Stack trace of thread 7759:
#0  0x0000000005ed2565 _ZN7seastar7reactor6do_runEv (scylla + 0x5cd2565)
#1  0x0000000005ef6311 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureERKNS1_11smp_optionsERKNS1_15reactor_optionsEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x5cf6311)
#2  0x0000000005e9182b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x5c9182b)
#3  0x00007f9781581947 start_thread (libc.so.6 + 0x8c947)
#4  0x00007f9781607970 __clone3 (libc.so.6 + 0x112970)
Stack trace of thread 7750:
#0  0x00007ffe5c74e75e n/a (linux-vdso.so.1 + 0x75e)
#1  0x00007ffe5c74e80a n/a (linux-vdso.so.1 + 0x80a)
#2  0x00007f97815cb2fd clock_gettime@@GLIBC_2.17 (libc.so.6 + 0xd62fd)
#3  0x00007f97817d05c2 _ZNSt6chrono3_V212system_clock3nowEv (libstdc++.so.6 + 0xd95c2)
#4  0x0000000005ed2479 _ZN7seastar7reactor6do_runEv (scylla + 0x5cd2479)
#5  0x0000000005ef6311 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureERKNS1_11smp_optionsERKNS1_15reactor_optionsEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x5cf6311)
#6  0x0000000005e9182b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x5c9182b)
#7  0x00007f9781581947 start_thread (libc.so.6 + 0x8c947)
#8  0x00007f9781607970 __clone3 (libc.so.6 + 0x112970)
Stack trace of thread 7753:
#0  0x0000000005ed1496 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x5cd1496)
#1  0x0000000005ed2458 _ZN7seastar7reactor6do_runEv (scylla + 0x5cd2458)
#2  0x0000000005ef6311 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureERKNS1_11smp_optionsERKNS1_15reactor_optionsEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x5cf6311)
#3  0x0000000005e9182b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x5c9182b)
#4  0x00007f9781581947 start_thread (libc.so.6 + 0x8c947)
#5  0x00007f9781607970 __clone3 (libc.so.6 + 0x112970)
Stack trace of thread 7752:
#0  0x0000000005f1ada0 _ZN7seastar19reactor_backend_aio23reap_kernel_completionsEv (scylla + 0x5d1ada0)
#1  0x0000000005ef52e9 _ZNSt17_Function_handlerIFbvEZN7seastar7reactor6do_runEvE3$_5E9_M_invokeERKSt9_Any_data (scylla + 0x5cf52e9)
#2  0x0000000005ed2496 _ZN7seastar7reactor6do_runEv (scylla + 0x5cd2496)
#3  0x0000000005ef6311 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureERKNS1_11smp_optionsERKNS1_15reactor_optionsEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x5cf6311)
#4  0x0000000005e9182b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x5c9182b)
#5  0x00007f9781581947 start_thread (libc.so.6 + 0x8c947)
#6  0x00007f9781607970 __clone3 (libc.so.6 + 0x112970)
Stack trace of thread 7751:
#0  0x0000000003ed581d _ZNK3dht19murmur3_partitioner9get_tokenERK6schema18partition_key_view (scylla + 0x3cd581d)
#1  0x0000000003bfae4f _ZN2db4view12view_updates7move_toERN5utils14chunked_vectorI26frozen_mutation_and_schemaLm131072EEE (scylla + 0x39fae4f)
#2  0x0000000003c0d9d0 _ZN2db4view19view_update_builder10build_someEv (scylla + 0x3a0d9d0)
#3  0x0000000003c9b4e4 _ZN2db4view21view_update_generator35generate_and_propagate_view_updatesERKN7replica5tableERKN7seastar13lw_shared_ptrIK6schemaEE13reader_permitOSt6vectorINS0_13view_and_baseESaISF_EEO8mutationNS6_18optimized_optionalI23flat_mutation_reader_v2EEN7tracing15trace_state_ptrENSt6chrono10time_pointI8gc_clockNSQ_8durationIlSt5ratioILl1ELl1EEEEEENSR_INS6_12lowres_clockENST_IlSU_ILl1ELl1000000000EEEEEE (scylla + 0x3a9b4e4)
#4  0x0000000001c64503 _ZNK7replica5table28do_push_view_replica_updatesEN7seastar10shared_ptrIN2db4view21view_update_generatorEEENS1_13lw_shared_ptrIK6schemaEE8mutationNSt6chrono10time_pointINS1_12lowres_clockENSC_8durationIlSt5ratioILl1ELl1000000000EEEEEE15mutation_sourceN7tracing15trace_state_ptrER28reader_concurrency_semaphore8enum_setI10super_enumIN5query15partition_slice6optionEJLST_0ELST_1ELST_2ELST_3ELST_4ELST_5ELST_6ELST_7ELST_8ELST_9ELST_10ELST_11ELST_12ELST_13EEEE (scylla + 0x1a64503)
#5  0x0000000001c6240a _ZNK7replica5table25push_view_replica_updatesEN7seastar10shared_ptrIN2db4view21view_update_generatorEEERKNS1_13lw_shared_ptrIK6schemaEERK15frozen_mutationNSt6chrono10time_pointINS1_12lowres_clockENSG_8durationIlSt5ratioILl1ELl1000000000EEEEEEN7tracing15trace_state_ptrER28reader_concurrency_semaphore (scylla + 0x1a6240a)
#6  0x0000000001ac5678 _ZN7replica8database8do_applyEN7seastar13lw_shared_ptrIK6schemaEERK15frozen_mutationN7tracing15trace_state_ptrENSt6chrono10time_pointINS1_12lowres_clockENSB_8durationIlSt5ratioILl1ELl1000000000EEEEEENS1_10bool_classIN2db14force_sync_tagEEESt7variantIJSt9monostateNSK_24per_partition_rate_limit12account_onlyENSP_19account_and_enforceEEE (scylla + 0x18c5678)
#7  0x0000000001b5cd32 _ZN7seastar20noncopyable_functionIFNS_6futureIvEEPN7replica8databaseENS_13lw_shared_ptrIK6schemaEERK15frozen_mutationN7tracing15trace_state_ptrENSt6chrono10time_pointINS_12lowres_clockENSF_8durationIlSt5ratioILl1ELl1000000000EEEEEENS_10bool_classIN2db14force_sync_tagEEESt7variantIJSt9monostateNSO_24per_partition_rate_limit12account_onlyENST_19account_and_enforceEEEEE17direct_vtable_forISt7_Mem_fnIMS4_FS2_S9_SC_SE_SM_SQ_SW_EEE4callEPKSY_S5_S9_SC_SE_SM_SQ_SW_ (scylla + 0x195cd32)
#8  0x0000000001b7d46c _ZN7seastar20noncopyable_functionIFNS_6futureIvEEPN7replica8databaseENS_13lw_shared_ptrIK6schemaEERK15frozen_mutationN7tracing15trace_state_ptrENSt6chrono10time_pointINS_12lowres_clockENSF_8durationIlSt5ratioILl1ELl1000000000EEEEEENS_10bool_classIN2db14force_sync_tagEEESt7variantIJSt9monostateNSO_24per_partition_rate_limit12account_onlyENST_19account_and_enforceEEEEE17direct_vtable_forIZNS_35inheriting_concrete_execution_stageIS2_JS5_S9_SC_SE_SM_SQ_SW_EE20make_stage_for_groupENS_16scheduling_groupEEUlS5_S9_SC_SE_SM_SQ_SW_E_E4callEPKSY_S5_S9_SC_SE_SM_SQ_SW_ (scylla + 0x197d46c)
#9  0x0000000001b7d039 _ZN7seastar24concrete_execution_stageINS_6futureIvEEJPN7replica8databaseENS_13lw_shared_ptrIK6schemaEERK15frozen_mutationN7tracing15trace_state_ptrENSt6chrono10time_pointINS_12lowres_clockENSF_8durationIlSt5ratioILl1ELl1000000000EEEEEENS_10bool_classIN2db14force_sync_tagEEESt7variantIJSt9monostateNSO_24per_partition_rate_limit12account_onlyENST_19account_and_enforceEEEEE8do_flushEv (scylla + 0x197d039)
#10 0x0000000005e67503 _ZN7seastar11lambda_taskIZNS_15execution_stage5flushEvE3$_0E15run_and_disposeEv (scylla + 0x5c67503)
#11 0x0000000005ecfbe0 _ZN7seastar7reactor9run_tasksERNS0_10task_queueE (scylla + 0x5ccfbe0)
#12 0x0000000005ed11db _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x5cd11db)
#13 0x0000000005ed2458 _ZN7seastar7reactor6do_runEv (scylla + 0x5cd2458)
#14 0x0000000005ef6311 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureERKNS1_11smp_optionsERKNS1_15reactor_optionsEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x5cf6311)
#15 0x0000000005e9182b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x5c9182b)
#16 0x00007f9781581947 start_thread (libc.so.6 + 0x8c947)
#17 0x00007f9781607970 __clone3 (libc.so.6 + 0x112970)
Stack trace of thread 7754:
#0  0x0000000003ed55cc _ZNK3dht19murmur3_partitioner9get_tokenERK6schema18partition_key_view (scylla + 0x3cd55cc)
#1  0x0000000003bfae4f _ZN2db4view12view_updates7move_toERN5utils14chunked_vectorI26frozen_mutation_and_schemaLm131072EEE (scylla + 0x39fae4f)
#2  0x0000000003c0d9d0 _ZN2db4view19view_update_builder10build_someEv (scylla + 0x3a0d9d0)
#3  0x0000000003c9b4e4 _ZN2db4view21view_update_generator35generate_and_propagate_view_updatesERKN7replica5tableERKN7seastar13lw_shared_ptrIK6schemaEE13reader_permitOSt6vectorINS0_13view_and_baseESaISF_EEO8mutationNS6_18optimized_optionalI23flat_mutation_reader_v2EEN7tracing15trace_state_ptrENSt6chrono10time_pointI8gc_clockNSQ_8durationIlSt5ratioILl1ELl1EEEEEENSR_INS6_12lowres_clockENST_IlSU_ILl1ELl1000000000EEEEEE (scylla + 0x3a9b4e4)
#4  0x0000000001c64503 _ZNK7replica5table28do_push_view_replica_updatesEN7seastar10shared_ptrIN2db4view21view_update_generatorEEENS1_13lw_shared_ptrIK6schemaEE8mutationNSt6chrono10time_pointINS1_12lowres_clockENSC_8durationIlSt5ratioILl1ELl1000000000EEEEEE15mutation_sourceN7tracing15trace_state_ptrER28reader_concurrency_semaphore8enum_setI10super_enumIN5query15partition_slice6optionEJLST_0ELST_1ELST_2ELST_3ELST_4ELST_5ELST_6ELST_7ELST_8ELST_9ELST_10ELST_11ELST_12ELST_13EEEE (scylla + 0x1a64503)
#5  0x0000000001c6240a _ZNK7replica5table25push_view_replica_updatesEN7seastar10shared_ptrIN2db4view21view_update_generatorEEERKNS1_13lw_shared_ptrIK6schemaEERK15frozen_mutationNSt6chrono10time_pointINS1_12lowres_clockENSG_8durationIlSt5ratioILl1ELl1000000000EEEEEEN7tracing15trace_state_ptrER28reader_concurrency_semaphore (scylla + 0x1a6240a)
#6  0x0000000001ac5678 _ZN7replica8database8do_applyEN7seastar13lw_shared_ptrIK6schemaEERK15frozen_mutationN7tracing15trace_state_ptrENSt6chrono10time_pointINS1_12lowres_clockENSB_8durationIlSt5ratioILl1ELl1000000000EEEEEENS1_10bool_classIN2db14force_sync_tagEEESt7variantIJSt9monostateNSK_24per_partition_rate_limit12account_onlyENSP_19account_and_enforceEEE (scylla + 0x18c5678)
#7  0x0000000001b5cd32 _ZN7seastar20noncopyable_functionIFNS_6futureIvEEPN7replica8databaseENS_13lw_shared_ptrIK6schemaEERK15frozen_mutationN7tracing15trace_state_ptrENSt6chrono10time_pointINS_12lowres_clockENSF_8durationIlSt5ratioILl1ELl1000000000EEEEEENS_10bool_classIN2db14force_sync_tagEEESt7variantIJSt9monostateNSO_24per_partition_rate_limit12account_onlyENST_19account_and_enforceEEEEE17direct_vtable_forISt7_Mem_fnIMS4_FS2_S9_SC_SE_SM_SQ_SW_EEE4callEPKSY_S5_S9_SC_SE_SM_SQ_SW_ (scylla + 0x195cd32)
#8  0x0000000001b7d46c _ZN7seastar20noncopyable_functionIFNS_6futureIvEEPN7replica8databaseENS_13lw_shared_ptrIK6schemaEERK15frozen_mutationN7tracing15trace_state_ptrENSt6chrono10time_pointINS_12lowres_clockENSF_8durationIlSt5ratioILl1ELl1000000000EEEEEENS_10bool_classIN2db14force_sync_tagEEESt7variantIJSt9monostateNSO_24per_partition_rate_limit12account_onlyENST_19account_and_enforceEEEEE17direct_vtable_forIZNS_35inheriting_concrete_execution_stageIS2_JS5_S9_SC_SE_SM_SQ_SW_EE20make_stage_for_groupENS_16scheduling_groupEEUlS5_S9_SC_SE_SM_SQ_SW_E_E4callEPKSY_S5_S9_SC_SE_SM_SQ_SW_ (scylla + 0x197d46c)
#9  0x0000000001b7d039 _ZN7seastar24concrete_execution_stageINS_6futureIvEEJPN7replica8databaseENS_13lw_shared_ptrIK6schemaEERK15frozen_mutationN7tracing15trace_state_ptrENSt6chrono10time_pointINS_12lowres_clockENSF_8durationIlSt5ratioILl1ELl1000000000EEEEEENS_10bool_classIN2db14force_sync_tagEEESt7variantIJSt9monostateNSO_24per_partition_rate_limit12account_onlyENST_19account_and_enforceEEEEE8do_flushEv (scylla + 0x197d039)
#10 0x0000000005e67503 _ZN7seastar11lambda_taskIZNS_15execution_stage5flushEvE3$_0E15run_and_disposeEv (scylla + 0x5c67503)
#11 0x0000000005ecfbe0 _ZN7seastar7reactor9run_tasksERNS0_10task_queueE (scylla + 0x5ccfbe0)
#12 0x0000000005ed11db _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x5cd11db)
#13 0x0000000005ed2458 _ZN7seastar7reactor6do_runEv (scylla + 0x5cd2458)
#14 0x0000000005ef6311 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureERKNS1_11smp_optionsERKNS1_15reactor_optionsEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x5cf6311)
#15 0x0000000005e9182b _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x5c9182b)
#16 0x00007f9781581947 start_thread (libc.so.6 + 0x8c947)
#17 0x00007f9781607970 __clone3 (libc.so.6 + 0x112970)
Stack trace of thread 7755:
#0  0x00000000025a8dcc _ZN8sstables2mc6writer21consume_new_partitionERKN3dht13decorated_keyE (scylla + 0x23a8dcc)
#1  0x0000000002610a47 _ZN8sstables26compacted_fragments_writer21consume_new_partitionERKN3dht13decorated_keyE (scylla + 0x2410a47)
#2  0x0000000002658dcd _ZN22compact_mutation_stateIL20compact_for_sstables1EE7consumeIN8sstables26compacted_fragments_writerE33noop_compacted_fragments_consumerEEN7seastar10bool_classINS6_18stop_iteration_tagEEEO14clustering_rowRT_RT0_ (scylla + 0x2458dcd)
#3  0x0000000002654908 _ZNO20mutation_fragment_v27consumeIN23flat_mutation_reader_v24impl16consumer_adapterI25compact_for_compaction_v2IN8sstables26compacted_fragments_writerE33noop_compacted_fragments_consumerEEEEEDcRT_ (scylla + 0x2454908)
#4  0x0000000002653bd5 _ZN23flat_mutation_reader_v24impl17consume_in_threadI25compact_for_compaction_v2IN8sstables26compacted_fragments_writerE33noop_compacted_fragments_consumerENS_9no_filterEEEDaT_T0_ (scylla + 0x2453bd5)
#5  0x0000000002649c3a _ZN23flat_mutation_reader_v217consume_in_threadI25compact_for_compaction_v2IN8sstables26compacted_fragments_writerE33noop_compacted_fragments_consumerEEEDaT_ (scylla + 0x2449c3a)
#6  0x00000000026478b2 _ZZZN8sstables10compaction7consumeEvENUl23flat_mutation_reader_v2E_clES1_ENUlvE_clEv (scylla + 0x24478b2)
#7  0x00000000026467d6 _ZN7seastar20noncopyable_functionIFvvEE17direct_vtable_forIZNS_5asyncIZZN8sstables10compaction7consumeEvENUl23flat_mutation_reader_v2E_clES7_EUlvE_JEEENS_8futurizeINSt13invoke_resultIT_JDpT0_EE4typeEE4typeENS_17thread_attributesEOSC_DpOSD_EUlvE_E4callEPKS2_ (scylla + 0x24467d6)
#8  0x00000000062aff47 _ZN7seastar14thread_context4mainEv (scylla + 0x60aff47)
Info about modules can be found in SCT logs by search for 'Coredump Modules info'

download_instructions: gsutil cp gs://upload.scylladb.com/core.scylla.113.23b5b930df264c819a308f973f22996a.7745.1717218305000000/core.scylla.113.23b5b930df264c819a308f973f22996a.7745.1717218305000000.gz . gunzip core.scylla.113.23b5b930df264c819a308f973f22996a.7745.1717218305000000.gz

Impact

Describe the impact this issue causes to the user.

How frequently does it reproduce?

Describe the frequency with how this issue can be reproduced.

Installation details

Cluster size: 5 nodes (n2-highmem-16)

Scylla Nodes used in this run:

OS / Image: https://www.googleapis.com/compute/v1/projects/scylla-images/global/images/scylla-6-1-0-dev-x86-64-2024-06-01t02-24-09 (gce: undefined_region)

Test: longevity-large-partition-200k-pks-4days-gce-test Test id: 74d1069e-7913-4990-a02b-48b9bc65025e Test name: scylla-master/tier1/longevity-large-partition-200k-pks-4days-gce-test Test config file(s):

Logs and commands - Restore Monitor Stack command: `$ hydra investigate show-monitor 74d1069e-7913-4990-a02b-48b9bc65025e` - Restore monitor on AWS instance using [Jenkins job](https://jenkins.scylladb.com/view/QA/job/QA-tools/job/hydra-show-monitor/parambuild/?test_id=74d1069e-7913-4990-a02b-48b9bc65025e) - Show all stored logs command: `$ hydra investigate show-logs 74d1069e-7913-4990-a02b-48b9bc65025e` ## Logs: - **core.scylla-longevity-large-partitions-200k-pks-db-node-74d1069e-0-4-2024-06-01_05-15-26.gz** - [https://storage.cloud.google.com/upload.scylladb.com/core.scylla.113.23b5b930df264c819a308f973f22996a.7745.1717218305000000/core.scylla.113.23b5b930df264c819a308f973f22996a.7745.1717218305000000.gz](https://storage.cloud.google.com/upload.scylladb.com/core.scylla.113.23b5b930df264c819a308f973f22996a.7745.1717218305000000/core.scylla.113.23b5b930df264c819a308f973f22996a.7745.1717218305000000.gz) - **db-cluster-74d1069e.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/74d1069e-7913-4990-a02b-48b9bc65025e/20240601_052020/db-cluster-74d1069e.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/74d1069e-7913-4990-a02b-48b9bc65025e/20240601_052020/db-cluster-74d1069e.tar.gz) - **sct-runner-events-74d1069e.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/74d1069e-7913-4990-a02b-48b9bc65025e/20240601_052020/sct-runner-events-74d1069e.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/74d1069e-7913-4990-a02b-48b9bc65025e/20240601_052020/sct-runner-events-74d1069e.tar.gz) - **sct-74d1069e.log.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/74d1069e-7913-4990-a02b-48b9bc65025e/20240601_052020/sct-74d1069e.log.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/74d1069e-7913-4990-a02b-48b9bc65025e/20240601_052020/sct-74d1069e.log.tar.gz) - **loader-set-74d1069e.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/74d1069e-7913-4990-a02b-48b9bc65025e/20240601_052020/loader-set-74d1069e.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/74d1069e-7913-4990-a02b-48b9bc65025e/20240601_052020/loader-set-74d1069e.tar.gz) - **monitor-set-74d1069e.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/74d1069e-7913-4990-a02b-48b9bc65025e/20240601_052020/monitor-set-74d1069e.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/74d1069e-7913-4990-a02b-48b9bc65025e/20240601_052020/monitor-set-74d1069e.tar.gz) [Jenkins job URL](https://jenkins.scylladb.com/job/scylla-master/job/tier1/job/longevity-large-partition-200k-pks-4days-gce-test/18/) [Argus](https://argus.scylladb.com/test/996501ad-c137-40ef-8dcc-a7d12f0ba3b6/runs?additionalRuns[]=74d1069e-7913-4990-a02b-48b9bc65025e)
mykaul commented 5 months ago

Related to https://github.com/scylladb/scylladb/pull/18979 ?

bhalevy commented 5 months ago 
node-4 got an error of: tablets - SSTable set wasn't found for tablet 103:

Similar to https://github.com/scylladb/scylladb/issues/19052 Might be a duplicate of https://github.com/scylladb/scylladb/issues/18561 as well.

raphaelsc commented 5 months ago

Duplicate of issue found in https://github.com/scylladb/scylladb/issues/19052, so I guess we can close this one as dup.

bhalevy commented 5 months ago

Duplicate of https://github.com/scylladb/scylladb/issues/18561