search

scylladb / scylladb

NoSQL data store using the seastar framework, compatible with Apache Cassandra
http://scylladb.com
GNU Affero General Public License v3.0
13.08k stars 1.24k forks source link

coredump during compaction on node with cdc #9915

Closed aleksbykov closed 2 years ago

aleksbykov commented 2 years ago

Installation details Kernel version: 5.4.0-1035-aws Scylla version (or git commit hash): 4.6.rc2-0.20220102.e8a1cfb6f with build-id 5d7b96e39c909424e8224207a162fc2c82b67214 Cluster size: 6 nodes (i3.4xlarge) Scylla running with shards number (live nodes): longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-1 (54.229.69.26 | 10.0.2.7): 14 shards longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-2 (34.249.197.95 | 10.0.3.238): 14 shards longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-3 (54.229.235.1 | 10.0.0.222): 14 shards longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 (54.195.157.71 | 10.0.2.27): 14 shards longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-5 (54.194.152.213 | 10.0.0.131): 14 shards longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-6 (52.49.242.12 | 10.0.0.95): 14 shards OS (RHEL/CentOS/Ubuntu/AWS AMI): ami-07d92096e7ab05aae (aws: eu-west-1)

Test: longevity-cdc-3d-400gb-test Test name: longevity_test.LongevityTest.test_custom_time Test config file(s):

Issue description During Job cdc-3d.

Nemesis StartStopScrubCompaction run on node6. The nemesis start scrubbing and then stop it. Before starting scrubbing, it is waiting that all previous compactions finished. Cluster have 4 tables with enabled cdc features with varios options, delta, preimage + delta, postimage + delta, preimage + delta + postimage Compactions have been running all the time while nemesis is waiting. But right after that on node4 next coredumpt triggered:

Coredump happened on node 4:

2022-01-13T14:12:18+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |  [shard 7] compaction - [Compact cdc_test.test_table c9ee9110-747a-11ec-b877-ab45375b97bc] Compacted 1 sstables to [/var/lib/scylla/data/cdc_test/test_table-78e04990746011ec912df398a4b6171a/md-3745-big-Data.db:level=2,/var/lib/scylla/data/cdc_test/test_table-78e04990746011ec912df398a4b6171a/md-3759-big-Data.db:level=2]. 210MB to 210MB (~100% of original) in 12200ms = 17MB/s. ~41984 total partitions merged to 41866.
2022-01-13T14:12:18+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |  [shard 7] compaction - [Compact cdc_test.test_table d137a600-747a-11ec-b877-ab45375b97bc] Compacting [/var/lib/scylla/data/cdc_test/test_table-78e04990746011ec912df398a4b6171a/md-3213-big-Data.db:level=1:origin=compaction]
2022-01-13T14:12:19+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |  [shard 12] compaction - [Compact cdc_test.test_table_scylla_cdc_log cf688b50-747a-11ec-8000-ab42375b97bc] Compacted 2 sstables to [/var/lib/scylla/data/cdc_test/test_table_scylla_cdc_log-78e04992746011ec912df398a4b6171a/md-2070-big-Data.db:level=0]. 293MB to 293MB (~99% of original) in 3828ms = 76MB/s. ~1536 total partitions merged to 678.
2022-01-13T14:12:20+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |  [shard 13] compaction - [Compact cdc_test.test_table_preimage c7149610-747a-11ec-8d9c-ab43375b97bc] Compacted 2 sstables to [/var/lib/scylla/data/cdc_test/test_table_preimage-7df76350746011ec95a3f394a4b6171a/md-951-big-Data.db:level=0]. 72MB to 58MB (~80% of original) in 18317ms = 3MB/s. ~20224 total partitions merged to 15852.
2022-01-13T14:12:22+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO | Accepted publickey for scyllaadm from 10.0.1.127 port 59172 ssh2: RSA SHA256:1YmYWh7roz4Y3FSY7N8CetmQNlK/K5FeBPITC8KF0u8
2022-01-13T14:12:22+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO | pam_unix(sshd:session): session opened for user scyllaadm by (uid=0)
2022-01-13T14:12:22+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO | New session 1933 of user scyllaadm.
2022-01-13T14:12:22+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO | Started Session 1933 of user scyllaadm.
2022-01-13T14:12:22+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO | scylla: ./utils/bptree.hh:1699: void bplus::node<unsigned long, cached_file::cached_page, cached_file::page_idx_less_comparator, 12, bplus::key_search::linear, bplus::with_debug::no>::refill(Less) [Key = unsigned long, T = cached_file::cached_page, Less = cached_file::page_idx_less_comparator, NodeSize = 12, Search = bplus::key_search::linear, Debug = bplus::with_debug::no]: Assertion `p._kids[i].n == this' failed.
2022-01-13T14:12:22+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO | Aborting on shard 5.
2022-01-13T14:12:22+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO | Backtrace:
2022-01-13T14:12:22+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |   0x3db44f8
2022-01-13T14:12:22+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |   0x3de5352
2022-01-13T14:12:22+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |   0x7f7db45a4a1f
2022-01-13T14:12:22+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |   /opt/scylladb/libreloc/libc.so.6+0x3d2a1
2022-01-13T14:12:22+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |   /opt/scylladb/libreloc/libc.so.6+0x268a3
2022-01-13T14:12:22+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |   /opt/scylladb/libreloc/libc.so.6+0x26788
2022-01-13T14:12:22+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |   /opt/scylladb/libreloc/libc.so.6+0x35a15
2022-01-13T14:12:22+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |   0x164a5ab
2022-01-13T14:12:22+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |   0x1649e54
2022-01-13T14:12:22+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |   0x134b472
2022-01-13T14:12:22+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |   0x13dd4dd
2022-01-13T14:12:22+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |   0x13e9538
2022-01-13T14:12:22+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |   0x13e50c3
2022-01-13T14:12:22+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |   0x13e21aa
2022-01-13T14:12:22+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |   0x13f94c1
2022-01-13T14:12:22+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |   0x1417f6d
2022-01-13T14:12:22+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |   0x3dc6ec4
2022-01-13T14:12:22+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |   0x3dc82b7
2022-01-13T14:12:22+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |   0x3de58e5
2022-01-13T14:12:22+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |   0x3d9f4fa
2022-01-13T14:12:22+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |   /opt/scylladb/libreloc/libpthread.so.0+0x9298
2022-01-13T14:12:22+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |   /opt/scylladb/libreloc/libc.so.6+0x100352

Decoded bactrace:

void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) at ./build/release/seastar/./seastar/include/seastar/util/backtrace.hh:59
 (inlined by) seastar::backtrace_buffer::append_backtrace() at ./build/release/seastar/./seastar/src/core/reactor.cc:756
 (inlined by) seastar::print_with_backtrace(seastar::backtrace_buffer&, bool) at ./build/release/seastar/./seastar/src/core/reactor.cc:786
seastar::print_with_backtrace(char const*, bool) at ./build/release/seastar/./seastar/src/core/reactor.cc:798
 (inlined by) seastar::sigabrt_action() at ./build/release/seastar/./seastar/src/core/reactor.cc:3716
 (inlined by) operator() at ./build/release/seastar/./seastar/src/core/reactor.cc:3698
 (inlined by) __invoke at ./build/release/seastar/./seastar/src/core/reactor.cc:3694
?? ??:0
?? ??:0
?? ??:0
?? ??:0
?? ??:0
bplus::node<unsigned long, cached_file::cached_page, cached_file::page_idx_less_comparator, 12ul, (bplus::key_search)0, (bplus::with_debug)0>::refill(cached_file::page_idx_less_comparator) at ././utils/bptree.hh:1699
 (inlined by) bplus::node<unsigned long, cached_file::cached_page, cached_file::page_idx_less_comparator, 12ul, (bplus::key_search)0, (bplus::with_debug)0>::remove_from(unsigned long, cached_file::page_idx_less_comparator) at ././utils/bptree.hh:1669
 (inlined by) bplus::node<unsigned long, cached_file::cached_page, cached_file::page_idx_less_comparator, 12ul, (bplus::key_search)0, (bplus::with_debug)0>::remove(unsigned long, cached_file::page_idx_less_comparator) at ././utils/bptree.hh:1794
 (inlined by) bplus::tree<unsigned long, cached_file::cached_page, cached_file::page_idx_less_comparator, 12ul, (bplus::key_search)0, (bplus::with_debug)0>::iterator bplus::tree<unsigned long, cached_file::cached_page, cached_file::page_idx_less_comparator, 12ul, (bplus::key_search)0, (bplus::with_debug)0>::iterator::erase_and_dispose<void (&)(cached_file::cached_page*) noexcept>(void (&)(cached_file::cached_page*) noexcept, cached_file::page_idx_less_comparator) at ././utils/bptree.hh:807
 (inlined by) bplus::tree<unsigned long, cached_file::cached_page, cached_file::page_idx_less_comparator, 12ul, (bplus::key_search)0, (bplus::with_debug)0>::iterator::erase(cached_file::page_idx_less_comparator) at ././utils/bptree.hh:813
cached_file::cached_page::on_evicted() at ././utils/cached_file.hh:475
lru::evict() at ././utils/lru.hh:91
 (inlined by) operator() at ./row_cache.cc:83
 (inlined by) decltype(auto) with_allocator<cache_tracker::cache_tracker(mutation_application_stats&, seastar::bool_class<register_metrics_tag>)::$_0::operator()() const::{lambda()#1}>(allocation_strategy&, cache_tracker::cache_tracker(mutation_application_stats&, seastar::bool_class<register_metrics_tag>)::$_0::operator()() const::{lambda()#1}&&) at ././utils/allocation_strategy.hh:328
 (inlined by) operator() at ./row_cache.cc:70
 (inlined by) seastar::memory::reclaiming_result std::__invoke_impl<seastar::memory::reclaiming_result, cache_tracker::cache_tracker(mutation_application_stats&, seastar::bool_class<register_metrics_tag>)::$_0&>(std::__invoke_other, cache_tracker::cache_tracker(mutation_application_stats&, seastar::bool_class<register_metrics_tag>)::$_0&) at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/invoke.h:61
 (inlined by) std::enable_if<is_invocable_r_v<seastar::memory::reclaiming_result, cache_tracker::cache_tracker(mutation_application_stats&, seastar::bool_class<register_metrics_tag>)::$_0&>, seastar::memory::reclaiming_result>::type std::__invoke_r<seastar::memory::reclaiming_result, cache_tracker::cache_tracker(mutation_application_stats&, seastar::bool_class<register_metrics_tag>)::$_0&>(cache_tracker::cache_tracker(mutation_application_stats&, seastar::bool_class<register_metrics_tag>)::$_0&) at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/invoke.h:114
 (inlined by) std::_Function_handler<seastar::memory::reclaiming_result (), cache_tracker::cache_tracker(mutation_application_stats&, seastar::bool_class<register_metrics_tag>)::$_0>::_M_invoke(std::_Any_data const&) at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/std_function.h:291
std::function<seastar::memory::reclaiming_result ()>::operator()() const at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/std_function.h:560
 (inlined by) logalloc::region_impl::evict_some() at ./utils/logalloc.cc:1855
logalloc::reclaim_from_evictable(logalloc::region_impl&, unsigned long, seastar::bool_class<is_preemptible_tag>) at ./utils/logalloc.cc:2095
logalloc::tracker::impl::compact_and_evict_locked(unsigned long, unsigned long, seastar::bool_class<is_preemptible_tag>) at ./utils/logalloc.cc:2362
logalloc::tracker::impl::reclaim_locked(unsigned long, seastar::bool_class<is_preemptible_tag>) at ./utils/logalloc.cc:2278
logalloc::tracker::impl::reclaim(unsigned long, seastar::bool_class<is_preemptible_tag>) at ./utils/logalloc.cc:2260
 (inlined by) operator() at ./utils/logalloc.cc:508
 (inlined by) seastar::noncopyable_function<void (unsigned long)>::direct_vtable_for<logalloc::tracker::impl::setup_background_reclaim(seastar::scheduling_group)::{lambda(unsigned long)#1}>::call(seastar::noncopyable_function<void (unsigned long)> const*, unsigned long) at ././seastar/include/seastar/util/noncopyable_function.hh:124
seastar::noncopyable_function<void (unsigned long)>::operator()(unsigned long) const at ././seastar/include/seastar/util/noncopyable_function.hh:209
 (inlined by) logalloc::background_reclaimer::main_loop() at ./utils/logalloc.cc:416
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at ./build/release/seastar/./seastar/src/core/reactor.cc:2374
 (inlined by) seastar::reactor::run_some_tasks() at ./build/release/seastar/./seastar/src/core/reactor.cc:2783
seastar::reactor::do_run() at ./build/release/seastar/./seastar/src/core/reactor.cc:2952
operator() at ./build/release/seastar/./seastar/src/core/reactor.cc:4166
 (inlined by) void std::__invoke_impl<void, seastar::smp::configure(boost::program_options::variables_map, seastar::reactor_config)::$_98&>(std::__invoke_other, seastar::smp::configure(boost::program_options::variables_map, seastar::reactor_config)::$_98&) at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/invoke.h:61
 (inlined by) std::enable_if<is_invocable_r_v<void, seastar::smp::configure(boost::program_options::variables_map, seastar::reactor_config)::$_98&>, void>::type std::__invoke_r<void, seastar::smp::configure(boost::program_options::variables_map, seastar::reactor_config)::$_98&>(seastar::smp::configure(boost::program_options::variables_map, seastar::reactor_config)::$_98&) at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/invoke.h:111
 (inlined by) std::_Function_handler<void (), seastar::smp::configure(boost::program_options::variables_map, seastar::reactor_config)::$_98>::_M_invoke(std::_Any_data const&) at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/std_function.h:291
std::function<void ()>::operator()() const at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/std_function.h:560
 (inlined by) seastar::posix_thread::start_routine(void*) at ./build/release/seastar/./seastar/src/core/posix.cc:60
?? ??:0
?? ??:0
?? ??:0

           PID: 19287 (scylla)
           UID: 113 (scylla)
           GID: 119 (scylla)
        Signal: 6 (ABRT)
     Timestamp: Thu 2022-01-13 14:12:22 UTC (1h 52min ago)
  Command Line: /usr/bin/scylla --blocked-reactor-notify-ms 100 --abort-on-lsa-bad-alloc 1 --abort-on-seastar-bad-alloc --abort-on-internal-error 1 --abort-on-ebadf 1 --enable-sstable-key-validation 1 --log-to-syslog 1 --log-to-stdout 0 --default-log-level info --network-stack posix --io-properties-file=/etc/scylla.d/io_properties.yaml --cpuset 1-7,9-15 --lock-memory=1
    Executable: /opt/scylladb/libexec/scylla
 Control Group: /scylla.slice/scylla-server.slice/scylla-server.service
          Unit: scylla-server.service
         Slice: scylla-server.slice
       Boot ID: 400cf49caabd49b399b7376493d30dcc
    Machine ID: 6fa2171583af4b13bec1e274475e7467
      Hostname: longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4
       Storage: /var/lib/systemd/coredump/core.scylla.113.400cf49caabd49b399b7376493d30dcc.19287.1642083142000000000000
       Message: Process 19287 (scylla) of user 113 dumped core.

                Stack trace of thread 19292:
                #0  0x00007f7db39e92a2 raise (libc.so.6 + 0x3d2a2)
                #1  0x00007f7db39d2950 abort (libc.so.6 + 0x26950)
                #2  0x00007f7db39d2789 __assert_fail_base.cold (libc.so.6 + 0x26789)
                #3  0x00007f7db39e1a16 __assert_fail (libc.so.6 + 0x35a16)
                #4  0x000000000164a5ac _ZN5bplus4treeImN11cached_file11cached_pageENS1_24page_idx_less_comparatorELm12ELNS_10key_searchE0ELNS_10with_debugE0EE8iterator5eraseES3_ (scylla + 0x144a5ac)
                #5  0x0000000001649e55 _ZN11cached_file11cached_page10on_evictedEv (scylla + 0x1449e55)
                #6  0x000000000134b473 _ZNSt17_Function_handlerIFN7seastar6memory17reclaiming_resultEvEZN13cache_trackerC1ER26mutation_application_statsNS0_10bool_classI20register_metrics_tagEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x114b473)
                #7  0x00000000013dd4de _ZN8logalloc11region_impl10evict_someEv (scylla + 0x11dd4de)
                #8  0x00000000013e9539 _ZN8logallocL22reclaim_from_evictableERNS_11region_implEmN7seastar10bool_classI18is_preemptible_tagEE (scylla + 0x11e9539)
                #9  0x00000000013e50c4 _ZN8logalloc7tracker4impl24compact_and_evict_lockedEmmN7seastar10bool_classI18is_preemptible_tagEE (scylla + 0x11e50c4)
                #10 0x00000000013e21ab _ZN8logalloc7tracker4impl14reclaim_lockedEmN7seastar10bool_classI18is_preemptible_tagEE (scylla + 0x11e21ab)
                #11 0x00000000013f94c2 _ZN7seastar20noncopyable_functionIFvmEE17direct_vtable_forIZN8logalloc7tracker4impl24setup_background_reclaimENS_16scheduling_groupEEUlmE_E4callEPKS2_m (scylla + 0x11f94c2)
                #12 0x0000000001417f6e _ZN8logalloc20background_reclaimer9main_loopEv.resume (scylla + 0x1217f6e)
                #13 0x0000000003dc6ec5 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x3bc6ec5)
                #14 0x0000000003dc82b8 _ZN7seastar7reactor6do_runEv (scylla + 0x3bc82b8)
                #15 0x0000000003de58e6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3be58e6)
                #16 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb)
                #17 0x00007f7db459a299 start_thread (libpthread.so.0 + 0x9299)
                #18 0x00007f7db3aac353 __clone (libc.so.6 + 0x100353)

                Stack trace of thread 19302:
                #0  0x00007f7db45a394c read (libpthread.so.0 + 0x1294c)
                #1  0x0000000003e08a45 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3c08a45)
                #2  0x0000000003e08db0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3c08db0)
                #3  0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb)
                #4  0x00007f7db459a299 start_thread (libpthread.so.0 + 0x9299)
                #5  0x00007f7db3aac353 __clone (libc.so.6 + 0x100353)

                Stack trace of thread 19301:
                #0  0x00007f7db45a394c read (libpthread.so.0 + 0x1294c)
                #1  0x0000000003e08a45 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3c08a45)
                #2  0x0000000003e08db0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3c08db0)
                #3  0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb)
                #4  0x00007f7db459a299 start_thread (libpthread.so.0 + 0x9299)
                #5  0x00007f7db3aac353 __clone (libc.so.6 + 0x100353)

                Stack trace of thread 19303:
                #0  0x00007f7db45a394c read (libpthread.so.0 + 0x1294c)
                #1  0x0000000003e08a45 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3c08a45)
                #2  0x0000000003e08db0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3c08db0)
                #3  0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb)
                #4  0x00007f7db459a299 start_thread (libpthread.so.0 + 0x9299)
                #5  0x00007f7db3aac353 __clone (libc.so.6 + 0x100353)

                Stack trace of thread 19311:
                #0  0x00007f7db45a394c read (libpthread.so.0 + 0x1294c)
                #1  0x0000000003e08a45 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3c08a45)
                #2  0x0000000003e08db0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3c08db0)
                #3  0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb)
                #4  0x00007f7db459a299 start_thread (libpthread.so.0 + 0x9299)
                #5  0x00007f7db3aac353 __clone (libc.so.6 + 0x100353)

                Stack trace of thread 19307:
                #0  0x00007f7db45a394c read (libpthread.so.0 + 0x1294c)
                #1  0x0000000003e08a45 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3c08a45)
                #2  0x0000000003e08db0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3c08db0)
                #3  0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb)
                #4  0x00007f7db459a299 start_thread (libpthread.so.0 + 0x9299)
                #5  0x00007f7db3aac353 __clone (libc.so.6 + 0x100353)

                Stack trace of thread 19305:
                #0  0x00007f7db45a394c read (libpthread.so.0 + 0x1294c)
                #1  0x0000000003e08a45 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3c08a45)
                #2  0x0000000003e08db0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3c08db0)
                #3  0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb)
                #4  0x00007f7db459a299 start_thread (libpthread.so.0 + 0x9299)
                #5  0x00007f7db3aac353 __clone (libc.so.6 + 0x100353)

                Stack trace of thread 19287:
                #0  0x00000000041814d3 HUF_buildCTable_wksp (scylla + 0x3f814d3)
                #1  0x000000000418352f HUF_compress4X_repeat (scylla + 0x3f8352f)
                #2  0x000000000418a0ea ZSTD_compressLiterals (scylla + 0x3f8a0ea)
                #3  0x000000000417291e ZSTD_compressBlock_internal (scylla + 0x3f7291e)
                #4  0x0000000004174831 ZSTD_compress_frameChunk (scylla + 0x3f74831)
                #5  0x00000000041769c9 ZSTD_compressEnd (scylla + 0x3f769c9)
                #6  0x0000000001647703 _ZNK14zstd_processor8compressEPKcmPcm (scylla + 0x1447703)
                #7  0x000000000193dbf7 _ZN30compressed_file_data_sink_implI11crc32_utilsL24compressed_checksum_mode1EE3putEN7seastar16temporary_bufferIcEE (scylla + 0x173dbf7)
                #8  0x00000000011da0ef _ZN7seastar13output_streamIcE10slow_writeEPKcm (scylla + 0xfda0ef)
                #9  0x00000000018ebea3 _ZN8sstables2mc6writer15write_clusteredERK14clustering_rowm (scylla + 0x16ebea3)
                #10 0x00000000018ed59c _ZN8sstables2mc6writer15write_clusteredI14clustering_rowEEvRKT_ (scylla + 0x16ed59c)
                #11 0x00000000018ecf33 _ZN8sstables2mc6writer7consumeEO14clustering_row (scylla + 0x16ecf33)
                #12 0x0000000001809eaa _ZNO17mutation_fragment7consumeIN20flat_mutation_reader4impl16consumer_adapterIN8sstables14sstable_writerEEEEEDcRT_ (scylla + 0x1609eaa)
                #13 0x0000000001809025 _ZN7seastar20noncopyable_functionIFvvEE17direct_vtable_forIZNS_5asyncIZN8sstables7sstable16write_componentsE20flat_mutation_readermNS_13lw_shared_ptrIK6schemaEERKNS5_21sstable_writer_configE14encoding_statsRKNS_17io_priority_classEE4$_62JEEENS_8futurizeINSt9result_ofIFNSt5decayIT_E4typeEDpNSM_IT0_E4typeEEE4typeEE4typeENS_17thread_attributesEOSN_DpOSQ_EUlvE_E4callEPKS2_ (scylla + 0x1609025)
                #14 0x0000000004037d22 _ZN7seastar14thread_context4mainEv (scylla + 0x3e37d22)

                Stack trace of thread 19288:
                #0  0x00000000033e6d70 _ZN7seastar19lambda_deleter_implIZ29make_tracked_temporary_bufferIhENS_16temporary_bufferIT_EES4_R13reader_permitEUlvE_ED0Ev (scylla + 0x31e6d70)
                #1  0x000000000193c6bb _ZN32compressed_file_data_source_implI11crc32_utilsE3getEv (scylla + 0x173c6bb)
                #2  0x00000000018b81e1 _ZZN7seastar12input_streamIcE7consumeISt17reference_wrapperIN8sstables2mx27data_consume_rows_context_mEEEENS_6futureIvEEOT_ENUlvE_clEv (scylla + 0x16b81e1)
                #3  0x00000000018bcd1a _ZZZN8sstables2mx26mx_sstable_mutation_reader11fill_bufferEvENKUlvE1_clEvENKUlvE0_clEv (scylla + 0x16bcd1a)
                #4  0x00000000018b5048 _ZZN8sstables2mx26mx_sstable_mutation_reader11fill_bufferEvENKUlvE1_clEv (scylla + 0x16b5048)
                #5  0x0000000001895aec _ZN8sstables2mx26mx_sstable_mutation_reader11fill_bufferEv (scylla + 0x1695aec)
                #6  0x000000000189812d _ZThn64_N8sstables2mx26mx_sstable_mutation_reader11fill_bufferEv (scylla + 0x169812d)
                #7  0x000000000160b6af _ZN7seastar8futurizeINS_6futureINS_10bool_classINS_18stop_iteration_tagEEEEEE6invokeIRZN23flat_mutation_reader_v24impl16consume_pausableIZ15downgrade_to_v1S8_EN19transforming_reader8consumerEEENS1_IvEET_EUlvE_JEEES5_OSE_DpOT0_ (scylla + 0x140b6af)
                #8  0x0000000001608451 _ZZ15downgrade_to_v123flat_mutation_reader_v2EN19transforming_reader11fill_bufferEv (scylla + 0x1408451)
                #9  0x000000000154ba42 _ZN22mutation_reader_merger12prepare_nextEv (scylla + 0x134ba42)
                #10 0x000000000154ca4f _ZN22mutation_reader_mergerclEv (scylla + 0x134ca4f)
                #11 0x0000000001584341 _ZZN14merging_readerI22mutation_reader_mergerE11fill_bufferEvENKUlvE_clEv (scylla + 0x1384341)
                #12 0x00000000015834bb _ZN14merging_readerI22mutation_reader_mergerE11fill_bufferEv (scylla + 0x13834bb)
                #13 0x0000000001339c2b _ZZ32consume_mutation_fragments_untilIZN5cache26cache_flat_mutation_reader20read_from_underlyingEvEUlvE_ZNS1_20read_from_underlyingEvEUl17mutation_fragmentE_ZNS1_20read_from_underlyingEvEUlvE0_EN7seastar6futureIvEER20flat_mutation_readerOT_OT0_OT1_ENKUlvE0_clEv (scylla + 0x1139c2b)
                #14 0x0000000001338458 _ZN5cache26cache_flat_mutation_reader14do_fill_bufferEv (scylla + 0x1138458)
                #15 0x0000000001337687 _ZN5cache26cache_flat_mutation_reader14do_fill_bufferEv (scylla + 0x1137687)
                #16 0x00000000012fae13 _ZZN5cache26cache_flat_mutation_reader11fill_bufferEvENKUlvE_clEv (scylla + 0x10fae13)
                #17 0x00000000012f9c61 _ZN5cache26cache_flat_mutation_reader11fill_bufferEv (scylla + 0x10f9c61)
                #18 0x00000000015bebcc _ZN20flat_mutation_reader4impl16fill_buffer_fromIS_EEN7seastar6futureIbEERT_ (scylla + 0x13bebcc)
                #19 0x0000000001328846 _ZN7seastar8futurizeINS_6futureIvEEE22satisfy_with_result_ofIZZNS2_14then_impl_nrvoIZN34single_partition_populating_reader11fill_bufferEvEUlvE_S2_EET0_OT_ENKUlONS_8internal22promise_base_with_typeIvEERS7_ONS_12future_stateINSB_9monostateEEEE_clESE_SF_SJ_EUlvE_EEvSE_SA_ (scylla + 0x1128846)
                #20 0x0000000001328699 _ZN7seastar12continuationINS_8internal22promise_base_with_typeIvEEZN34single_partition_populating_reader11fill_bufferEvEUlvE_ZNS_6futureIvE14then_impl_nrvoIS5_S7_EET0_OT_EUlOS3_RS5_ONS_12future_stateINS1_9monostateEEEE_vE15run_and_disposeEv (scylla + 0x1128699)
                #21 0x0000000003dc6ec5 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x3bc6ec5)
                #22 0x0000000003dc82b8 _ZN7seastar7reactor6do_runEv (scylla + 0x3bc82b8)
                #23 0x0000000003de58e6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3be58e6)
                #24 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb)
                #25 0x00007f7db459a299 start_thread (libpthread.so.0 + 0x9299)
                #26 0x00007f7db3aac353 __clone (libc.so.6 + 0x100353)

                Stack trace of thread 19304:
                #0  0x00007f7db45a394c read (libpthread.so.0 + 0x1294c)
                #1  0x0000000003e08a45 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3c08a45)
                #2  0x0000000003e08db0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3c08db0)
                #3  0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb)
                #4  0x00007f7db459a299 start_thread (libpthread.so.0 + 0x9299)
                #5  0x00007f7db3aac353 __clone (libc.so.6 + 0x100353)

                Stack trace of thread 19306:
                #0  0x00007f7db45a394c read (libpthread.so.0 + 0x1294c)
                #1  0x0000000003e08a45 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3c08a45)
                #2  0x0000000003e08db0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3c08db0)
                #3  0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb)
                #4  0x00007f7db459a299 start_thread (libpthread.so.0 + 0x9299)
                #5  0x00007f7db3aac353 __clone (libc.so.6 + 0x100353)

                Stack trace of thread 19312:
                #0  0x00007f7db45a394c read (libpthread.so.0 + 0x1294c)
                #1  0x0000000003e08a45 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3c08a45)
                #2  0x0000000003e08db0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3c08db0)
                #3  0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb)
                #4  0x00007f7db459a299 start_thread (libpthread.so.0 + 0x9299)
                #5  0x00007f7db3aac353 __clone (libc.so.6 + 0x100353)

                Stack trace of thread 19289:
                #0  0x0000000003dd50ac _ZN7seastar3smp11poll_queuesEv (scylla + 0x3bd50ac)
                #1  0x0000000003e013db _ZN7seastar7reactor10smp_pollfn4pollEv (scylla + 0x3c013db)
                #2  0x0000000003de49c9 _ZNSt17_Function_handlerIFbvEZN7seastar7reactor6do_runEvE4$_80E9_M_invokeERKSt9_Any_data (scylla + 0x3be49c9)
                #3  0x0000000003dc82e6 _ZN7seastar7reactor6do_runEv (scylla + 0x3bc82e6)
                #4  0x0000000003de58e6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3be58e6)
                #5  0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb)
                #6  0x00007f7db459a299 start_thread (libpthread.so.0 + 0x9299)
                #7  0x00007f7db3aac353 __clone (libc.so.6 + 0x100353)

                Stack trace of thread 19308:
                #0  0x00007f7db45a394c read (libpthread.so.0 + 0x1294c)
                #1  0x0000000003e08a45 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3c08a45)
                #2  0x0000000003e08db0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3c08db0)
                #3  0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb)
                #4  0x00007f7db459a299 start_thread (libpthread.so.0 + 0x9299)
                #5  0x00007f7db3aac353 __clone (libc.so.6 + 0x100353)

                Stack trace of thread 19310:
                #0  0x00007f7db45a394c read (libpthread.so.0 + 0x1294c)
                #1  0x0000000003e08a45 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3c08a45)
                #2  0x0000000003e08db0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3c08db0)
                #3  0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb)
                #4  0x00007f7db459a299 start_thread (libpthread.so.0 + 0x9299)
                #5  0x00007f7db3aac353 __clone (libc.so.6 + 0x100353)

                Stack trace of thread 19293:
                #0  0x0000000001731c2e _ZN8sstables12index_reader6readerC2EN7seastar13lw_shared_ptrINS_7sstableEEE13reader_permitRKNS2_17io_priority_classEN7tracing15trace_state_ptrEmmmNS2_10bool_classINS_15use_caching_tagEEE (scylla + 0x1531c2e)
                #1  0x0000000001719ef4 _ZN8sstables21partition_index_cache11get_or_loadIRZNS_12index_reader15advance_to_pageERNS_11index_boundEmEUlmE_EEN7seastar6futureINS0_9entry_ptrEEERKmOT_ (scylla + 0x1519ef4)
                #2  0x000000000171738d _ZN8sstables12index_reader15advance_to_pageERNS_11index_boundEm (scylla + 0x151738d)
                #3  0x00000000017169b3 _ZN8sstables12index_reader10advance_toERNS_11index_boundEN3dht18ring_position_viewE (scylla + 0x15169b3)
                #4  0x0000000001899bc7 _ZN8sstables2mx26mx_sstable_mutation_reader10initializeEv (scylla + 0x1699bc7)
                #5  0x0000000001895b6a _ZN8sstables2mx26mx_sstable_mutation_reader11fill_bufferEv (scylla + 0x1695b6a)
                #6  0x000000000189812d _ZThn64_N8sstables2mx26mx_sstable_mutation_reader11fill_bufferEv (scylla + 0x169812d)
                #7  0x000000000160b6af _ZN7seastar8futurizeINS_6futureINS_10bool_classINS_18stop_iteration_tagEEEEEE6invokeIRZN23flat_mutation_reader_v24impl16consume_pausableIZ15downgrade_to_v1S8_EN19transforming_reader8consumerEEENS1_IvEET_EUlvE_JEEES5_OSE_DpOT0_ (scylla + 0x140b6af)
                #8  0x0000000001608451 _ZZ15downgrade_to_v123flat_mutation_reader_v2EN19transforming_reader11fill_bufferEv (scylla + 0x1408451)
                #9  0x000000000154ba42 _ZN22mutation_reader_merger12prepare_nextEv (scylla + 0x134ba42)
                #10 0x000000000154ca4f _ZN22mutation_reader_mergerclEv (scylla + 0x134ca4f)
                #11 0x0000000001584341 _ZZN14merging_readerI22mutation_reader_mergerE11fill_bufferEvENKUlvE_clEv (scylla + 0x1384341)
                #12 0x00000000015834bb _ZN14merging_readerI22mutation_reader_mergerE11fill_bufferEv (scylla + 0x13834bb)
                #13 0x0000000001326dbe _ZN34single_partition_populating_reader13create_readerEv (scylla + 0x1126dbe)
                #14 0x0000000001324815 _ZN34single_partition_populating_reader11fill_bufferEv (scylla + 0x1124815)
                #15 0x000000000154ba42 _ZN22mutation_reader_merger12prepare_nextEv (scylla + 0x134ba42)
                #16 0x000000000154ca4f _ZN22mutation_reader_mergerclEv (scylla + 0x134ca4f)
                #17 0x0000000001584341 _ZZN14merging_readerI22mutation_reader_mergerE11fill_bufferEvENKUlvE_clEv (scylla + 0x1384341)
                #18 0x00000000015834bb _ZN14merging_readerI22mutation_reader_mergerE11fill_bufferEv (scylla + 0x13834bb)
                #19 0x00000000011b5d10 _ZN20flat_mutation_reader4peekEv (scylla + 0xfb5d10)
                #20 0x000000000117efe8 _ZN5query7querierIL19emit_only_live_rows1EE12consume_pageI20query_result_builderEEDaOT_mjNSt6chrono10time_pointI8gc_clockNS7_8durationIlSt5ratioILl1ELl1EEEEEEN7tracing15trace_state_ptrE (scylla + 0xf7efe8)
                #21 0x0000000001179bb4 _ZN5table5queryEN7seastar13lw_shared_ptrIK6schemaEE13reader_permitRKN5query12read_commandENS6_14result_optionsERKSt6vectorI20nonwrapping_intervalIN3dht13ring_positionEESaISF_EEN7tracing15trace_state_ptrERNS6_21result_memory_limiterENSt6chrono10time_pointINS0_12lowres_clockENSO_8durationIlSt5ratioILl1ELl1000EEEEEEPSt8optionalINS6_7querierIL19emit_only_live_rows1EEEE (scylla + 0xf79bb4)
                #22 0x00000000010f89ac _ZN7seastar20noncopyable_functionIFNS_6futureIvEE13reader_permitEE19indirect_vtable_forIZN8database5queryENS_13lw_shared_ptrIK6schemaEERKN5query12read_commandENSC_14result_optionsERKSt6vectorI20nonwrapping_intervalIN3dht13ring_positionEESaISL_EEN7tracing15trace_state_ptrENSt6chrono10time_pointINS_12lowres_clockENSS_8durationIlSt5ratioILl1ELl1000EEEEEEE4$_38E4callEPKS5_S3_ (scylla + 0xef89ac)
                #23 0x00000000033e7594 _ZN28reader_concurrency_semaphore14execution_loopEv.resume (scylla + 0x31e7594)
                #24 0x0000000003dc6ec5 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x3bc6ec5)
                #25 0x0000000003dc82b8 _ZN7seastar7reactor6do_runEv (scylla + 0x3bc82b8)
                #26 0x0000000003de58e6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3be58e6)
                #27 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb)
                #28 0x00007f7db459a299 start_thread (libpthread.so.0 + 0x9299)
                #29 0x00007f7db3aac353 __clone (libc.so.6 + 0x100353)

                Stack trace of thread 19290:
                #0  0x0000000004180df1 HUF_buildCTable_wksp (scylla + 0x3f80df1)
                #1  0x000000000418352f HUF_compress4X_repeat (scylla + 0x3f8352f)
                #2  0x000000000418a0ea ZSTD_compressLiterals (scylla + 0x3f8a0ea)
                #3  0x000000000417291e ZSTD_compressBlock_internal (scylla + 0x3f7291e)
                #4  0x0000000004174831 ZSTD_compress_frameChunk (scylla + 0x3f74831)
                #5  0x00000000041769c9 ZSTD_compressEnd (scylla + 0x3f769c9)
                #6  0x0000000001647703 _ZNK14zstd_processor8compressEPKcmPcm (scylla + 0x1447703)
                #7  0x000000000193dbf7 _ZN30compressed_file_data_sink_implI11crc32_utilsL24compressed_checksum_mode1EE3putEN7seastar16temporary_bufferIcEE (scylla + 0x173dbf7)
                #8  0x00000000011da0ef _ZN7seastar13output_streamIcE10slow_writeEPKcm (scylla + 0xfda0ef)
                #9  0x00000000018e40b2 _ZN8sstables2mc6writer21consume_new_partitionERKN3dht13decorated_keyE (scylla + 0x16e40b2)
                #10 0x0000000001809e73 _ZNO17mutation_fragment7consumeIN20flat_mutation_reader4impl16consumer_adapterIN8sstables14sstable_writerEEEEEDcRT_ (scylla + 0x1609e73)
                #11 0x0000000001809025 _ZN7seastar20noncopyable_functionIFvvEE17direct_vtable_forIZNS_5asyncIZN8sstables7sstable16write_componentsE20flat_mutation_readermNS_13lw_shared_ptrIK6schemaEERKNS5_21sstable_writer_configE14encoding_statsRKNS_17io_priority_classEE4$_62JEEENS_8futurizeINSt9result_ofIFNSt5decayIT_E4typeEDpNSM_IT0_E4typeEEE4typeEE4typeENS_17thread_attributesEOSN_DpOSQ_EUlvE_E4callEPKS2_ (scylla + 0x1609025)
                #12 0x0000000004037d22 _ZN7seastar14thread_context4mainEv (scylla + 0x3e37d22)

                Stack trace of thread 19314:
                #0  0x00007f7db45a394c read (libpthread.so.0 + 0x1294c)
                #1  0x0000000003e08a45 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3c08a45)
                #2  0x0000000003e08db0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3c08db0)
                #3  0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb)
                #4  0x00007f7db459a299 start_thread (libpthread.so.0 + 0x9299)
                #5  0x00007f7db3aac353 __clone (libc.so.6 + 0x100353)

                Stack trace of thread 19296:
                #0  0x0000000003dd50d8 _ZN7seastar3smp11poll_queuesEv (scylla + 0x3bd50d8)
                #1  0x0000000003e013db _ZN7seastar7reactor10smp_pollfn4pollEv (scylla + 0x3c013db)
                #2  0x0000000003de49c9 _ZNSt17_Function_handlerIFbvEZN7seastar7reactor6do_runEvE4$_80E9_M_invokeERKSt9_Any_data (scylla + 0x3be49c9)
                #3  0x0000000003dc82e6 _ZN7seastar7reactor6do_runEv (scylla + 0x3bc82e6)
                #4  0x0000000003de58e6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3be58e6)
                #5  0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb)
                #6  0x00007f7db459a299 start_thread (libpthread.so.0 + 0x9299)
                #7  0x00007f7db3aac353 __clone (libc.so.6 + 0x100353)

                Stack trace of thread 19291:
                #0  0x00007f7db3b0f984 __memmove_avx_unaligned_erms (libc.so.6 + 0x163984)
                #1  0x000000000172c573 _ZZZN8sstables14index_consumer13consume_entryEONS_28parsed_partition_index_entryEENKUlvE_clEvENKUlvE_clEv (scylla + 0x152c573)
                #2  0x000000000172a876 _ZN8sstables27index_consume_entry_contextINS_14index_consumerEE13process_stateERN7seastar16temporary_bufferIcEE (scylla + 0x152a876)
                #3  0x000000000172735f _ZZN7seastar12input_streamIcE7consumeISt17reference_wrapperIN8sstables27index_consume_entry_contextINS4_14index_consumerEEEEEENS_6futureIvEEOT_ENUlvE_clEv (scylla + 0x152735f)
                #4  0x000000000172f821 _ZN7seastar8internal8repeaterIZNS_12input_streamIcE7consumeISt17reference_wrapperIN8sstables27index_consume_entry_contextINS6_14index_consumerEEEEEENS_6futureIvEEOT_EUlvE_E15run_and_disposeEv (scylla + 0x152f821)
                #5  0x0000000003dc6ec5 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x3bc6ec5)
                #6  0x0000000003dc82b8 _ZN7seastar7reactor6do_runEv (scylla + 0x3bc82b8)
                #7  0x0000000003de58e6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3be58e6)
                #8  0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb)
                #9  0x00007f7db459a299 start_thread (libpthread.so.0 + 0x9299)
                #10 0x00007f7db3aac353 __clone (libc.so.6 + 0x100353)

                Stack trace of thread 19295:
                #0  0x000000000266ae02 _ZN2db9commitlog9add_entryERKN5utils4UUIDERK22commitlog_entry_writerNSt6chrono10time_pointIN7seastar12lowres_clockENS8_8durationIlSt5ratioILl1ELl1000EEEEEE (scylla + 0x246ae02)
                #1  0x00000000010c7a87 _ZN8database20apply_with_commitlogEN7seastar13lw_shared_ptrIK6schemaEER5tableN5utils4UUIDERK15frozen_mutationNSt6chrono10time_pointINS0_12lowres_clockENSC_8durationIlSt5ratioILl1ELl1000EEEEEENS0_10bool_classIN2db14force_sync_tagEEE (scylla + 0xec7a87)
                #2  0x000000000106762f _ZN8database8do_applyEN7seastar13lw_shared_ptrIK6schemaEERK15frozen_mutationN7tracing15trace_state_ptrENSt6chrono10time_pointINS0_12lowres_clockENSA_8durationIlSt5ratioILl1ELl1000EEEEEENS0_10bool_classIN2db14force_sync_tagEEE (scylla + 0xe6762f)
                #3  0x00000000010eb70b _ZN7seastar20noncopyable_functionIFNS_6futureIvEEP8databaseNS_13lw_shared_ptrIK6schemaEERK15frozen_mutationN7tracing15trace_state_ptrENSt6chrono10time_pointINS_12lowres_clockENSE_8durationIlSt5ratioILl1ELl1000EEEEEENS_10bool_classIN2db14force_sync_tagEEEEE17direct_vtable_forISt7_Mem_fnIMS3_FS2_S8_SB_SD_SL_SP_EEE4callEPKSR_S4_S8_SB_SD_SL_SP_ (scylla + 0xeeb70b)
                #4  0x000000000110b28d _ZN7seastar20noncopyable_functionIFNS_6futureIvEEP8databaseNS_13lw_shared_ptrIK6schemaEERK15frozen_mutationN7tracing15trace_state_ptrENSt6chrono10time_pointINS_12lowres_clockENSE_8durationIlSt5ratioILl1ELl1000EEEEEENS_10bool_classIN2db14force_sync_tagEEEEE17direct_vtable_forIZNS_35inheriting_concrete_execution_stageIS2_JS4_S8_SB_SD_SL_SP_EE20make_stage_for_groupENS_16scheduling_groupEEUlS4_S8_SB_SD_SL_SP_E_E4callEPKSR_S4_S8_SB_SD_SL_SP_ (scylla + 0xf0b28d)
                #5  0x00000000010d701c _ZN7seastar24concrete_execution_stageINS_6futureIvEEJP8databaseNS_13lw_shared_ptrIK6schemaEERK15frozen_mutationN7tracing15trace_state_ptrENSt6chrono10time_pointINS_12lowres_clockENSE_8durationIlSt5ratioILl1ELl1000EEEEEENS_10bool_classIN2db14force_sync_tagEEEEE8do_flushEv (scylla + 0xed701c)
                #6  0x0000000003d7c8e6 _ZN7seastar11lambda_taskIZNS_15execution_stage5flushEvE3$_5E15run_and_disposeEv (scylla + 0x3b7c8e6)
                #7  0x0000000003dc6ec5 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x3bc6ec5)
                #8  0x0000000003dc82b8 _ZN7seastar7reactor6do_runEv (scylla + 0x3bc82b8)
                #9  0x0000000003de58e6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3be58e6)
                #10 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb)
                #11 0x00007f7db459a299 start_thread (libpthread.so.0 + 0x9299)
                #12 0x00007f7db3aac353 __clone (libc.so.6 + 0x100353)

                Stack trace of thread 19313:
                #0  0x00007f7db45a394c read (libpthread.so.0 + 0x1294c)
                #1  0x0000000003e08a45 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3c08a45)
                #2  0x0000000003e08db0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3c08db0)
                #3  0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb)
                #4  0x00007f7db459a299 start_thread (libpthread.so.0 + 0x9299)
                #5  0x00007f7db3aac353 __clone (libc.so.6 + 0x100353)

                Stack trace of thread 19297:
                #0  0x0000000001409e37 _ZN8logalloc11region_impl4freeEPvm (scylla + 0x1209e37)
                #1  0x000000000171d671 _ZNSt8__detail9__variant17__gen_vtable_implINS0_12_Multi_arrayIPFvOZNS0_16_Variant_storageILb0EJN7seastar13lw_shared_ptrINS4_14shared_promiseIJEEEEEN8sstables20partition_index_pageEEE8_M_resetEvEUlOT_E_RSt7variantIJS8_SA_EEEJEEESt16integer_sequenceImJLm1EEEE14__visit_invokeESF_SI_ (scylla + 0x151d671)
                #2  0x0000000001649c13 _ZN5bplus4treeImN8sstables21partition_index_cache5entryENS2_19key_less_comparatorELm8ELNS_10key_searchE0ELNS_10with_debugE0EE8iterator5eraseES4_ (scylla + 0x1449c13)
                #3  0x0000000001649595 _ZN8sstables21partition_index_cache5entry10on_evictedEv (scylla + 0x1449595)
                #4  0x000000000134b473 _ZNSt17_Function_handlerIFN7seastar6memory17reclaiming_resultEvEZN13cache_trackerC1ER26mutation_application_statsNS0_10bool_classI20register_metrics_tagEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x114b473)
                #5  0x00000000013dd4de _ZN8logalloc11region_impl10evict_someEv (scylla + 0x11dd4de)
                #6  0x00000000013e9539 _ZN8logallocL22reclaim_from_evictableERNS_11region_implEmN7seastar10bool_classI18is_preemptible_tagEE (scylla + 0x11e9539)
                #7  0x00000000013e50c4 _ZN8logalloc7tracker4impl24compact_and_evict_lockedEmmN7seastar10bool_classI18is_preemptible_tagEE (scylla + 0x11e50c4)
                #8  0x00000000013d49bc _ZN8logalloc12segment_pool16allocate_segmentEm (scylla + 0x11d49bc)
                #9  0x00000000013f3158 _ZN8logalloc18allocating_section7reserveEv (scylla + 0x11f3158)
                #10 0x000000000138a38a _ZN8memtable5applyERK15frozen_mutationRKN7seastar13lw_shared_ptrIK6schemaEEON2db9rp_handleE (scylla + 0x118a38a)
                #11 0x000000000117767c _ZN5table5applyERK15frozen_mutationRKN7seastar13lw_shared_ptrIK6schemaEEON2db9rp_handleE (scylla + 0xf7767c)
                #12 0x00000000010c4c41 _ZN8database15apply_in_memoryERK15frozen_mutationN7seastar13lw_shared_ptrIK6schemaEEON2db9rp_handleENSt6chrono10time_pointINS3_12lowres_clockENSB_8durationIlSt5ratioILl1ELl1000EEEEEE (scylla + 0xec4c41)
                #13 0x00000000010c7bcf _ZN8database20apply_with_commitlogEN7seastar13lw_shared_ptrIK6schemaEER5tableN5utils4UUIDERK15frozen_mutationNSt6chrono10time_pointINS0_12lowres_clockENSC_8durationIlSt5ratioILl1ELl1000EEEEEENS0_10bool_classIN2db14force_sync_tagEEE (scylla + 0xec7bcf)
                #14 0x000000000106762f _ZN8database8do_applyEN7seastar13lw_shared_ptrIK6schemaEERK15frozen_mutationN7tracing15trace_state_ptrENSt6chrono10time_pointINS0_12lowres_clockENSA_8durationIlSt5ratioILl1ELl1000EEEEEENS0_10bool_classIN2db14force_sync_tagEEE (scylla + 0xe6762f)
                #15 0x00000000010eb70b _ZN7seastar20noncopyable_functionIFNS_6futureIvEEP8databaseNS_13lw_shared_ptrIK6schemaEERK15frozen_mutationN7tracing15trace_state_ptrENSt6chrono10time_pointINS_12lowres_clockENSE_8durationIlSt5ratioILl1ELl1000EEEEEENS_10bool_classIN2db14force_sync_tagEEEEE17direct_vtable_forISt7_Mem_fnIMS3_FS2_S8_SB_SD_SL_SP_EEE4callEPKSR_S4_S8_SB_SD_SL_SP_ (scylla + 0xeeb70b)
                #16 0x000000000110b28d _ZN7seastar20noncopyable_functionIFNS_6futureIvEEP8databaseNS_13lw_shared_ptrIK6schemaEERK15frozen_mutationN7tracing15trace_state_ptrENSt6chrono10time_pointINS_12lowres_clockENSE_8durationIlSt5ratioILl1ELl1000EEEEEENS_10bool_classIN2db14force_sync_tagEEEEE17direct_vtable_forIZNS_35inheriting_concrete_execution_stageIS2_JS4_S8_SB_SD_SL_SP_EE20make_stage_for_groupENS_16scheduling_groupEEUlS4_S8_SB_SD_SL_SP_E_E4callEPKSR_S4_S8_SB_SD_SL_SP_ (scylla + 0xf0b28d)
                #17 0x00000000010d701c _ZN7seastar24concrete_execution_stageINS_6futureIvEEJP8databaseNS_13lw_shared_ptrIK6schemaEERK15frozen_mutationN7tracing15trace_state_ptrENSt6chrono10time_pointINS_12lowres_clockENSE_8durationIlSt5ratioILl1ELl1000EEEEEENS_10bool_classIN2db14force_sync_tagEEEEE8do_flushEv (scylla + 0xed701c)
                #18 0x0000000003d7c8e6 _ZN7seastar11lambda_taskIZNS_15execution_stage5flushEvE3$_5E15run_and_disposeEv (scylla + 0x3b7c8e6)
                #19 0x0000000003dc6ec5 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x3bc6ec5)
                #20 0x0000000003dc82b8 _ZN7seastar7reactor6do_runEv (scylla + 0x3bc82b8)
                #21 0x0000000003de58e6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3be58e6)
                #22 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb)
                #23 0x00007f7db459a299 start_thread (libpthread.so.0 + 0x9299)
                #24 0x00007f7db3aac353 __clone (libc.so.6 + 0x100353)

                Stack trace of thread 19298:
                #0  0x00007f7db3b0f86b __memmove_avx_unaligned_erms (libc.so.6 + 0x16386b)
                #1  0x0000000001262253 _ZNK36collection_mutation_view_description9serializeERK13abstract_type (scylla + 0x1062253)
                #2  0x0000000001265fc3 _Z5mergeRK13abstract_type24collection_mutation_viewS2_ (scylla + 0x1065fc3)
                #3  0x000000000147fa75 _ZN3row19apply_monotonicallyERK17column_definitionO25atomic_cell_or_collectionN7seastar18optimized_optionalI9cell_hashEE (scylla + 0x127fa75)
                #4  0x00000000014e13b6 _ZN18compact_radix_tree4treeI13cell_and_hashjE14weed_from_slotIRZN3row12consume_withIZNS4_19apply_monotonicallyERK6schema11column_kindOS4_E4$_17EEvOT_EUljRS1_E_EEbPNS2_13node_head_ptrESD_j (scylla + 0x12e13b6)
                #5  0x000000000147de1b _ZN13deletable_row5applyERK6schemaOS_ (scylla + 0x127de1b)
                #6  0x00000000012d396f _ZN17mutation_fragment5applyERK6schemaOS_ (scylla + 0x10d396f)
                #7  0x0000000001584e2d _ZN7seastar12continuationINS_8internal22promise_base_with_typeINS_18optimized_optionalI17mutation_fragmentEEEEZN24mutation_fragment_mergerI22mutation_reader_mergerEclEvEUlvE_ZNS_6futureIvE14then_impl_nrvoISA_NSB_IS5_EEEET0_OT_EUlOS6_RSA_ONS_12future_stateINS1_9monostateEEEE_vE15run_and_disposeEv (scylla + 0x1384e2d)
                #8  0x0000000003dc6ec5 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x3bc6ec5)
                #9  0x0000000003dc82b8 _ZN7seastar7reactor6do_runEv (scylla + 0x3bc82b8)
                #10 0x0000000003de58e6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3be58e6)
                #11 0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb)
                #12 0x00007f7db459a299 start_thread (libpthread.so.0 + 0x9299)
                #13 0x00007f7db3aac353 __clone (libc.so.6 + 0x100353)

                Stack trace of thread 19299:
                #0  0x0000000003d78a66 _ZN7seastar8internal23execution_stage_manager5flushEv (scylla + 0x3b78a66)
                #1  0x0000000003de49c9 _ZNSt17_Function_handlerIFbvEZN7seastar7reactor6do_runEvE4$_80E9_M_invokeERKSt9_Any_data (scylla + 0x3be49c9)
                #2  0x0000000003dc82e6 _ZN7seastar7reactor6do_runEv (scylla + 0x3bc82e6)
                #3  0x0000000003de58e6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3be58e6)
                #4  0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb)
                #5  0x00007f7db459a299 start_thread (libpthread.so.0 + 0x9299)
                #6  0x00007f7db3aac353 __clone (libc.so.6 + 0x100353)

                Stack trace of thread 19300:
                #0  0x0000000002a2bd00 _ZZNK8sstables8key_view11tri_compareERK6schema18partition_key_viewENKUlSt17basic_string_viewIaSt11char_traitsIaEEE_clES8_ (scylla + 0x282bd00)
                #1  0x0000000002a29c6c _ZNK3dht37ring_position_comparator_for_sstablesclEN8sstables18decorated_key_viewENS_18ring_position_viewE (scylla + 0x2829c6c)
                #2  0x000000000173967b _ZZN8sstables12index_reader34advance_lower_and_check_if_presentEN3dht18ring_position_viewESt8optionalI26position_in_partition_viewEENKUlvE_clEv (scylla + 0x153967b)
                #3  0x0000000001763404 _ZN7seastar12continuationINS_8internal22promise_base_with_typeIbEEZN8sstables12index_reader34advance_lower_and_check_if_presentEN3dht18ring_position_viewESt8optionalI26position_in_partition_viewEEUlvE_ZNS_6futureIvE14then_impl_nrvoISB_NSC_IbEEEET0_OT_EUlOS3_RSB_ONS_12future_stateINS1_9monostateEEEE_vE15run_and_disposeEv (scylla + 0x1563404)
                #4  0x0000000003dc6ec5 _ZN7seastar7reactor14run_some_tasksEv (scylla + 0x3bc6ec5)
                #5  0x0000000003dc82b8 _ZN7seastar7reactor6do_runEv (scylla + 0x3bc82b8)
                #6  0x0000000003de58e6 _ZNSt17_Function_handlerIFvvEZN7seastar3smp9configureEN5boost15program_options13variables_mapENS1_14reactor_configEE4$_98E9_M_invokeERKSt9_Any_data (scylla + 0x3be58e6)
                #7  0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb)
                #8  0x00007f7db459a299 start_thread (libpthread.so.0 + 0x9299)
                #9  0x00007f7db3aac353 __clone (libc.so.6 + 0x100353)

                Stack trace of thread 19309:
                #0  0x00007f7db45a394c read (libpthread.so.0 + 0x1294c)
                #1  0x0000000003e08a45 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla + 0x3c08a45)
                #2  0x0000000003e08db0 _ZNSt17_Function_handlerIFvvEZN7seastar11thread_poolC1EPNS1_7reactorENS1_13basic_sstringIcjLj15ELb1EEEE3$_0E9_M_invokeERKSt9_Any_data (scylla + 0x3c08db0)
                #3  0x0000000003d9f4fb _ZN7seastar12posix_thread13start_routineEPv (scylla + 0x3b9f4fb)
                #4  0x00007f7db459a299 start_thread (libpthread.so.0 + 0x9299)
                #5  0x00007f7db3aac353 __clone (libc.so.6 + 0x100353)

                Stack trace of thread 19294:
                #0  0x0000000001244420 _ZN16atomic_cell_type9make_liveIN27fragmented_temporary_buffer4viewEEE13managed_byteslRKT_ (scylla + 0x1044420)
                #1  0x00000000012443a6 _ZN11atomic_cell9make_liveERK13abstract_typelRKN27fragmented_temporary_buffer4viewEN7seastar10bool_classINS_21collection_member_tagEEE (scylla + 0x10443a6)
                #2  0x00000000018a9c2d _ZN8sstables2mx17mp_row_consumer_m14consume_columnERKNS_18column_translation11column_infoESt17basic_string_viewIaSt11char_traitsIaEEN27fragmented_temporary_buffer4viewElNSt6chrono8durationIlSt5ratioILl1ELl1EEEENSC_10time_pointI8gc_clockSG_EEb (scylla + 0x16a9c2d)
                #3  0x00000000018cc487 _ZN8sstables2mx27data_consume_rows_context_m16do_process_stateEv.resume (scylla + 0x16cc487)
                #4  0x00000000018b87ee _ZN13data_consumer24continuous_data_consumerIN8sstables2mx27data_consume_rows_context_mEEclEN7seastar16temporary_bufferIcEE (scylla + 0x16b87ee)
                #5  0x00000000018b7fbc _ZZN7seastar12input_streamIcE7consumeISt17reference_wrapperIN8sstables2mx27data_consume_rows_context_mEEEENS_6futureIvEEOT_ENUlvE_clEv (scylla + 0x16b7fbc)
                #6  0x00000000018bcd1a _ZZZN8sstables2mx26mx_sstable_mutation_reader11fill_bufferEvENKUlvE1_clEvENKUlvE0_clEv (scylla + 0x16bcd1a)
                #7  0x00000000018b5048 _ZZN8sstables2mx26mx_sstable_mutation_reader11fill_bufferEvENKUlvE1_clEv (scylla + 0x16b5048)
                #8  0x0000000001895aec _ZN8sstables2mx26mx_sstable_mutation_reader11fill_bufferEv (scylla + 0x1695aec)
                #9  0x000000000189812d _ZThn64_N8sstables2mx26mx_sstable_mutation_reader11fill_bufferEv (scylla + 0x169812d)
                #10 0x000000000160b6af _ZN7seastar8futurizeINS_6futureINS_10bool_classINS_18stop_iteration_tagEEEEEE6invokeIRZN23flat_mutation_reader_v24impl16consume_pausableIZ15downgrade_to_v1S8_EN19transforming_reader8consumerEEENS1_IvEET_EUlvE_JEEES5_OSE_DpOT0_ (scylla + 0x140b6af)
                #11 0x0000000001608451 _ZZ15downgrade_to_v123flat_mutation_reader_v2EN19transforming_reader11fill_bufferEv (scylla + 0x1408451)
                #12 0x000000000198b884 _ZZZN8sstables10compaction5setupI33noop_compacted_fragments_consumerEEN7seastar6futureIvEET_ENUl20flat_mutation_readerE_clES7_ENUlvE_clEv (scylla + 0x178b884)
                #13 0x000000000198b5b4 _ZN7seastar20noncopyable_functionIFvvEE17direct_vtable_forIZNS_5asyncIZZN8sstables10compaction5setupI33noop_compacted_fragments_consumerEENS_6futureIvEET_ENUl20flat_mutation_readerE_clESC_EUlvE_JEEENS_8futurizeINSt9result_ofIFNSt5decayISB_E4typeEDpNSH_IT0_E4typeEEE4typeEE4typeENS_17thread_attributesEOSB_DpOSK_EUlvE_E4callEPKS2_ (scylla + 0x178b5b4)
                #14 0x0000000004037d22 _ZN7seastar14thread_context4mainEv (scylla + 0x3e37d22)

After that coredump, scylla was restarted. Scylla started and node status was UN.

2022-01-13T14:17:57+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |  [shard 0] storage_service - Node 10.0.2.7 state jump to normal
2022-01-13T14:17:57+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |  [shard 0] gossip - InetAddress 10.0.2.7 is now UP, status = NORMAL
2022-01-13T14:17:57+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |  [shard 0] gossip - InetAddress 10.0.0.222 is now UP, status = NORMAL
2022-01-13T14:17:57+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |  [shard 0] gossip - InetAddress 10.0.0.95 is now UP, status = NORMAL
2022-01-13T14:17:57+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |  [shard 0] gossip - InetAddress 10.0.0.131 is now UP, status = NORMAL
2022-01-13T14:17:57+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |  [shard 0] gossip - InetAddress 10.0.3.238 is now UP, status = NORMAL
2022-01-13T14:17:57+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |  [shard 5] compaction - [Compact cdc_test.test_table_preimage_postimage_scylla_cdc_log 99372450-747b-11ec-b5ca-d624db9a1917] Compacted 4 sstables to [/var/lib/scylla/data/cdc_test/test_table_preimage_postimage_scylla_cdc_log-85221ad2746011ec9aa6f395a4b6171a/md-2749-big-Data.db:level=0]. 54MB to 50MB (~93% of original) in 3132ms = 16MB/s. ~2176 total partitions merged to 657.
2022-01-13T14:17:57+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |  [shard 0] storage_service - Node 10.0.0.222 state jump to normal
2022-01-13T14:17:58+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |  [shard 0] storage_service - Node 10.0.0.95 state jump to normal
2022-01-13T14:17:58+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |  [shard 0] storage_service - Node 10.0.0.131 state jump to normal
2022-01-13T14:17:59+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |  [shard 0] storage_service - Node 10.0.3.238 state jump to normal

But then next info messages appeared:

2022-01-13T14:18:02+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |  [shard 0] reader_concurrency_semaphore - Semaphore _read_concurrency_sem with 56/100 count and 70713920/171001774 memory resources: timed out, dumping permit diagnostics:
2022-01-13T14:18:02+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |  [shard 8] reader_concurrency_semaphore - Semaphore _read_concurrency_sem with 99/100 count and 56321747/171001774 memory resources: timed out, dumping permit diagnostics:
2022-01-13T14:18:02+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |  [shard 2] reader_concurrency_semaphore - Semaphore _read_concurrency_sem with 100/100 count and 77430733/171001774 memory resources: timed out, dumping permit diagnostics:
2022-01-13T14:18:02+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |  [shard 3] reader_concurrency_semaphore - Semaphore _read_concurrency_sem with 93/100 count and 53645553/171001774 memory resources: timed out, dumping permit diagnostics:
2022-01-13T14:18:02+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |  [shard 12] reader_concurrency_semaphore - Semaphore _read_concurrency_sem with 98/100 count and 59026781/171001774 memory resources: timed out, dumping permit diagnostics:
2022-01-13T14:18:02+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |  [shard 10] reader_concurrency_semaphore - Semaphore _read_concurrency_sem with 100/100 count and 100960974/171001774 memory resources: timed out, dumping permit diagnostics:
2022-01-13T14:18:02+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |  [shard 7] reader_concurrency_semaphore - Semaphore _read_concurrency_sem with 91/100 count and 45776306/171001774 memory resources: timed out, dumping permit diagnostics:
2022-01-13T14:18:02+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |  [shard 9] reader_concurrency_semaphore - Semaphore _read_concurrency_sem with 72/100 count and 58198170/171001774 memory resources: timed out, dumping permit diagnostics:
2022-01-13T14:18:03+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |  [shard 1] reader_concurrency_semaphore - Semaphore _read_concurrency_sem with 58/100 count and 24319972/171001774 memory resources: timed out, dumping permit diagnostics:
2022-01-13T14:18:03+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |  [shard 11] reader_concurrency_semaphore - Semaphore _read_concurrency_sem with 77/100 count and 138865687/171001774 memory resources: timed out, dumping permit diagnostics:

And after that scylla start stopping:

-7714528417292492121, -8237897004717935262, 6894283390732636280, -5527213507624937271, -8535727460191393456, -8661687331876949591, -841835804988239647}
2022-01-13T14:18:10+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |  [shard 0] schema_tables - Schema version changed to b754fe21-52e7-348a-82f5-19987e2c949f
2022-01-13T14:18:10+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |  [shard 0] schema_tables - Schema version changed to b754fe21-52e7-348a-82f5-19987e2c949f
2022-01-13T14:18:10+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |  [shard 0] schema_tables - Schema version changed to b754fe21-52e7-348a-82f5-19987e2c949f
2022-01-13T14:18:10+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |  [shard 0] schema_tables - Schema version changed to b754fe21-52e7-348a-82f5-19987e2c949f
2022-01-13T14:18:11+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |  [shard 0] schema_tables - Schema version changed to b754fe21-52e7-348a-82f5-19987e2c949f
2022-01-13T14:18:11+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO | Accepted publickey for scyllaadm from 10.0.1.127 port 60710 ssh2: RSA SHA256:1YmYWh7roz4Y3FSY7N8CetmQNlK/K5FeBPITC8KF0u8
2022-01-13T14:18:11+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO | pam_unix(sshd:session): session opened for user scyllaadm by (uid=0)
2022-01-13T14:18:11+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO | New session 1998 of user scyllaadm.
2022-01-13T14:18:11+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO | Started Session 1998 of user scyllaadm.
2022-01-13T14:18:12+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO | pam_unix(sshd:session): session closed for user scyllaadm
2022-01-13T14:18:12+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO | session-1998.scope: Succeeded.
2022-01-13T14:18:12+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO | Session 1998 logged out. Waiting for processes to exit.
2022-01-13T14:18:12+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO | Removed session 1998.
2022-01-13T14:18:13+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |  [shard 0] init - Shutting down storage service notifications
2022-01-13T14:18:13+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |  [shard 0] init - Shutting down storage service notifications was successful
2022-01-13T14:18:13+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |  [shard 0] init - Shutting down system distributed keyspace
2022-01-13T14:18:13+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |  [shard 0] init - Shutting down system distributed keyspace was successful
2022-01-13T14:18:13+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |  [shard 0] init - Shutting down storage service notifications
2022-01-13T14:18:13+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |  [shard 0] init - Shutting down storage service notifications was successful
2022-01-13T14:18:13+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |  [shard 0] init - Shutting down sstables loader API
2022-01-13T14:18:13+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !    INFO |  [shard 0] init - Shutting down sstables loader API was successful

And goes down with error:

2022-01-13T14:18:27+00:00 longevity-cdc-3d-400gb-4-6-db-node-760d9ed7-4 !     ERR |  [shard 0] init - Startup failed: exceptions::mutation_write_timeout_exception (Operation timed out for system.local - received only 0 responses from 1 CL=ONE.)

Restore Monitor Stack command: $ hydra investigate show-monitor 760d9ed7-10f3-4912-ba85-fbc3918db0ab Restore monitor on AWS instance using Jenkins job Show all stored logs command: $ hydra investigate show-logs 760d9ed7-10f3-4912-ba85-fbc3918db0ab

Test id: 760d9ed7-10f3-4912-ba85-fbc3918db0ab

Logs: grafana - https://cloudius-jenkins-test.s3.amazonaws.com/760d9ed7-10f3-4912-ba85-fbc3918db0ab/20220113_155406/grafana-screenshot-longevity-cdc-3d-400gb-test-scylla-per-server-metrics-nemesis-20220113_155640-longevity-cdc-3d-400gb-4-6-monitor-node-760d9ed7-1.png grafana - https://cloudius-jenkins-test.s3.amazonaws.com/760d9ed7-10f3-4912-ba85-fbc3918db0ab/20220113_155406/grafana-screenshot-overview-20220113_155406-longevity-cdc-3d-400gb-4-6-monitor-node-760d9ed7-1.png db-cluster - https://cloudius-jenkins-test.s3.amazonaws.com/760d9ed7-10f3-4912-ba85-fbc3918db0ab/20220113_160435/db-cluster-760d9ed7.tar.gz loader-set - https://cloudius-jenkins-test.s3.amazonaws.com/760d9ed7-10f3-4912-ba85-fbc3918db0ab/20220113_160435/loader-set-760d9ed7.tar.gz monitor-set - https://cloudius-jenkins-test.s3.amazonaws.com/760d9ed7-10f3-4912-ba85-fbc3918db0ab/20220113_160435/monitor-set-760d9ed7.tar.gz sct - https://cloudius-jenkins-test.s3.amazonaws.com/760d9ed7-10f3-4912-ba85-fbc3918db0ab/20220113_160435/sct-runner-760d9ed7.tar.gz

Jenkins job URL

slivne commented 2 years ago

AFAIK 4.6.rc2 includes tomeks patches fixing the issues we had in the cache with index caching fc312b3021aab6938dcf4416998343b38cd5e765 - so this is a new issue

This is blocking 4.6 from going out

avikivity commented 2 years ago

Already backported to all vulnerable branches, removing "Backport candidate" label.