search

scylladb / scylladb

NoSQL data store using the seastar framework, compatible with Apache Cassandra
http://scylladb.com
GNU Affero General Public License v3.0
13.53k stars 1.29k forks source link

[DTEST][DEBUG]: 3 nodeas aborted with coredump during test test_drop_index_while_building #20216

Open aleksbykov opened 2 months ago

aleksbykov commented 2 months ago

Scylla version 6.0.3-0.20240808.a56f7ce21ad4 with build-id 66a8b676ee9e21374a3b46538c930879e592260f

During test https://jenkins.scylladb.com/view/scylla-6.0/job/scylla-6.0/job/dtest-debug/24/testReport/secondary_indexes_test/TestSecondaryIndexes/Run_Dtest_Parallel_Cloud_Machines___FullDtest___full_split016___test_drop_index_while_building/

3 nodes aborted with coredump

INFO  2024-08-11 17:35:16,650 [shard 0: gms] view - Stopping to build view keyspace1.idx_index
INFO  2024-08-11 17:35:16,651 [shard 0: gms] query_processor - Column definitions for keyspace1.standard1 changed, invalidating related prepared statements
ERROR 2024-08-11 17:35:16,655 [shard 0: gms] view - Schema version used for view updates (0ff65800-5808-11ef-97df-864161ac8fa1) does not match the current base schema version of the view (0cf80c20-5808-11ef-98ff-41f97bc8141e) for view keyspace1.idx_index of keyspace1.standard1, at: 0xd812f8a /jenkins/workspace/scylla-6.0/dtest-debug/scylla/.ccm/scylla-repository/a56f7ce21ad4fb60c5f68abe2975d1b8f205f2e0/libreloc/libseastar.so+0x5682a2c /jenkins/workspace/scylla-6.0/dtest-debug/scylla/.ccm/scylla-repository/a56f7ce21ad4fb60c5f68abe2975d1b8f205f2e0/libreloc/libseastar.so+0x5682787 /jenkins/workspace/scylla-6.0/dtest-debug/scylla/.ccm/scylla-repository/a56f7ce21ad4fb60c5f68abe2975d1b8f205f2e0/libreloc/libseastar.so+0x5684b05 /jenkins/workspace/scylla-6.0/dtest-debug/scylla/.ccm/scylla-repository/a56f7ce21ad4fb60c5f68abe2975d1b8f205f2e0/libreloc/libseastar.so+0x568555f /jenkins/workspace/scylla-6.0/dtest-debug/scylla/.ccm/scylla-repository/a56f7ce21ad4fb60c5f68abe2975d1b8f205f2e0/libreloc/libseastar.so+0x448609e /jenkins/workspace/scylla-6.0/dtest-debug/scylla/.ccm/scylla-repository/a56f7ce21ad4fb60c5f68abe2975d1b8f205f2e0/libreloc/libseastar.so+0x4485c56 0x123c9628 0x124c7f26 0x124878ff 0x1248494f 0x1248d645 0x124832f6 0x124830a4 0x12482389 0x12481b1d 0x1248181b 0x123e9554 0x1244718c /jenkins/workspace/scylla-6.0/dtest-debug/scylla/.ccm/scylla-repository/a56f7ce21ad4fb60c5f68abe2975d1b8f205f2e0/libreloc/libseastar.so+0x434fe2d /jenkins/workspace/scylla-6.0/dtest-debug/scylla/.ccm/scylla-repository/a56f7ce21ad4fb60c5f68abe2975d1b8f205f2e0/libreloc/libseastar.so+0x4c04327
   --------
   seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::noncopyable_function<seastar::future<void> ()>, seastar::future<void>::then_impl_nrvo<seastar::noncopyable_function<seastar::future<void> ()>, seastar::future<void> >(seastar::noncopyable_function<seastar::future<void> ()>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> ()>&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>
   --------
   seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>, seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)> >(seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>
   --------
   seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>, seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)> >(seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>
   --------
   seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>, seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)> >(seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>
Aborting on shard 0.
Backtrace:
  0xd812f8a
  /jenkins/workspace/scylla-6.0/dtest-debug/scylla/.ccm/scylla-repository/a56f7ce21ad4fb60c5f68abe2975d1b8f205f2e0/libreloc/libseastar.so+0x488563c
  /jenkins/workspace/scylla-6.0/dtest-debug/scylla/.ccm/scylla-repository/a56f7ce21ad4fb60c5f68abe2975d1b8f205f2e0/libreloc/libseastar.so+0x4885293
  /jenkins/workspace/scylla-6.0/dtest-debug/scylla/.ccm/scylla-repository/a56f7ce21ad4fb60c5f68abe2975d1b8f205f2e0/libreloc/libseastar.so+0x46bffc8
  /jenkins/workspace/scylla-6.0/dtest-debug/scylla/.ccm/scylla-repository/a56f7ce21ad4fb60c5f68abe2975d1b8f205f2e0/libreloc/libseastar.so+0x46f0b95
  /jenkins/workspace/scylla-6.0/dtest-debug/scylla/.ccm/scylla-repository/a56f7ce21ad4fb60c5f68abe2975d1b8f205f2e0/libreloc/libseastar.so+0x47eb7a9
  /jenkins/workspace/scylla-6.0/dtest-debug/scylla/.ccm/scylla-repository/a56f7ce21ad4fb60c5f68abe2975d1b8f205f2e0/libreloc/libseastar.so+0x47ebaee
  /jenkins/workspace/scylla-6.0/dtest-debug/scylla/.ccm/scylla-repository/a56f7ce21ad4fb60c5f68abe2975d1b8f205f2e0/libreloc/libseastar.so+0x47eb8fe
  /jenkins/workspace/scylla-6.0/dtest-debug/scylla/.ccm/scylla-repository/a56f7ce21ad4fb60c5f68abe2975d1b8f205f2e0/libreloc/libc.so.6+0x3dbaf
  /jenkins/workspace/scylla-6.0/dtest-debug/scylla/.ccm/scylla-repository/a56f7ce21ad4fb60c5f68abe2975d1b8f205f2e0/libreloc/libc.so.6+0x8e883
  /jenkins/workspace/scylla-6.0/dtest-debug/scylla/.ccm/scylla-repository/a56f7ce21ad4fb60c5f68abe2975d1b8f205f2e0/libreloc/libc.so.6+0x3dafd
  /jenkins/workspace/scylla-6.0/dtest-debug/scylla/.ccm/scylla-repository/a56f7ce21ad4fb60c5f68abe2975d1b8f205f2e0/libreloc/libc.so.6+0x2687e
  /jenkins/workspace/scylla-6.0/dtest-debug/scylla/.ccm/scylla-repository/a56f7ce21ad4fb60c5f68abe2975d1b8f205f2e0/libreloc/libseastar.so+0x4485c7e
  0x123c9628
  0x124c7f26
  0x124878ff
  0x1248494f
  0x1248d645
  0x124832f6
  0x124830a4
  0x12482389
  0x12481b1d
  0x1248181b
  0x123e9554
  0x1244718c
  /jenkins/workspace/scylla-6.0/dtest-debug/scylla/.ccm/scylla-repository/a56f7ce21ad4fb60c5f68abe2975d1b8f205f2e0/libreloc/libseastar.so+0x434fe2d
  /jenkins/workspace/scylla-6.0/dtest-debug/scylla/.ccm/scylla-repository/a56f7ce21ad4fb60c5f68abe2975d1b8f205f2e0/libreloc/libseastar.so+0x4c04327

Logs: https://jenkins.scylladb.com/job/scylla-6.0/job/dtest-debug/24/artifact/logs-full.debug.016/1723397803029_secondary_indexes_test.py%3A%3ATestSecondaryIndexes%3A%3Atest_drop_index_while_building/

aleksbykov commented 2 months ago

Decoded backtrace

[Backtrace #0]
__interceptor_backtrace at ??:?
void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) at ./build/debug/seastar/./seastar/include/seastar/util/backtrace.hh:68
seastar::backtrace_buffer::append_backtrace() at ./build/debug/seastar/./seastar/src/core/reactor.cc:825
seastar::print_with_backtrace(seastar::backtrace_buffer&, bool) at ./build/debug/seastar/./seastar/src/core/reactor.cc:855
seastar::print_with_backtrace(char const*, bool) at ./build/debug/seastar/./seastar/src/core/reactor.cc:867
seastar::sigabrt_action() at ./build/debug/seastar/./seastar/src/core/reactor.cc:4071
seastar::install_oneshot_signal_handler<6, (void (*)())(&seastar::sigabrt_action)>()::{lambda(int, siginfo_t*, void*)#1}::operator()(int, siginfo_t*, void*) const at ./build/debug/seastar/./seastar/src/core/reactor.cc:4047
seastar::install_oneshot_signal_handler<6, (void (*)())(&seastar::sigabrt_action)>()::{lambda(int, siginfo_t*, void*)#1}::__invoke(int, siginfo_t*, void*) at ./build/debug/seastar/./seastar/src/core/reactor.cc:4043
/data/scylla-s3-reloc.cache/by-build-id/66a8b676ee9e21374a3b46538c930879e592260f/extracted/scylla/libreloc/libc.so.6: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=9148cab1b932d44ef70e306e9c02ee38d06cad51, for GNU/Linux 3.2.0, not stripped

__GI___sigaction at :?
__pthread_kill_implementation at ??:?
__GI_raise at :?
__GI_abort at :?
seastar::on_internal_error(seastar::logger&, std::basic_string_view<char, std::char_traits<char> >) at ./build/debug/seastar/./seastar/src/core/on_internal_error.cc:57
db::view::make_view_update_builder(data_dictionary::database, replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_0::operator()(db::view::view_and_base) const at ./db/view/view.cc:1502
 (inlined by) db::view::view_updates boost::range_detail::default_constructible_unary_fn_wrapper<db::view::make_view_update_builder(data_dictionary::database, replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_0, db::view::view_updates>::operator()<db::view::view_and_base>(db::view::view_and_base&) const at /usr/include/boost/range/detail/default_constructible_unary_fn.hpp:62
 (inlined by) boost::iterators::transform_iterator<boost::range_detail::default_constructible_unary_fn_wrapper<db::view::make_view_update_builder(data_dictionary::database, replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_0, db::view::view_updates>, __gnu_cxx::__normal_iterator<db::view::view_and_base*, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > >, boost::use_default, boost::use_default>::dereference() const at /usr/include/boost/iterator/transform_iterator.hpp:126
 (inlined by) boost::iterators::transform_iterator<boost::range_detail::default_constructible_unary_fn_wrapper<db::view::make_view_update_builder(data_dictionary::database, replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_0, db::view::view_updates>, __gnu_cxx::__normal_iterator<db::view::view_and_base*, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > >, boost::use_default, boost::use_default>::reference boost::iterators::iterator_core_access::dereference<boost::iterators::transform_iterator<boost::range_detail::default_constructible_unary_fn_wrapper<db::view::make_view_update_builder(data_dictionary::database, replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_0, db::view::view_updates>, __gnu_cxx::__normal_iterator<db::view::view_and_base*, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > >, boost::use_default, boost::use_default> >(boost::iterators::transform_iterator<boost::range_detail::default_constructible_unary_fn_wrapper<db::view::make_view_update_builder(data_dictionary::database, replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_0, db::view::view_updates>, __gnu_cxx::__normal_iterator<db::view::view_and_base*, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > >, boost::use_default, boost::use_default> const&) at /usr/include/boost/iterator/iterator_facade.hpp:550
 (inlined by) boost::iterators::detail::iterator_facade_base<boost::iterators::transform_iterator<boost::range_detail::default_constructible_unary_fn_wrapper<db::view::make_view_update_builder(data_dictionary::database, replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_0, db::view::view_updates>, __gnu_cxx::__normal_iterator<db::view::view_and_base*, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > >, boost::use_default, boost::use_default>, db::view::view_updates, boost::iterators::random_access_traversal_tag, db::view::view_updates, long, false, false>::operator*() const at /usr/include/boost/iterator/iterator_facade.hpp:656
 (inlined by) void std::vector<db::view::view_updates, std::allocator<db::view::view_updates> >::_M_range_initialize<boost::iterators::transform_iterator<boost::range_detail::default_constructible_unary_fn_wrapper<db::view::make_view_update_builder(data_dictionary::database, replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_0, db::view::view_updates>, __gnu_cxx::__normal_iterator<db::view::view_and_base*, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > >, boost::use_default, boost::use_default> >(boost::iterators::transform_iterator<boost::range_detail::default_constructible_unary_fn_wrapper<db::view::make_view_update_builder(data_dictionary::database, replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_0, db::view::view_updates>, __gnu_cxx::__normal_iterator<db::view::view_and_base*, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > >, boost::use_default, boost::use_default>, boost::iterators::transform_iterator<boost::range_detail::default_constructible_unary_fn_wrapper<db::view::make_view_update_builder(data_dictionary::database, replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_0, db::view::view_updates>, __gnu_cxx::__normal_iterator<db::view::view_and_base*, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > >, boost::use_default, boost::use_default>, std::input_iterator_tag) at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/bits/stl_vector.h:1670
 (inlined by) std::vector<db::view::view_updates, std::allocator<db::view::view_updates> >::vector<boost::iterators::transform_iterator<boost::range_detail::default_constructible_unary_fn_wrapper<db::view::make_view_update_builder(data_dictionary::database, replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_0, db::view::view_updates>, __gnu_cxx::__normal_iterator<db::view::view_and_base*, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > >, boost::use_default, boost::use_default>, void>(boost::iterators::transform_iterator<boost::range_detail::default_constructible_unary_fn_wrapper<db::view::make_view_update_builder(data_dictionary::database, replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_0, db::view::view_updates>, __gnu_cxx::__normal_iterator<db::view::view_and_base*, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > >, boost::use_default, boost::use_default>, boost::iterators::transform_iterator<boost::range_detail::default_constructible_unary_fn_wrapper<db::view::make_view_update_builder(data_dictionary::database, replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_0, db::view::view_updates>, __gnu_cxx::__normal_iterator<db::view::view_and_base*, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > >, boost::use_default, boost::use_default>, std::allocator<db::view::view_updates> const&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/bits/stl_vector.h:708
 (inlined by) std::vector<db::view::view_updates, std::allocator<db::view::view_updates> > boost::copy_range<std::vector<db::view::view_updates, std::allocator<db::view::view_updates> >, boost::range_detail::transformed_range<db::view::make_view_update_builder(data_dictionary::database, replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_0, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > > >(boost::range_detail::transformed_range<db::view::make_view_update_builder(data_dictionary::database, replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_0, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > > const&) at /usr/include/boost/range/iterator_range_core.hpp:842
 (inlined by) db::view::make_view_update_builder(data_dictionary::database, replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >) at ./db/view/view.cc:1500
db::view::view_update_generator::populate_views(replica::table const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >, dht::token, flat_mutation_reader_v2&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >) at ./db/view/view_update_generator.cc:339
db::view::view_builder::consumer::flush_fragments() at ./db/view/view.cc:2485
db::view::view_builder::consumer::consume_end_of_partition() at ./db/view/view.cc:2499
 (inlined by) seastar::bool_class<seastar::stop_iteration_tag> compact_mutation_state<(compact_for_sstables)0>::consume_end_of_partition<db::view::view_builder::consumer, noop_compacted_fragments_consumer>(db::view::view_builder::consumer&, noop_compacted_fragments_consumer&) at ././mutation/mutation_compactor.hh:493
compact_mutation_v2<(compact_for_sstables)0, db::view::view_builder::consumer, noop_compacted_fragments_consumer>::consume_end_of_partition() at ././mutation/mutation_compactor.hh:680
 (inlined by) flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >::consume(partition_end&&)::{lambda()#1}::operator()() const at ././readers/flat_mutation_reader_v2.hh:289
 (inlined by) seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > seastar::futurize<seastar::bool_class<seastar::stop_iteration_tag> >::invoke<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >::consume(partition_end&&)::{lambda()#1}>(flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >::consume(partition_end&&)::{lambda()#1}&&) at ././seastar/include/seastar/core/future.hh:2037
auto seastar::futurize_invoke<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >::consume(partition_end&&)::{lambda()#1}>(flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >::consume(partition_end&&)::{lambda()#1}&&) at ././seastar/include/seastar/core/future.hh:2066
 (inlined by) flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >::consume(partition_end&&) at ././readers/flat_mutation_reader_v2.hh:288
 (inlined by) decltype(auto) mutation_fragment_v2::consume<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >(flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >&) && at ././mutation/mutation_fragment_v2.hh:287
flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >::operator()(mutation_fragment_v2&&) at ././readers/flat_mutation_reader_v2.hh:268
 (inlined by) seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > std::__invoke_impl<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >, flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >&, mutation_fragment_v2>(std::__invoke_other, flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >&, mutation_fragment_v2&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/bits/invoke.h:61
 (inlined by) std::__invoke_result<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >&, mutation_fragment_v2>::type std::__invoke<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >&, mutation_fragment_v2>(flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >&, mutation_fragment_v2&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/bits/invoke.h:96
 (inlined by) std::__invoke_result<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >&, mutation_fragment_v2>::type std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >::operator()<mutation_fragment_v2>(mutation_fragment_v2&&) const at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/bits/refwrap.h:359
 (inlined by) flat_mutation_reader_v2::impl::consume_pausable_in_thread<std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter>(std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter)::{lambda()#1}::operator()() at ././readers/flat_mutation_reader_v2.hh:249
 (inlined by) seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > seastar::futurize<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > >::invoke<flat_mutation_reader_v2::impl::consume_pausable_in_thread<std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter>(std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter)::{lambda()#1}>(flat_mutation_reader_v2::impl::consume_pausable_in_thread<std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter>(std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter)::{lambda()#1}&&) at ././seastar/include/seastar/core/future.hh:2035
 (inlined by) auto seastar::futurize_invoke<flat_mutation_reader_v2::impl::consume_pausable_in_thread<std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter>(std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter)::{lambda()#1}>(flat_mutation_reader_v2::impl::consume_pausable_in_thread<std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter>(std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter)::{lambda()#1}&&) at ././seastar/include/seastar/core/future.hh:2066
void flat_mutation_reader_v2::impl::consume_pausable_in_thread<std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter>(std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter) at ././readers/flat_mutation_reader_v2.hh:248
auto flat_mutation_reader_v2::impl::consume_in_thread<compact_for_query_v2<db::view::view_builder::consumer>, flat_mutation_reader_v2::no_filter>(compact_for_query_v2<db::view::view_builder::consumer>, flat_mutation_reader_v2::no_filter) at ././readers/flat_mutation_reader_v2.hh:335
auto flat_mutation_reader_v2::consume_in_thread<compact_for_query_v2<db::view::view_builder::consumer>, flat_mutation_reader_v2::no_filter>(compact_for_query_v2<db::view::view_builder::consumer>, flat_mutation_reader_v2::no_filter) at ././readers/flat_mutation_reader_v2.hh:473
auto flat_mutation_reader_v2::consume_in_thread<compact_for_query_v2<db::view::view_builder::consumer> >(compact_for_query_v2<db::view::view_builder::consumer>) at ././readers/flat_mutation_reader_v2.hh:479
 (inlined by) db::view::view_builder::execute(db::view::view_builder::build_step&, exponential_backoff_retry) at ./db/view/view.cc:2544
db::view::view_builder::do_build_step()::$_0::operator()() const at ./db/view/view.cc:2298
 (inlined by) void std::__invoke_impl<void, db::view::view_builder::do_build_step()::$_0>(std::__invoke_other, db::view::view_builder::do_build_step()::$_0&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/bits/invoke.h:61
 (inlined by) std::__invoke_result<db::view::view_builder::do_build_step()::$_0>::type std::__invoke<db::view::view_builder::do_build_step()::$_0>(db::view::view_builder::do_build_step()::$_0&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/bits/invoke.h:96
 (inlined by) decltype(auto) std::__apply_impl<db::view::view_builder::do_build_step()::$_0, std::tuple<>>(db::view::view_builder::do_build_step()::$_0&&, std::tuple<>&&, std::integer_sequence<unsigned long>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/tuple:2288
 (inlined by) decltype(auto) std::apply<db::view::view_builder::do_build_step()::$_0, std::tuple<> >(db::view::view_builder::do_build_step()::$_0&&, std::tuple<>&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/tuple:2299
 (inlined by) seastar::future<void> seastar::futurize<void>::apply<db::view::view_builder::do_build_step()::$_0>(db::view::view_builder::do_build_step()::$_0&&, std::tuple<>&&) at ././seastar/include/seastar/core/future.hh:2000
 (inlined by) seastar::async<db::view::view_builder::do_build_step()::$_0>(seastar::thread_attributes, db::view::view_builder::do_build_step()::$_0&&)::{lambda()#1}::operator()() const at ././seastar/include/seastar/core/thread.hh:260
 (inlined by) seastar::noncopyable_function<void ()>::direct_vtable_for<seastar::async<db::view::view_builder::do_build_step()::$_0>(seastar::thread_attributes, db::view::view_builder::do_build_step()::$_0&&)::{lambda()#1}>::call(seastar::noncopyable_function<void ()> const*) at ././seastar/include/seastar/util/noncopyable_function.hh:129
seastar::noncopyable_function<void ()>::operator()() const at ./build/debug/seastar/./seastar/include/seastar/util/noncopyable_function.hh:215
seastar::thread_context::main() at ./build/debug/seastar/./seastar/src/core/thread.cc:311
kbr-scylla commented 2 months ago

Looks like a duplicate of https://github.com/scylladb/scylladb/issues/9059

kbr-scylla commented 2 months ago

Marking as blocked by https://github.com/scylladb/scylladb/issues/9059 -- we'll have to retest once that is fixed.

kostja commented 2 months ago

Seen again in https://jenkins.scylladb.com/job/scylla-6.0/job/dtest-debug/24/testReport/secondary_indexes_test/TestSecondaryIndexes/Run_Dtest_Parallel_Cloud_Machines___FullDtest___full_split016___test_drop_index_while_building/