search

scylladb / scylladb

NoSQL data store using the seastar framework, compatible with Apache Cassandra
http://scylladb.com
GNU Affero General Public License v3.0
13.57k stars 1.29k forks source link

heap-buffer-overflow in sstables::mc::writer::write_clustered due to wrong compound clustering insert via thrift #7568

Closed bhalevy closed 3 years ago

bhalevy commented 4 years ago

Seen in https://jenkins.scylladb.com/view/master/job/scylla-master/job/next/2759/testReport/junit/cql_tests/MiscellaneousCQLTester/dtest___cql3_insert_thrift_test/

Traceback (most recent call last):
  File "/usr/lib64/python3.7/unittest/case.py", line 60, in testPartExecutor
    yield
  File "/usr/lib64/python3.7/unittest/case.py", line 648, in run
    self.tearDown()
  File "/jenkins/workspace/scylla-master/next/scylla-dtest/dtest.py", line 1118, in tearDown
    self.cleanUpCluster()
  File "/jenkins/workspace/scylla-master/next/scylla-dtest/dtest.py", line 1170, in cleanUpCluster
    raise AssertionError("Core file(s) found.{}".format("" if failed else " Marking test as failed."))
AssertionError: Core file(s) found. Marking test as failed.

This was apparently triggered by https://github.com/scylladb/scylla-dtest/commit/360ff58b29964f607dab512a03908a13c0fd80d4

Reproduced with scylla version edf04cd34851e6f8e009e44629dbeb5c58d25428 in debug mode

INFO  2020-11-08 14:50:34,787 [shard 0] storage_service - Drain on shutdown: hints manager is stopped
=================================================================
==893967==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x602000111708 at pc 0x0000148e40da bp 0x7f3e8a3a2f00 sp 0x7f3e8a3a2ef8
READ of size 8 at 0x602000111708 thread T0
    #0 0x148e40d9  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x148e40d9)
    #1 0x163a66aa  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x163a66aa)
    #2 0x163a572d  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x163a572d)
    #3 0x163a496b  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x163a496b)
    #4 0x163661e7  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x163661e7)
    #5 0x1634318b  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x1634318b)
    #6 0x16366b67  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x16366b67)
    #7 0x16343b39  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x16343b39)
    #8 0x16c26989  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x16c26989)
    #9 0x161eafa6  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x161eafa6)
    #10 0x161ea2f7  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x161ea2f7)
    #11 0x161e9a68  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x161e9a68)
    #12 0x161e9739  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x161e9739)
    #13 0x161e93af  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x161e93af)
    #14 0x161e90d7  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x161e90d7)
    #15 0x161e8963  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x161e8963)
    #16 0x161e7c4e  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x161e7c4e)
    #17 0x161e72c3  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x161e72c3)
    #18 0x15b56b84  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x15b56b84)
    #19 0x15b561e7  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x15b561e7)
    #20 0x15b56137  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x15b56137)
    #21 0x15b5607b  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x15b5607b)
    #22 0x15b55fc4  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x15b55fc4)
    #23 0x15b55de8  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x15b55de8)
    #24 0x15b55bae  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x15b55bae)
    #25 0x15b55954  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x15b55954)
    #26 0x140aeb70  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x140aeb70)
    #27 0x205bb656  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x205bb656)

0x602000111708 is located 8 bytes to the left of 16-byte region [0x602000111710,0x602000111720)
allocated by thread T0 here:
    #0 0x13436107  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x13436107)
    #1 0x14b61fcf  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x14b61fcf)
    #2 0x14b61f33  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x14b61f33)
    #3 0x14b6162c  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x14b6162c)
    #4 0x14b5ff96  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x14b5ff96)
    #5 0x14b5f687  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x14b5f687)
    #6 0x14ac907f  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x14ac907f)
    #7 0x14a4ade2  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x14a4ade2)
    #8 0x14a57943  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x14a57943)
    #9 0x14b9cb10  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x14b9cb10)
    #10 0x14adf59d  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x14adf59d)
    #11 0x14a7bae4  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x14a7bae4)
    #12 0x19880557  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x19880557)
    #13 0x14bb444a  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x14bb444a)
    #14 0x14bd774a  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x14bd774a)
    #15 0x14bd85c4  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x14bd85c4)
    #16 0x14bdc61f  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x14bdc61f)
    #17 0x14be1289  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x14be1289)
    #18 0x14be0577  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x14be0577)
    #19 0x19b1c8a2  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x19b1c8a2)
    #20 0x19b1afbb  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x19b1afbb)
    #21 0x19b1b6b9  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x19b1b6b9)
    #22 0x19b0deb7  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x19b0deb7)
    #23 0x1996e9fb  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x1996e9fb)
    #24 0x1993e4f9  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x1993e4f9)
    #25 0x19938c27  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x19938c27)
    #26 0x19935387  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x19935387)
    #27 0x199352d7  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x199352d7)
    #28 0x1993521b  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x1993521b)
    #29 0x19935164  (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x19935164)

SUMMARY: AddressSanitizer: heap-buffer-overflow (/local/home/bhalevy/.dtest/dtest-86jm_q12/test/node1/bin/scylla+0x148e40d9) 
Shadow bytes around the buggy address:
  0x0c048001a290: fa fa 00 00 fa fa 00 00 fa fa 00 00 fa fa fd fd
  0x0c048001a2a0: fa fa 00 00 fa fa fd fa fa fa fd fa fa fa fd fd
  0x0c048001a2b0: fa fa fd fd fa fa fd fd fa fa fd fa fa fa 00 00
  0x0c048001a2c0: fa fa fd fd fa fa fd fa fa fa fd fd fa fa 04 fa
  0x0c048001a2d0: fa fa 00 fa fa fa fd fd fa fa fd fd fa fa 00 00
=>0x0c048001a2e0: fa[fa]00 00 fa fa 00 00 fa fa 04 fa fa fa fd fd
  0x0c048001a2f0: fa fa fd fd fa fa fd fd fa fa fd fd fa fa fd fd
  0x0c048001a300: fa fa fd fd fa fa fd fa fa fa fd fd fa fa fd fd
  0x0c048001a310: fa fa fd fd fa fa fd fd fa fa fd fd fa fa fd fd
  0x0c048001a320: fa fa fd fd fa fa fd fd fa fa fd fd fa fa fd fd
  0x0c048001a330: fa fa fd fd fa fa fd fd fa fa fd fd fa fa fd fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==893967==ABORTING

Decoded backtrace:

seastar::shared_ptr<abstract_type const>::operator*() const at ././seastar/include/seastar/core/shared_ptr.hh:577
sstables::mc::clustering_blocks_input_range::next() const at ./sstables/mx/writer.cc:180
iterator at ./sstables/mx/writer.cc:106
sstables::mc::input_range_base<sstables::mc::clustering_blocks_input_range, sstables::mc::clustering_block>::begin() const at ./sstables/mx/writer.cc:130
void sstables::mc::write_clustering_prefix<sstables::file_writer>(sstables::sstable_version_types, sstables::file_writer&, schema const&, clustering_key_prefix const&, seastar::bool_class<sstables::mc::ephemerally_full_prefix_tag>) at ./sstables/mx/writer.cc:213
sstables::mc::writer::write_clustered(clustering_row const&, unsigned long) at ./sstables/mx/writer.cc:1263
void sstables::mc::writer::write_clustered<clustering_row>(clustering_row const&) at ./sstables/mx/writer.cc:717
sstables::mc::writer::consume(clustering_row&&) at ./sstables/mx/writer.cc:1284
sstables::sstable_writer::consume(clustering_row&&) at ./sstables/writer.cc:63
flat_mutation_reader::impl::consumer_adapter<sstables::sstable_writer>::consume(clustering_row&&) at ././flat_mutation_reader.hh:216
decltype(auto) mutation_fragment::consume<flat_mutation_reader::impl::consumer_adapter<sstables::sstable_writer> >(flat_mutation_reader::impl::consumer_adapter<sstables::sstable_writer>&) && at ././mutation_fragment.hh:427
flat_mutation_reader::impl::consumer_adapter<sstables::sstable_writer>::operator()(mutation_fragment&&) at ././flat_mutation_reader.hh:210
seastar::bool_class<seastar::stop_iteration_tag> std::__invoke_impl<seastar::bool_class<seastar::stop_iteration_tag>, flat_mutation_reader::impl::consumer_adapter<sstables::sstable_writer>&, mutation_fragment>(std::__invoke_other, flat_mutation_reader::impl::consumer_adapter<sstables::sstable_writer>&, mutation_fragment&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/10/../../../../include/c++/10/bits/invoke.h:60
std::__invoke_result<flat_mutation_reader::impl::consumer_adapter<sstables::sstable_writer>&, mutation_fragment>::type std::__invoke<flat_mutation_reader::impl::consumer_adapter<sstables::sstable_writer>&, mutation_fragment>(flat_mutation_reader::impl::consumer_adapter<sstables::sstable_writer>&, mutation_fragment&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/10/../../../../include/c++/10/bits/invoke.h:95
std::result_of<flat_mutation_reader::impl::consumer_adapter<sstables::sstable_writer>& (mutation_fragment&&)>::type std::reference_wrapper<flat_mutation_reader::impl::consumer_adapter<sstables::sstable_writer> >::operator()<mutation_fragment>(mutation_fragment&&) const at /usr/bin/../lib/gcc/x86_64-redhat-linux/10/../../../../include/c++/10/bits/refwrap.h:349
void flat_mutation_reader::impl::consume_pausable_in_thread<std::reference_wrapper<flat_mutation_reader::impl::consumer_adapter<sstables::sstable_writer> >, mutation_fragment_stream_validating_filter>(std::reference_wrapper<flat_mutation_reader::impl::consumer_adapter<sstables::sstable_writer> >, mutation_fragment_stream_validating_filter, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at ././flat_mutation_reader.hh:193
auto flat_mutation_reader::impl::consume_in_thread<sstables::sstable_writer, mutation_fragment_stream_validating_filter>(sstables::sstable_writer, mutation_fragment_stream_validating_filter, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at ././flat_mutation_reader.hh:273
auto flat_mutation_reader::consume_in_thread<sstables::sstable_writer, mutation_fragment_stream_validating_filter>(sstables::sstable_writer, mutation_fragment_stream_validating_filter, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at ././flat_mutation_reader.hh:383
operator() at ./sstables/sstables.cc:1703
void std::__invoke_impl<void, sstables::sstable::write_components(flat_mutation_reader, unsigned long, seastar::lw_shared_ptr<schema const>, sstables::sstable_writer_config const&, encoding_stats, seastar::io_priority_class const&)::$_64>(std::__invoke_other, sstables::sstable::write_components(flat_mutation_reader, unsigned long, seastar::lw_shared_ptr<schema const>, sstables::sstable_writer_config const&, encoding_stats, seastar::io_priority_class const&)::$_64&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/10/../../../../include/c++/10/bits/invoke.h:60
std::__invoke_result<sstables::sstable::write_components(flat_mutation_reader, unsigned long, seastar::lw_shared_ptr<schema const>, sstables::sstable_writer_config const&, encoding_stats, seastar::io_priority_class const&)::$_64>::type std::__invoke<sstables::sstable::write_components(flat_mutation_reader, unsigned long, seastar::lw_shared_ptr<schema const>, sstables::sstable_writer_config const&, encoding_stats, seastar::io_priority_class const&)::$_64>(sstables::sstable::write_components(flat_mutation_reader, unsigned long, seastar::lw_shared_ptr<schema const>, sstables::sstable_writer_config const&, encoding_stats, seastar::io_priority_class const&)::$_64&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/10/../../../../include/c++/10/bits/invoke.h:95
decltype(auto) std::__apply_impl<sstables::sstable::write_components(flat_mutation_reader, unsigned long, seastar::lw_shared_ptr<schema const>, sstables::sstable_writer_config const&, encoding_stats, seastar::io_priority_class const&)::$_64, std::tuple<>>(sstables::sstable::write_components(flat_mutation_reader, unsigned long, seastar::lw_shared_ptr<schema const>, sstables::sstable_writer_config const&, encoding_stats, seastar::io_priority_class const&)::$_64&&, std::tuple<>&&, std::integer_sequence<unsigned long>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/10/../../../../include/c++/10/tuple:1723
decltype(auto) std::apply<sstables::sstable::write_components(flat_mutation_reader, unsigned long, seastar::lw_shared_ptr<schema const>, sstables::sstable_writer_config const&, encoding_stats, seastar::io_priority_class const&)::$_64, std::tuple<> >(sstables::sstable::write_components(flat_mutation_reader, unsigned long, seastar::lw_shared_ptr<schema const>, sstables::sstable_writer_config const&, encoding_stats, seastar::io_priority_class const&)::$_64&&, std::tuple<>&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/10/../../../../include/c++/10/tuple:1734
seastar::future<void> seastar::futurize<void>::apply<sstables::sstable::write_components(flat_mutation_reader, unsigned long, seastar::lw_shared_ptr<schema const>, sstables::sstable_writer_config const&, encoding_stats, seastar::io_priority_class const&)::$_64>(sstables::sstable::write_components(flat_mutation_reader, unsigned long, seastar::lw_shared_ptr<schema const>, sstables::sstable_writer_config const&, encoding_stats, seastar::io_priority_class const&)::$_64&&, std::tuple<>&&) at ././seastar/include/seastar/core/future.hh:2099
operator() at ././seastar/include/seastar/core/thread.hh:258
seastar::noncopyable_function<void ()>::direct_vtable_for<seastar::async<sstables::sstable::write_components(flat_mutation_reader, unsigned long, seastar::lw_shared_ptr<schema const>, sstables::sstable_writer_config const&, encoding_stats, seastar::io_priority_class const&)::$_64>(seastar::thread_attributes, sstables::sstable::write_components(flat_mutation_reader, unsigned long, seastar::lw_shared_ptr<schema const>, sstables::sstable_writer_config const&, encoding_stats, seastar::io_priority_class const&)::$_64&&)::{lambda()#1}>::call(seastar::noncopyable_function<void ()> const*) at ././seastar/include/seastar/util/noncopyable_function.hh:116
seastar::noncopyable_function<void ()>::operator()() const at ././seastar/include/seastar/util/noncopyable_function.hh:201
seastar::thread_context::main() at ./build/debug/seastar/./seastar/src/core/thread.cc:297
bhalevy commented 4 years ago

Decoded allocated-by backtrace:

operator new(unsigned long) at ??:?
__gnu_cxx::new_allocator<seastar::shared_ptr<abstract_type const> >::allocate(unsigned long, void const*) at /usr/bin/../lib/gcc/x86_64-redhat-linux/10/../../../../include/c++/10/ext/new_allocator.h:115
std::allocator<seastar::shared_ptr<abstract_type const> >::allocate(unsigned long) at /usr/bin/../lib/gcc/x86_64-redhat-linux/10/../../../../include/c++/10/bits/allocator.h:173
 (inlined by) std::allocator_traits<std::allocator<seastar::shared_ptr<abstract_type const> > >::allocate(std::allocator<seastar::shared_ptr<abstract_type const> >&, unsigned long) at /usr/bin/../lib/gcc/x86_64-redhat-linux/10/../../../../include/c++/10/bits/alloc_traits.h:460
std::_Vector_base<seastar::shared_ptr<abstract_type const>, std::allocator<seastar::shared_ptr<abstract_type const> > >::_M_allocate(unsigned long) at /usr/bin/../lib/gcc/x86_64-redhat-linux/10/../../../../include/c++/10/bits/stl_vector.h:346
void std::vector<seastar::shared_ptr<abstract_type const>, std::allocator<seastar::shared_ptr<abstract_type const> > >::_M_realloc_insert<seastar::shared_ptr<abstract_type const> const&>(__gnu_cxx::__normal_iterator<seastar::shared_ptr<abstract_type const>*, std::vector<seastar::shared_ptr<abstract_type const>, std::allocator<seastar::shared_ptr<abstract_type const> > > >, seastar::shared_ptr<abstract_type const> const&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/10/../../../../include/c++/10/bits/vector.tcc:440
std::vector<seastar::shared_ptr<abstract_type const>, std::allocator<seastar::shared_ptr<abstract_type const> > >::push_back(seastar::shared_ptr<abstract_type const> const&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/10/../../../../include/c++/10/bits/stl_vector.h:1198
std::vector<seastar::shared_ptr<abstract_type const>, std::allocator<seastar::shared_ptr<abstract_type const> > > get_column_types<boost::iterator_range<__gnu_cxx::__normal_iterator<column_definition const*, std::vector<column_definition, std::allocator<column_definition> > > > >(boost::iterator_range<__gnu_cxx::__normal_iterator<column_definition const*, std::vector<column_definition, std::allocator<column_definition> > > > const&) at ./schema.cc:105
schema::rebuild() at ./schema.cc:270
schema at ./schema.cc:456
seastar::lw_shared_ptr<schema> seastar::lw_shared_ptr<schema>::make<schema>(schema&&) at ././seastar/include/seastar/core/shared_ptr.hh:267
seastar::lw_shared_ptr<schema> seastar::make_lw_shared<schema>(schema&&) at ././seastar/include/seastar/core/shared_ptr.hh:423
schema_builder::build() at ./schema.cc:1221
db::schema_tables::create_table_from_mutations(db::schema_ctxt const&, schema_mutations, std::optional<utils::UUID>) at ./db/schema_tables.cc:2506
frozen_schema::unfreeze(db::schema_ctxt const&) const at ./frozen_schema.cc:54
schema_registry_entry::get_schema() at ./schema_registry.cc:193
schema_registry_entry::load(frozen_schema) at ./schema_registry.cc:152
schema_registry::get_or_load(utils::UUID, std::function<frozen_schema (utils::UUID)> const&) at ./schema_registry.cc:139
operator() at ./schema_registry.cc:307
global_schema_ptr at ./schema_registry.cc:300
_ZSt12construct_atI17global_schema_ptrJRN7seastar13lw_shared_ptrIK6schemaEEEEDTgsnwcvPvLi0E_T_pispclsr3stdE7declvalIT0_EEEEPS8_DpOS9_ at /usr/bin/../lib/gcc/x86_64-redhat-linux/10/../../../../include/c++/10/bits/stl_construct.h:97
void std::allocator_traits<std::allocator<global_schema_ptr> >::construct<global_schema_ptr, seastar::lw_shared_ptr<schema const>&>(std::allocator<global_schema_ptr>&, global_schema_ptr*, seastar::lw_shared_ptr<schema const>&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/10/../../../../include/c++/10/bits/alloc_traits.h:514
void std::vector<global_schema_ptr, std::allocator<global_schema_ptr> >::_M_realloc_insert<seastar::lw_shared_ptr<schema const>&>(__gnu_cxx::__normal_iterator<global_schema_ptr*, std::vector<global_schema_ptr, std::allocator<global_schema_ptr> > >, seastar::lw_shared_ptr<schema const>&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/10/../../../../include/c++/10/bits/vector.tcc:449
global_schema_ptr& std::vector<global_schema_ptr, std::allocator<global_schema_ptr> >::emplace_back<seastar::lw_shared_ptr<schema const>&>(seastar::lw_shared_ptr<schema const>&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/10/../../../../include/c++/10/bits/vector.tcc:121
db::schema_tables::diff_table_or_view(seastar::sharded<service::storage_proxy>&, std::map<utils::UUID, schema_mutations, std::less<utils::UUID>, std::allocator<std::pair<utils::UUID const, schema_mutations> > >&&, std::map<utils::UUID, schema_mutations, std::less<utils::UUID>, std::allocator<std::pair<utils::UUID const, schema_mutations> > >&&, seastar::noncopyable_function<seastar::lw_shared_ptr<schema const> (schema_mutations)>) at ./db/schema_tables.cc:1179
db::schema_tables::merge_tables_and_views(seastar::sharded<service::storage_proxy>&, std::map<utils::UUID, schema_mutations, std::less<utils::UUID>, std::allocator<std::pair<utils::UUID const, schema_mutations> > >&&, std::map<utils::UUID, schema_mutations, std::less<utils::UUID>, std::allocator<std::pair<utils::UUID const, schema_mutations> > >&&, std::map<utils::UUID, schema_mutations, std::less<utils::UUID>, std::allocator<std::pair<utils::UUID const, schema_mutations> > >&&, std::map<utils::UUID, schema_mutations, std::less<utils::UUID>, std::allocator<std::pair<utils::UUID const, schema_mutations> > >&&) at ./db/schema_tables.cc:1201
operator() at ./db/schema_tables.cc:1072
void std::__invoke_impl<void, db::schema_tables::do_merge_schema(seastar::sharded<service::storage_proxy>&, std::vector<mutation, std::allocator<mutation> >, bool)::$_17>(std::__invoke_other, db::schema_tables::do_merge_schema(seastar::sharded<service::storage_proxy>&, std::vector<mutation, std::allocator<mutation> >, bool)::$_17&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/10/../../../../include/c++/10/bits/invoke.h:60
std::__invoke_result<db::schema_tables::do_merge_schema(seastar::sharded<service::storage_proxy>&, std::vector<mutation, std::allocator<mutation> >, bool)::$_17>::type std::__invoke<db::schema_tables::do_merge_schema(seastar::sharded<service::storage_proxy>&, std::vector<mutation, std::allocator<mutation> >, bool)::$_17>(db::schema_tables::do_merge_schema(seastar::sharded<service::storage_proxy>&, std::vector<mutation, std::allocator<mutation> >, bool)::$_17&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/10/../../../../include/c++/10/bits/invoke.h:95
decltype(auto) std::__apply_impl<db::schema_tables::do_merge_schema(seastar::sharded<service::storage_proxy>&, std::vector<mutation, std::allocator<mutation> >, bool)::$_17, std::tuple<>>(db::schema_tables::do_merge_schema(seastar::sharded<service::storage_proxy>&, std::vector<mutation, std::allocator<mutation> >, bool)::$_17&&, std::tuple<>&&, std::integer_sequence<unsigned long>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/10/../../../../include/c++/10/tuple:1723
decltype(auto) std::apply<db::schema_tables::do_merge_schema(seastar::sharded<service::storage_proxy>&, std::vector<mutation, std::allocator<mutation> >, bool)::$_17, std::tuple<> >(db::schema_tables::do_merge_schema(seastar::sharded<service::storage_proxy>&, std::vector<mutation, std::allocator<mutation> >, bool)::$_17&&, std::tuple<>&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/10/../../../../include/c++/10/tuple:1734
avikivity commented 4 years ago

Very strange. It looks like _offset is -1, but it is unsigned, initialized to 0, and never decremented.

bhalevy commented 4 years ago

_offset is actually 1.

With this:

diff --git a/sstables/mx/writer.cc b/sstables/mx/writer.cc
index f57ba13737..52a4b00d7e 100644
--- a/sstables/mx/writer.cc
+++ b/sstables/mx/writer.cc
@@ -177,7 +177,16 @@ class clustering_blocks_input_range
                 if (value.empty()) {
                     _current_block.header |= (uint64_t(1) << (shift * 2));
                 } else {
-                    _current_block.values.push_back({value, *_prefix.get_compound_type(_schema)->types()[_offset]});
+//                    _current_block.values.push_back({value, *_prefix.get_compound_type(_schema)->types()[_offset]});
+                    const auto& compound_type = _prefix.get_compound_type(_schema);
+                    const auto& types = compound_type->types();
+                    if (_offset < types.size()) {
+                        const auto& type = *types[_offset];
+                        _current_block.values.push_back({value, type});
+                    } else {
+                        slogger.warn("prefix {} (size={}): offset {} >= types.size {}", _prefix, _prefix.size(_schema), _offset, types.size());
+                        _current_block.header |= (uint64_t(1) << ((shift * 2) + 1));
+                    }
                 }
             } else {
                 // This (and all subsequent) values of the prefix are missing (null)

I see

WARN  2020-11-08 17:31:42,812 [shard 0] mc_writer - prefix ckp{000400000004000176} (size=2): offset 1 >= types.size 1
bhalevy commented 4 years ago

We happend to hit this on shut down, but I hit the same issue running nodetool flush after inserting in the thrift test. The test itself does this

280         session = self.prepare(start_rpc=True)
281 
282         session.execute("""
283             CREATE TABLE test (
284                 k int,
285                 c int,
286                 v int,
287                 PRIMARY KEY (k, c)
288             )
289         """)
290 
291         node = self.cluster.nodelist()[0]
292         host, port = node.network_interfaces['thrift']
293         client = get_thrift_client(host, port)
294         client.transport.open()
295         client.set_keyspace('ks')
296         key = struct.pack('>i', 2)
297         column_name_component = struct.pack('>i', 4)
298         # component length + component + EOC + component length + component + EOC
299         column_name = b'\x00\x04' + column_name_component + b'\x00' + b'\x00\x01' + 'v'.encode() + b'\x00'
300         value = struct.pack('>i', 8)
301         client.batch_mutate(
302             {key: {'test': [Mutation(ColumnOrSuperColumn(
303                 column=Column(name=column_name, value=value, timestamp=100)))]}},
304             ThriftConsistencyLevel.ONE)
305 
306         res = session.execute("SELECT * FROM test")
307         assert rows_to_list(res) == [[2, 4, 8]], res

hit also if the row below is added
308         node.flush()
bhalevy commented 4 years ago

Cc @tgrabiec

bhalevy commented 4 years ago 
(gdb) fr 7
#7  0x00000000015ca4e3 in sstables::mc::writer::consume (this=0x6010001a5c00, cr=...) at sstables/mx/writer.cc:1284
1284        write_clustered(cr);

(gdb) p cr._ck
$17 = {
  <prefix_compound_wrapper<clustering_key_prefix, clustering_key_prefix_view, clustering_key_prefix>> = {
    <compound_wrapper<clustering_key_prefix, clustering_key_prefix_view>> = {
      _bytes = 000400000004000176
    }, <No data fields>}, <No data fields>}

(gdb) p _schema
$18 = (const schema &) @0x60100044f180: {
  <seastar::enable_lw_shared_from_this<schema>> = {
    <seastar::lw_shared_ptr_counter_base> = {
      _count = 14
    }, <No data fields>}, 
  members of schema:
  _raw = {
    _id = e8f8a010-21b4-11eb-86fb-000000000001,
    _ks_name = "ks",
    _cf_name = "test",
    _columns = {
      <std::_Vector_base<column_definition, std::allocator<column_definition> >> = {
        _M_impl = {
          <std::allocator<column_definition>> = {
            <__gnu_cxx::new_allocator<column_definition>> = {<No data fields>}, <No data fields>}, 
          <std::_Vector_base<column_definition, std::allocator<column_definition> >::_Vector_impl_data> = {
            _M_start = 0x60100057e280,
            _M_finish = 0x60100057e3b8,
            _M_end_of_storage = 0x60100057e3b8
          }, <No data fields>}
      }, <No data fields>},
    _comment = "",
    _default_time_to_live = {
      __r = 0
    },
    _regular_column_name_type = {
      _b = 0x6010000f51c0,
      _p = 0x6010000f51c0
    },
    _default_validation_class = {
      _b = 0x6010000f5230,
      _p = 0x6010000f5230
    },
    _bloom_filter_fp_chance = 0.01,
    _compressor_params = {
      _compressor = {
        _b = 0x6010000f77e0,
        _p = 0x6010000f77f0
      },
      _chunk_length = {
        <std::_Optional_base<int, true, true>> = {
          <std::_Optional_base_impl<int, std::_Optional_base<int, true, true> >> = {<No data fields>}, 
          members of std::_Optional_base<int, true, true>:
          _M_payload = {
            <std::_Optional_payload_base<int>> = {
              _M_payload = {
                _M_empty = {<No data fields>},
                _M_value = 1702102016
              },
              _M_engaged = false
            }, <No data fields>}
        }, 
        <std::_Enable_copy_move<true, true, true, true, std::optional<int> >> = {<No data fields>}, <No data fields>},
      _crc_check_chance = {
        <std::_Optional_base<double, true, true>> = {
          <std::_Optional_base_impl<double, std::_Optional_base<double, true, true> >> = {<No data fields>}, 
          members of std::_Optional_base<double, true, true>:
          _M_payload = {
            <std::_Optional_payload_base<double>> = {
              _M_payload = {
                _M_empty = {<No data fields>},
                _M_value = 2.5392275005826067e-265
              },
              _M_engaged = false
            }, <No data fields>}
        }, 
        <std::_Enable_copy_move<true, true, true, true, std::optional<double> >> = {<No data fields>}, <No data fields>}
    },
    _extensions = {
      _M_t = {
        _M_impl = {
          <std::allocator<std::_Rb_tree_node<std::pair<seastar::basic_sstring<char, unsigned int, 15, true> const, seastar::shared_ptr<schema_extension> > > >> = {
--Type <RET> for more, q to quit, c to continue without paging--c
            <__gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<seastar::basic_sstring<char, unsigned int, 15, true> const, seastar::shared_ptr<schema_extension> > > >> = {<No data fields>}, <No data fields>}, 
          <std::_Rb_tree_key_compare<std::less<seastar::basic_sstring<char, unsigned int, 15, true> > >> = {
            _M_key_compare = {
              <std::binary_function<seastar::basic_sstring<char, unsigned int, 15, true>, seastar::basic_sstring<char, unsigned int, 15, true>, bool>> = {<No data fields>}, <No data fields>}
          }, 
          <std::_Rb_tree_header> = {
            _M_header = {
              _M_color = std::_S_red,
              _M_parent = 0x0,
              _M_left = 0x60100044f240,
              _M_right = 0x60100044f240
            },
            _M_node_count = 0
          }, <No data fields>}
      }
    },
    _is_dense = false,
    _is_compound = true,
    _is_counter = false,
    _type = cf_type::standard,
    _gc_grace_seconds = 864000,
    _paxos_grace_seconds = {
      <std::_Optional_base<int, true, true>> = {
        <std::_Optional_base_impl<int, std::_Optional_base<int, true, true> >> = {<No data fields>}, 
        members of std::_Optional_base<int, true, true>:
        _M_payload = {
          <std::_Optional_payload_base<int>> = {
            _M_payload = {
              _M_empty = {<No data fields>},
              _M_value = 9
            },
            _M_engaged = false
          }, <No data fields>}
      }, 
      <std::_Enable_copy_move<true, true, true, true, std::optional<int> >> = {<No data fields>}, <No data fields>},
    _dc_local_read_repair_chance = 0,
    _read_repair_chance = 0,
    _crc_check_chance = 1,
    _min_compaction_threshold = 4,
    _max_compaction_threshold = 32,
    _min_index_interval = 128,
    _max_index_interval = 2048,
    _memtable_flush_period = 0,
    _speculative_retry = {
      _t = speculative_retry::type::PERCENTILE,
      _v = 0.98999999999999999
    },
    _compaction_strategy = sstables::compaction_strategy_type::size_tiered,
    _compaction_strategy_options = {
      _M_t = {
        _M_impl = {
          <std::allocator<std::_Rb_tree_node<std::pair<seastar::basic_sstring<char, unsigned int, 15, true> const, seastar::basic_sstring<char, unsigned int, 15, true> > > >> = {
            <__gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<seastar::basic_sstring<char, unsigned int, 15, true> const, seastar::basic_sstring<char, unsigned int, 15, true> > > >> = {<No data fields>}, <No data fields>}, 
          <std::_Rb_tree_key_compare<std::less<seastar::basic_sstring<char, unsigned int, 15, true> > >> = {
            _M_key_compare = {
              <std::binary_function<seastar::basic_sstring<char, unsigned int, 15, true>, seastar::basic_sstring<char, unsigned int, 15, true>, bool>> = {<No data fields>}, <No data fields>}
          }, 
          <std::_Rb_tree_header> = {
            _M_header = {
              _M_color = std::_S_red,
              _M_parent = 0x0,
              _M_left = 0x60100044f2c8,
              _M_right = 0x60100044f2c8
            },
            _M_node_count = 0
          }, <No data fields>}
      }
    },
    _compaction_enabled = true,
    _caching_options = {
      _key_cache = "ALL",
      _row_cache = "ALL",
      _enabled = true
    },
    _version = 3d703269-6011-3268-a8ac-710d0a6ccbea,
    _dropped_columns = {
      _M_h = {
        <std::__detail::_Hashtable_base<seastar::basic_sstring<char, unsigned int, 15, true>, std::pair<seastar::basic_sstring<char, unsigned int, 15, true> const, schema::dropped_column>, std::__detail::_Select1st, std::equal_to<seastar::basic_sstring<char, unsigned int, 15, true> >, std::hash<seastar::basic_sstring<char, unsigned int, 15, true> >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Hashtable_traits<true, false, true> >> = {
          <std::__detail::_Hash_code_base<seastar::basic_sstring<char, unsigned int, 15, true>, std::pair<seastar::basic_sstring<char, unsigned int, 15, true> const, schema::dropped_column>, std::__detail::_Select1st, std::hash<seastar::basic_sstring<char, unsigned int, 15, true> >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, true>> = {
            <std::__detail::_Hashtable_ebo_helper<0, std::__detail::_Select1st, true>> = {
              <std::__detail::_Select1st> = {<No data fields>}, <No data fields>}, 
            <std::__detail::_Hashtable_ebo_helper<1, std::hash<seastar::basic_sstring<char, unsigned int, 15, true> >, true>> = {
              <std::hash<seastar::basic_sstring<char, unsigned int, 15, true> >> = {<No data fields>}, <No data fields>}, 
            <std::__detail::_Hashtable_ebo_helper<2, std::__detail::_Mod_range_hashing, true>> = {
              <std::__detail::_Mod_range_hashing> = {<No data fields>}, <No data fields>}, <No data fields>}, 
          <std::__detail::_Hashtable_ebo_helper<0, std::equal_to<seastar::basic_sstring<char, unsigned int, 15, true> >, true>> = {
            <std::equal_to<seastar::basic_sstring<char, unsigned int, 15, true> >> = {
              <std::binary_function<seastar::basic_sstring<char, unsigned int, 15, true>, seastar::basic_sstring<char, unsigned int, 15, true>, bool>> = {<No data fields>}, <No data fields>}, <No data fields>}, <No data fields>}, 
        <std::__detail::_Map_base<seastar::basic_sstring<char, unsigned int, 15, true>, std::pair<seastar::basic_sstring<char, unsigned int, 15, true> const, schema::dropped_column>, std::allocator<std::pair<seastar::basic_sstring<char, unsigned int, 15, true> const, schema::dropped_column> >, std::__detail::_Select1st, std::equal_to<seastar::basic_sstring<char, unsigned int, 15, true> >, std::hash<seastar::basic_sstring<char, unsigned int, 15, true> >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true>, true>> = {<No data fields>}, 
        <std::__detail::_Insert<seastar::basic_sstring<char, unsigned int, 15, true>, std::pair<seastar::basic_sstring<char, unsigned int, 15, true> const, schema::dropped_column>, std::allocator<std::pair<seastar::basic_sstring<char, unsigned int, 15, true> const, schema::dropped_column> >, std::__detail::_Select1st, std::equal_to<seastar::basic_sstring<char, unsigned int, 15, true> >, std::hash<seastar::basic_sstring<char, unsigned int, 15, true> >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true>, false>> = {
          <std::__detail::_Insert_base<seastar::basic_sstring<char, unsigned int, 15, true>, std::pair<seastar::basic_sstring<char, unsigned int, 15, true> const, schema::dropped_column>, std::allocator<std::pair<seastar::basic_sstring<char, unsigned int, 15, true> const, schema::dropped_column> >, std::__detail::_Select1st, std::equal_to<seastar::basic_sstring<char, unsigned int, 15, true> >, std::hash<seastar::basic_sstring<char, unsigned int, 15, true> >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >> = {<No data fields>}, <No data fields>}, 
        <std::__detail::_Rehash_base<seastar::basic_sstring<char, unsigned int, 15, true>, std::pair<seastar::basic_sstring<char, unsigned int, 15, true> const, schema::dropped_column>, std::allocator<std::pair<seastar::basic_sstring<char, unsigned int, 15, true> const, schema::dropped_column> >, std::__detail::_Select1st, std::equal_to<seastar::basic_sstring<char, unsigned int, 15, true> >, std::hash<seastar::basic_sstring<char, unsigned int, 15, true> >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true>, std::integral_constant<bool, true> >> = {<No data fields>}, 
        <std::__detail::_Equality<seastar::basic_sstring<char, unsigned int, 15, true>, std::pair<seastar::basic_sstring<char, unsigned int, 15, true> const, schema::dropped_column>, std::allocator<std::pair<seastar::basic_sstring<char, unsigned int, 15, true> const, schema::dropped_column> >, std::__detail::_Select1st, std::equal_to<seastar::basic_sstring<char, unsigned int, 15, true> >, std::hash<seastar::basic_sstring<char, unsigned int, 15, true> >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true>, true>> = {<No data fields>}, 
        <std::__detail::_Hashtable_alloc<std::allocator<std::__detail::_Hash_node<std::pair<seastar::basic_sstring<char, unsigned int, 15, true> const, schema::dropped_column>, true> > >> = {
          <std::__detail::_Hashtable_ebo_helper<0, std::allocator<std::__detail::_Hash_node<std::pair<seastar::basic_sstring<char, unsigned int, 15, true> const, schema::dropped_column>, true> >, true>> = {
            <std::allocator<std::__detail::_Hash_node<std::pair<seastar::basic_sstring<char, unsigned int, 15, true> const, schema::dropped_column>, true> >> = {
              <__gnu_cxx::new_allocator<std::__detail::_Hash_node<std::pair<seastar::basic_sstring<char, unsigned int, 15, true> const, schema::dropped_column>, true> >> = {<No data fields>}, <No data fields>}, <No data fields>}, <No data fields>}, 
        members of std::_Hashtable<seastar::basic_sstring<char, unsigned int, 15, true>, std::pair<seastar::basic_sstring<char, unsigned int, 15, true> const, schema::dropped_column>, std::allocator<std::pair<seastar::basic_sstring<char, unsigned int, 15, true> const, schema::dropped_column> >, std::__detail::_Select1st, std::equal_to<seastar::basic_sstring<char, unsigned int, 15, true> >, std::hash<seastar::basic_sstring<char, unsigned int, 15, true> >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >:
        _M_buckets = 0x60100044f360,
        _M_bucket_count = 1,
        _M_before_begin = {
          _M_nxt = 0x0
        },
        _M_element_count = 0,
        _M_rehash_policy = {
          _M_max_load_factor = 1,
          _M_next_resize = 0
        },
        _M_single_bucket = 0x0
      }
    },
    _collections = {
      _M_t = {
        _M_impl = {
          <std::allocator<std::_Rb_tree_node<std::pair<seastar::basic_sstring<signed char, unsigned int, 31, false> const, seastar::shared_ptr<abstract_type const> > > >> = {
            <__gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<seastar::basic_sstring<signed char, unsigned int, 31, false> const, seastar::shared_ptr<abstract_type const> > > >> = {<No data fields>}, <No data fields>}, 
          <std::_Rb_tree_key_compare<std::less<seastar::basic_sstring<signed char, unsigned int, 31, false> > >> = {
            _M_key_compare = {
              <std::binary_function<seastar::basic_sstring<signed char, unsigned int, 31, false>, seastar::basic_sstring<signed char, unsigned int, 31, false>, bool>> = {<No data fields>}, <No data fields>}
          }, 
          <std::_Rb_tree_header> = {
            _M_header = {
              _M_color = std::_S_red,
              _M_parent = 0x0,
              _M_left = 0x60100044f370,
              _M_right = 0x60100044f370
            },
            _M_node_count = 0
          }, <No data fields>}
      }
    },
    _indices_by_name = {
      _M_h = {
        <std::__detail::_Hashtable_base<seastar::basic_sstring<char, unsigned int, 15, true>, std::pair<seastar::basic_sstring<char, unsigned int, 15, true> const, index_metadata>, std::__detail::_Select1st, std::equal_to<seastar::basic_sstring<char, unsigned int, 15, true> >, std::hash<seastar::basic_sstring<char, unsigned int, 15, true> >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Hashtable_traits<true, false, true> >> = {
          <std::__detail::_Hash_code_base<seastar::basic_sstring<char, unsigned int, 15, true>, std::pair<seastar::basic_sstring<char, unsigned int, 15, true> const, index_metadata>, std::__detail::_Select1st, std::hash<seastar::basic_sstring<char, unsigned int, 15, true> >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, true>> = {
            <std::__detail::_Hashtable_ebo_helper<0, std::__detail::_Select1st, true>> = {
              <std::__detail::_Select1st> = {<No data fields>}, <No data fields>}, 
            <std::__detail::_Hashtable_ebo_helper<1, std::hash<seastar::basic_sstring<char, unsigned int, 15, true> >, true>> = {
              <std::hash<seastar::basic_sstring<char, unsigned int, 15, true> >> = {<No data fields>}, <No data fields>}, 
            <std::__detail::_Hashtable_ebo_helper<2, std::__detail::_Mod_range_hashing, true>> = {
              <std::__detail::_Mod_range_hashing> = {<No data fields>}, <No data fields>}, <No data fields>}, 
          <std::__detail::_Hashtable_ebo_helper<0, std::equal_to<seastar::basic_sstring<char, unsigned int, 15, true> >, true>> = {
            <std::equal_to<seastar::basic_sstring<char, unsigned int, 15, true> >> = {
              <std::binary_function<seastar::basic_sstring<char, unsigned int, 15, true>, seastar::basic_sstring<char, unsigned int, 15, true>, bool>> = {<No data fields>}, <No data fields>}, <No data fields>}, <No data fields>}, 
        <std::__detail::_Map_base<seastar::basic_sstring<char, unsigned int, 15, true>, std::pair<seastar::basic_sstring<char, unsigned int, 15, true> const, index_metadata>, std::allocator<std::pair<seastar::basic_sstring<char, unsigned int, 15, true> const, index_metadata> >, std::__detail::_Select1st, std::equal_to<seastar::basic_sstring<char, unsigned int, 15, true> >, std::hash<seastar::basic_sstring<char, unsigned int, 15, true> >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true>, true>> = {<No data fields>}, 
        <std::__detail::_Insert<seastar::basic_sstring<char, unsigned int, 15, true>, std::pair<seastar::basic_sstring<char, unsigned int, 15, true> const, index_metadata>, std::allocator<std::pair<seastar::basic_sstring<char, unsigned int, 15, true> const, index_metadata> >, std::__detail::_Select1st, std::equal_to<seastar::basic_sstring<char, unsigned int, 15, true> >, std::hash<seastar::basic_sstring<char, unsigned int, 15, true> >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true>, false>> = {
          <std::__detail::_Insert_base<seastar::basic_sstring<char, unsigned int, 15, true>, std::pair<seastar::basic_sstring<char, unsigned int, 15, true> const, index_metadata>, std::allocator<std::pair<seastar::basic_sstring<char, unsigned int, 15, true> const, index_metadata> >, std::__detail::_Select1st, std::equal_to<seastar::basic_sstring<char, unsigned int, 15, true> >, std::hash<seastar::basic_sstring<char, unsigned int, 15, true> >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >> = {<No data fields>}, <No data fields>}, 
        <std::__detail::_Rehash_base<seastar::basic_sstring<char, unsigned int, 15, true>, std::pair<seastar::basic_sstring<char, unsigned int, 15, true> const, index_metadata>, std::allocator<std::pair<seastar::basic_sstring<char, unsigned int, 15, true> const, index_metadata> >, std::__detail::_Select1st, std::equal_to<seastar::basic_sstring<char, unsigned int, 15, true> >, std::hash<seastar::basic_sstring<char, unsigned int, 15, true> >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true>, std::integral_constant<bool, true> >> = {<No data fields>}, 
        <std::__detail::_Equality<seastar::basic_sstring<char, unsigned int, 15, true>, std::pair<seastar::basic_sstring<char, unsigned int, 15, true> const, index_metadata>, std::allocator<std::pair<seastar::basic_sstring<char, unsigned int, 15, true> const, index_metadata> >, std::__detail::_Select1st, std::equal_to<seastar::basic_sstring<char, unsigned int, 15, true> >, std::hash<seastar::basic_sstring<char, unsigned int, 15, true> >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true>, true>> = {<No data fields>}, 
        <std::__detail::_Hashtable_alloc<std::allocator<std::__detail::_Hash_node<std::pair<seastar::basic_sstring<char, unsigned int, 15, true> const, index_metadata>, true> > >> = {
          <std::__detail::_Hashtable_ebo_helper<0, std::allocator<std::__detail::_Hash_node<std::pair<seastar::basic_sstring<char, unsigned int, 15, true> const, index_metadata>, true> >, true>> = {
            <std::allocator<std::__detail::_Hash_node<std::pair<seastar::basic_sstring<char, unsigned int, 15, true> const, index_metadata>, true> >> = {
              <__gnu_cxx::new_allocator<std::__detail::_Hash_node<std::pair<seastar::basic_sstring<char, unsigned int, 15, true> const, index_metadata>, true> >> = {<No data fields>}, <No data fields>}, <No data fields>}, <No data fields>}, 
        members of std::_Hashtable<seastar::basic_sstring<char, unsigned int, 15, true>, std::pair<seastar::basic_sstring<char, unsigned int, 15, true> const, index_metadata>, std::allocator<std::pair<seastar::basic_sstring<char, unsigned int, 15, true> const, index_metadata> >, std::__detail::_Select1st, std::equal_to<seastar::basic_sstring<char, unsigned int, 15, true> >, std::hash<seastar::basic_sstring<char, unsigned int, 15, true> >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >:
        _M_buckets = 0x60100044f3c8,
        _M_bucket_count = 1,
        _M_before_begin = {
          _M_nxt = 0x0
        },
        _M_element_count = 0,
        _M_rehash_policy = {
          _M_max_load_factor = 1,
          _M_next_resize = 0
        },
        _M_single_bucket = 0x0
      }
    },
    _wait_for_sync = false,
    _partitioner = {
      _M_data = 0x6010000a8e20
    },
    _sharder = {
      _M_data = 0x6010000f7630
    }
  },
  _thrift = {
    _compound = true,
    _is_dynamic = true
  },
  _v3_columns = {
    _is_dense = false,
    _is_compound = true,
    _columns = {
      <std::_Vector_base<column_definition, std::allocator<column_definition> >> = {
        _M_impl = {
          <std::allocator<column_definition>> = {
            <__gnu_cxx::new_allocator<column_definition>> = {<No data fields>}, <No data fields>}, 
          <std::_Vector_base<column_definition, std::allocator<column_definition> >::_Vector_impl_data> = {
            _M_start = 0x6010031a6280,
            _M_finish = 0x6010031a63b8,
            _M_end_of_storage = 0x6010031a63b8
          }, <No data fields>}
      }, <No data fields>},
    _columns_by_name = {
      _M_h = {
        <std::__detail::_Hashtable_base<seastar::basic_sstring<signed char, unsigned int, 31, false>, std::pair<seastar::basic_sstring<signed char, unsigned int, 31, false> const, column_definition const*>, std::__detail::_Select1st, std::equal_to<seastar::basic_sstring<signed char, unsigned int, 31, false> >, std::hash<seastar::basic_sstring<signed char, unsigned int, 31, false> >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Hashtable_traits<true, false, true> >> = {
          <std::__detail::_Hash_code_base<seastar::basic_sstring<signed char, unsigned int, 31, false>, std::pair<seastar::basic_sstring<signed char, unsigned int, 31, false> const, column_definition const*>, std::__detail::_Select1st, std::hash<seastar::basic_sstring<signed char, unsigned int, 31, false> >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, true>> = {
            <std::__detail::_Hashtable_ebo_helper<0, std::__detail::_Select1st, true>> = {
              <std::__detail::_Select1st> = {<No data fields>}, <No data fields>}, 
            <std::__detail::_Hashtable_ebo_helper<1, std::hash<seastar::basic_sstring<signed char, unsigned int, 31, false> >, true>> = {
              <std::hash<seastar::basic_sstring<signed char, unsigned int, 31, false> >> = {<No data fields>}, <No data fields>}, 
            <std::__detail::_Hashtable_ebo_helper<2, std::__detail::_Mod_range_hashing, true>> = {
              <std::__detail::_Mod_range_hashing> = {<No data fields>}, <No data fields>}, <No data fields>}, 
          <std::__detail::_Hashtable_ebo_helper<0, std::equal_to<seastar::basic_sstring<signed char, unsigned int, 31, false> >, true>> = {
            <std::equal_to<seastar::basic_sstring<signed char, unsigned int, 31, false> >> = {
              <std::binary_function<seastar::basic_sstring<signed char, unsigned int, 31, false>, seastar::basic_sstring<signed char, unsigned int, 31, false>, bool>> = {<No data fields>}, <No data fields>}, <No data fields>}, <No data fields>}, 
        <std::__detail::_Map_base<seastar::basic_sstring<signed char, unsigned int, 31, false>, std::pair<seastar::basic_sstring<signed char, unsigned int, 31, false> const, column_definition const*>, std::allocator<std::pair<seastar::basic_sstring<signed char, unsigned int, 31, false> const, column_definition const*> >, std::__detail::_Select1st, std::equal_to<seastar::basic_sstring<signed char, unsigned int, 31, false> >, std::hash<seastar::basic_sstring<signed char, unsigned int, 31, false> >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true>, true>> = {<No data fields>}, 
        <std::__detail::_Insert<seastar::basic_sstring<signed char, unsigned int, 31, false>, std::pair<seastar::basic_sstring<signed char, unsigned int, 31, false> const, column_definition const*>, std::allocator<std::pair<seastar::basic_sstring<signed char, unsigned int, 31, false> const, column_definition const*> >, std::__detail::_Select1st, std::equal_to<seastar::basic_sstring<signed char, unsigned int, 31, false> >, std::hash<seastar::basic_sstring<signed char, unsigned int, 31, false> >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true>, false>> = {
          <std::__detail::_Insert_base<seastar::basic_sstring<signed char, unsigned int, 31, false>, std::pair<seastar::basic_sstring<signed char, unsigned int, 31, false> const, column_definition const*>, std::allocator<std::pair<seastar::basic_sstring<signed char, unsigned int, 31, false> const, column_definition const*> >, std::__detail::_Select1st, std::equal_to<seastar::basic_sstring<signed char, unsigned int, 31, false> >, std::hash<seastar::basic_sstring<signed char, unsigned int, 31, false> >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >> = {<No data fields>}, <No data fields>}, 
        <std::__detail::_Rehash_base<seastar::basic_sstring<signed char, unsigned int, 31, false>, std::pair<seastar::basic_sstring<signed char, unsigned int, 31, false> const, column_definition const*>, std::allocator<std::pair<seastar::basic_sstring<signed char, unsigned int, 31, false> const, column_definition const*> >, std::__detail::_Select1st, std::equal_to<seastar::basic_sstring<signed char, unsigned int, 31, false> >, std::hash<seastar::basic_sstring<signed char, unsigned int, 31, false> >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true>, std::integral_constant<bool, true> >> = {<No data fields>}, 
        <std::__detail::_Equality<seastar::basic_sstring<signed char, unsigned int, 31, false>, std::pair<seastar::basic_sstring<signed char, unsigned int, 31, false> const, column_definition const*>, std::allocator<std::pair<seastar::basic_sstring<signed char, unsigned int, 31, false> const, column_definition const*> >, std::__detail::_Select1st, std::equal_to<seastar::basic_sstring<signed char, unsigned int, 31, false> >, std::hash<seastar::basic_sstring<signed char, unsigned int, 31, false> >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true>, true>> = {<No data fields>}, 
        <std::__detail::_Hashtable_alloc<std::allocator<std::__detail::_Hash_node<std::pair<seastar::basic_sstring<signed char, unsigned int, 31, false> const, column_definition const*>, true> > >> = {
          <std::__detail::_Hashtable_ebo_helper<0, std::allocator<std::__detail::_Hash_node<std::pair<seastar::basic_sstring<signed char, unsigned int, 31, false> const, column_definition const*>, true> >, true>> = {
            <std::allocator<std::__detail::_Hash_node<std::pair<seastar::basic_sstring<signed char, unsigned int, 31, false> const, column_definition const*>, true> >> = {
              <__gnu_cxx::new_allocator<std::__detail::_Hash_node<std::pair<seastar::basic_sstring<signed char, unsigned int, 31, false> const, column_definition const*>, true> >> = {<No data fields>}, <No data fields>}, <No data fields>}, <No data fields>}, 
        members of std::_Hashtable<seastar::basic_sstring<signed char, unsigned int, 31, false>, std::pair<seastar::basic_sstring<signed char, unsigned int, 31, false> const, column_definition const*>, std::allocator<std::pair<seastar::basic_sstring<signed char, unsigned int, 31, false> const, column_definition const*> >, std::__detail::_Select1st, std::equal_to<seastar::basic_sstring<signed char, unsigned int, 31, false> >, std::hash<seastar::basic_sstring<signed char, unsigned int, 31, false> >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >:
        _M_buckets = 0x601000578d90,
        _M_bucket_count = 13,
        _M_before_begin = {
          _M_nxt = 0x601003122c40
        },
        _M_element_count = 3,
        _M_rehash_policy = {
          _M_max_load_factor = 1,
          _M_next_resize = 13
        },
        _M_single_bucket = 0x0
      }
    }
  },
  _registry_entry = 0x60100057ab40,
  _view_info = {
    _M_t = {
      <std::__uniq_ptr_impl<view_info, std::default_delete<view_info> >> = {
        _M_t = {
          <std::_Tuple_impl<0, view_info*, std::default_delete<view_info> >> = {
            <std::_Tuple_impl<1, std::default_delete<view_info> >> = {
              <std::_Head_base<1, std::default_delete<view_info>, true>> = {
                <std::default_delete<view_info>> = {<No data fields>}, <No data fields>}, <No data fields>}, 
            <std::_Head_base<0, view_info*, false>> = {
              _M_head_impl = 0x0
            }, <No data fields>}, <No data fields>}
      }, <No data fields>}
  },
  _offsets = {
    _M_elems = {1, 2, 2}
  },
  _columns_by_name = {
    _M_h = {
      <std::__detail::_Hashtable_base<seastar::basic_sstring<signed char, unsigned int, 31, false>, std::pair<seastar::basic_sstring<signed char, unsigned int, 31, false> const, column_definition const*>, std::__detail::_Select1st, std::equal_to<seastar::basic_sstring<signed char, unsigned int, 31, false> >, std::hash<seastar::basic_sstring<signed char, unsigned int, 31, false> >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Hashtable_traits<true, false, true> >> = {
        <std::__detail::_Hash_code_base<seastar::basic_sstring<signed char, unsigned int, 31, false>, std::pair<seastar::basic_sstring<signed char, unsigned int, 31, false> const, column_definition const*>, std::__detail::_Select1st, std::hash<seastar::basic_sstring<signed char, unsigned int, 31, false> >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, true>> = {
          <std::__detail::_Hashtable_ebo_helper<0, std::__detail::_Select1st, true>> = {
            <std::__detail::_Select1st> = {<No data fields>}, <No data fields>}, 
          <std::__detail::_Hashtable_ebo_helper<1, std::hash<seastar::basic_sstring<signed char, unsigned int, 31, false> >, true>> = {
            <std::hash<seastar::basic_sstring<signed char, unsigned int, 31, false> >> = {<No data fields>}, <No data fields>}, 
          <std::__detail::_Hashtable_ebo_helper<2, std::__detail::_Mod_range_hashing, true>> = {
            <std::__detail::_Mod_range_hashing> = {<No data fields>}, <No data fields>}, <No data fields>}, 
        <std::__detail::_Hashtable_ebo_helper<0, std::equal_to<seastar::basic_sstring<signed char, unsigned int, 31, false> >, true>> = {
          <std::equal_to<seastar::basic_sstring<signed char, unsigned int, 31, false> >> = {
            <std::binary_function<seastar::basic_sstring<signed char, unsigned int, 31, false>, seastar::basic_sstring<signed char, unsigned int, 31, false>, bool>> = {<No data fields>}, <No data fields>}, <No data fields>}, <No data fields>}, 
      <std::__detail::_Map_base<seastar::basic_sstring<signed char, unsigned int, 31, false>, std::pair<seastar::basic_sstring<signed char, unsigned int, 31, false> const, column_definition const*>, std::allocator<std::pair<seastar::basic_sstring<signed char, unsigned int, 31, false> const, column_definition const*> >, std::__detail::_Select1st, std::equal_to<seastar::basic_sstring<signed char, unsigned int, 31, false> >, std::hash<seastar::basic_sstring<signed char, unsigned int, 31, false> >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true>, true>> = {<No data fields>}, 
      <std::__detail::_Insert<seastar::basic_sstring<signed char, unsigned int, 31, false>, std::pair<seastar::basic_sstring<signed char, unsigned int, 31, false> const, column_definition const*>, std::allocator<std::pair<seastar::basic_sstring<signed char, unsigned int, 31, false> const, column_definition const*> >, std::__detail::_Select1st, std::equal_to<seastar::basic_sstring<signed char, unsigned int, 31, false> >, std::hash<seastar::basic_sstring<signed char, unsigned int, 31, false> >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true>, false>> = {
        <std::__detail::_Insert_base<seastar::basic_sstring<signed char, unsigned int, 31, false>, std::pair<seastar::basic_sstring<signed char, unsigned int, 31, false> const, column_definition const*>, std::allocator<std::pair<seastar::basic_sstring<signed char, unsigned int, 31, false> const, column_definition const*> >, std::__detail::_Select1st, std::equal_to<seastar::basic_sstring<signed char, unsigned int, 31, false> >, std::hash<seastar::basic_sstring<signed char, unsigned int, 31, false> >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >> = {<No data fields>}, <No data fields>}, 
      <std::__detail::_Rehash_base<seastar::basic_sstring<signed char, unsigned int, 31, false>, std::pair<seastar::basic_sstring<signed char, unsigned int, 31, false> const, column_definition const*>, std::allocator<std::pair<seastar::basic_sstring<signed char, unsigned int, 31, false> const, column_definition const*> >, std::__detail::_Select1st, std::equal_to<seastar::basic_sstring<signed char, unsigned int, 31, false> >, std::hash<seastar::basic_sstring<signed char, unsigned int, 31, false> >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true>, std::integral_constant<bool, true> >> = {<No data fields>}, 
      <std::__detail::_Equality<seastar::basic_sstring<signed char, unsigned int, 31, false>, std::pair<seastar::basic_sstring<signed char, unsigned int, 31, false> const, column_definition const*>, std::allocator<std::pair<seastar::basic_sstring<signed char, unsigned int, 31, false> const, column_definition const*> >, std::__detail::_Select1st, std::equal_to<seastar::basic_sstring<signed char, unsigned int, 31, false> >, std::hash<seastar::basic_sstring<signed char, unsigned int, 31, false> >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true>, true>> = {<No data fields>}, 
      <std::__detail::_Hashtable_alloc<std::allocator<std::__detail::_Hash_node<std::pair<seastar::basic_sstring<signed char, unsigned int, 31, false> const, column_definition const*>, true> > >> = {
        <std::__detail::_Hashtable_ebo_helper<0, std::allocator<std::__detail::_Hash_node<std::pair<seastar::basic_sstring<signed char, unsigned int, 31, false> const, column_definition const*>, true> >, true>> = {
          <std::allocator<std::__detail::_Hash_node<std::pair<seastar::basic_sstring<signed char, unsigned int, 31, false> const, column_definition const*>, true> >> = {
            <__gnu_cxx::new_allocator<std::__detail::_Hash_node<std::pair<seastar::basic_sstring<signed char, unsigned int, 31, false> const, column_definition const*>, true> >> = {<No data fields>}, <No data fields>}, <No data fields>}, <No data fields>}, 
      members of std::_Hashtable<seastar::basic_sstring<signed char, unsigned int, 31, false>, std::pair<seastar::basic_sstring<signed char, unsigned int, 31, false> const, column_definition const*>, std::allocator<std::pair<seastar::basic_sstring<signed char, unsigned int, 31, false> const, column_definition const*> >, std::__detail::_Select1st, std::equal_to<seastar::basic_sstring<signed char, unsigned int, 31, false> >, std::hash<seastar::basic_sstring<signed char, unsigned int, 31, false> >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >:
      _M_buckets = 0x601003265bd0,
      _M_bucket_count = 13,
      _M_before_begin = {
        _M_nxt = 0x601003122600
      },
      _M_element_count = 3,
      _M_rehash_policy = {
        _M_max_load_factor = 1,
        _M_next_resize = 13
      },
      _M_single_bucket = 0x0
    }
  },
  _partition_key_type = {
    _p = 0x601003120780
  },
  _clustering_key_type = {
    _p = 0x601003120750
  },
  _column_mapping = {
    _columns = {
      <std::_Vector_base<column_mapping_entry, std::allocator<column_mapping_entry> >> = {
        _M_impl = {
          <std::allocator<column_mapping_entry>> = {
            <__gnu_cxx::new_allocator<column_mapping_entry>> = {<No data fields>}, <No data fields>}, 
          <std::_Vector_base<column_mapping_entry, std::allocator<column_mapping_entry> >::_Vector_impl_data> = {
            _M_start = 0x601003122640,
            _M_finish = 0x601003122678,
            _M_end_of_storage = 0x601003122678
          }, <No data fields>}
      }, <No data fields>},
    _n_static = 0
  },
  _full_slice = {
    _b = 0x6010033539c0,
    _p = 0x6010033539d0
  },
  _clustering_key_size = 1,
  _regular_column_count = 1,
  _static_column_count = 0
}
bhalevy commented 4 years ago

It looks like the column_name passed via thrift contains 2 components, <4, 'v'>:

297         column_name_component = struct.pack('>i', 4)
298         # component length + component + EOC + component length + component + EOC
299         column_name = b'\x00\x04' + column_name_component + b'\x00' + b'\x00\x01' + 'v'.encode() + b'\x00'

I'm not familiar with thrift so I'm not quite sure how does this relate to the schema.

tgrabiec commented 4 years ago

Did this test ever pass with the mc format?

Looks like a bug in thrift which got exposed by the mc format, la/ka format is not affected.

add_to_mutation() is interpreting the whole cell name as clustering key, which is wrong, part of the cell name is the name of the cell:

 static void add_to_mutation(const schema& s, const Column& col, mutation& m_to_apply) {
        thrift_validation::validate_column_name(col.name);
        if (s.thrift().is_dynamic()) {
            auto&& value_col = s.regular_begin();
            add_live_cell(s, col, *value_col, make_clustering_prefix(s, to_bytes_view(col.name)), m_to_apply);
        } else {
static clustering_key_prefix make_clustering_prefix(const schema& s, bytes_view v) {
        auto composite = composite_view(v, s.thrift().has_compound_comparator());
        return clustering_key_prefix::from_exploded(composite.values());
    }
bhalevy commented 4 years ago

Did this test ever pass with the mc format?

It passed since the test didn't flush memtables. Once it did (due to an unrelated change to gracefully stop the cluster when the test is done), it hit the mc-format issue.

Looks like a bug in thrift which got exposed by the mc format, la/ka format is not affected.

add_to_mutation() is interpreting the whole cell name as clustering key, which is wrong, part of the cell name is the name of the cell:

part of the cell name is the name of the cell sounds true by definition :)

 static void add_to_mutation(const schema& s, const Column& col, mutation& m_to_apply) {
        thrift_validation::validate_column_name(col.name);
        if (s.thrift().is_dynamic()) {
            auto&& value_col = s.regular_begin();
            add_live_cell(s, col, *value_col, make_clustering_prefix(s, to_bytes_view(col.name)), m_to_apply);
        } else {
static clustering_key_prefix make_clustering_prefix(const schema& s, bytes_view v) {
        auto composite = composite_view(v, s.thrift().has_compound_comparator());
        return clustering_key_prefix::from_exploded(composite.values());
    }

How is col.name actually structured? @tgrabiec can you please point to an authoritative definition?

avikivity commented 4 years ago

In classic thrift, I believe there is no structure, Column is just a blob.

The test abuses thrift by sending a compound key via the thrift interface. It's even worse since the schema does not actually have a compound key.

Or maybe it is actually correct: the first component is the value of c, the second component tells us which column we are addressing - the column named v.

I think I'm repeating @tgrabiec in different words. So we should split the Column input, the first part names the clustering row, the second part names the column.

bhalevy commented 4 years ago

Hmm, so 0004 00000004 00 01 76 00 indicates int(4) as the value of the clustering column (c) and 'v' as the column name for which the mutation applies? and then we send the mutation with the column_name above and the value int(8) to insert 8 to 'v' and 4 to c, right?

bhalevy commented 4 years ago

Referring to https://github.com/scylladb/scylla-dtest/blob/c05fb8b8630d1ad56afb40ab71cd3749d39bfd6e/cql_tests.py#L296-L304

296         key = struct.pack('>i', 2)
297         column_name_component = struct.pack('>i', 4)
298         # component length + component + EOC + component length + component + EOC
299         column_name = b'\x00\x04' + column_name_component + b'\x00' + b'\x00\x01' + 'v'.encode() + b'\x00'
300         value = struct.pack('>i', 8)
301         client.batch_mutate(
302             {key: {'test': [Mutation(ColumnOrSuperColumn(
303                 column=Column(name=column_name, value=value, timestamp=100)))]}},
304             ThriftConsistencyLevel.ONE)
avikivity commented 4 years ago

Hmm, so 0004 00000004 00 01 76 00 indicates int(4) as the value of the clustering column (c) and 'v' as the column name for which the mutation applies? and then we send the mutation with the column_name above and the value int(8) to insert 8 to 'v' and 4 to c, right?

Yes.

tgrabiec commented 4 years ago

I think the test is correct.

The cell name in thrift is typed. The type of the cell name is called the "comparator". Cell name values should conform to the defined comparator. There is a type called a compound type, which is like a tuple<>.

In this case, the comparator will be a compound type which consists of two components, an int (which maps to the clustering key column) and a utf8 (which map to the regular column name in the row).

The format of compound type values is:

( <value-len> <value> <eoc byte> )+

tgrabiec commented 4 years ago

Did this test ever pass with the mc format?

It passed since the test didn't flush memtables. Once it did (due to an unrelated change to gracefully stop the cluster when the test is done), it hit the mc-format issue.

Looks like a bug in thrift which got exposed by the mc format, la/ka format is not affected. add_to_mutation() is interpreting the whole cell name as clustering key, which is wrong, part of the cell name is the name of the cell:

part of the cell name is the name of the cell sounds true by definition :)

Yeah :) I referred to different things in different instances of the word "cell". It should be:

"Part of the thrift cell name is the name of the CQL cell"

bhalevy commented 4 years ago

@tgrabiec please take it from here. Looks like you have much better insight into the existing code than I do.

tgrabiec commented 3 years ago

Looking closer at the issue, our thrift code is all-over unprepared for accessing regular CQL3 tables (!dense && compound). Supporting them would require a larger effort.

The code assumes that if the table has a clustering key then there is only one regular column and it's the fake one added by our translation layer. User code doesn't see it and thus should not try to put it in cell names.

Maybe we should close this issue by adding validation and disabling the test?

slivne commented 3 years ago

I do not want to increase thrift support - I want to make sure we retain the current support we have - if this is not a regression - lets do what @tgrabiec suggested

bhalevy commented 3 years ago

I do not want to increase thrift support - I want to make sure we retain the current support we have - if this is not a regression - lets do what @tgrabiec suggested

Agreed

avikivity commented 2 years ago

Fix present on all active branches, not backporting.