bhalevy
commented
3 years ago Cc @psarna @kostja
Refs #6762
Open bhalevy opened 3 years ago
bhalevy
commented
3 years ago Cc @psarna @kostja
Refs #6762
nyh
commented
3 years ago I had to look at the git history (commit aa1f952f78fbfdea62586143b1c415a613bd14bc in scylla-dtest) on what this IndexDropAdd test does, because the the comment didn't quite tell the whole story. Apparently this test runs an LWT workload on a table with a secondary index, and in parallel removes this index and then adds it again.
The message "Schema version used for view updates does not match the current base schema version of the view" comes from commit 3a6ec9933c by @tgrabiec which was later modified in commit 70e04c1123 by @eliransin. Maybe one of them remembers the logic behind this error.
In this test, the base schema is modified twice in quick succession (once to remove the index and again to re-add it), so it's not surprising that the base schema keeps changing, the question is why can't we recognize this and "do the right thing" (in this case, drop the view update) - and we do the fairly drastic on_internal_error. Another question is why we only saw this in an LWT workload - I suspect it's just coincidence, but maybe it isn't?
eliransin
commented
3 years ago At least according to the above analysis and the commit message of my own commit, there are some situations where we can read from a view table but not write to it. If a write is attempted at this point, this is probably due to the schema integrity being compromised. I'll need to validate this assumption. I will try to see if nothing stands out.
bhalevy
commented
2 years ago Scylla version 5.1.dev-0.20220603.72f629c2b6e8 with build-id 707b25a814383d0016ab743c4419823c06c479f1 starting ...
INFO 2022-06-04 07:47:27,977 [shard 1] schema_tables - Altering lwt_load_ks.table1 id=72c26180-e3da-11ec-9ac8-1729ebea2593 version=b2a9914b-43ff-3d4e-b9d5-dbf34536c11b
INFO 2022-06-04 07:47:27,980 [shard 1] schema_tables - Dropping lwt_load_ks.table1_v_idx_index id=9147cbe0-e3da-11ec-b64c-4064e47c502f version=9819d2ed-d2e9-3e1a-9e9b-ab8c945ec8f6
ERROR 2022-06-04 07:47:27,981 [shard 1] view - Error applying view update to 127.0.54.4 (view: lwt_load_ks.table1_v_idx_index, base token: 3253377485216539567, view token: 3253377485216539567): data_dictionary::no_such_column_family (Can't find a column family with UUID 9147cbe0-e3da-11ec-b64c-4064e47c502f)
ERROR 2022-06-04 07:47:27,982 [shard 2] view - Error applying view update to 127.0.54.4 (view: lwt_load_ks.table1_v_idx_index, base token: 2772720296986215918, view token: 2772720296986215918): data_dictionary::no_such_column_family (Can't find a column family with UUID 9147cbe0-e3da-11ec-b64c-4064e47c502f)
WARN 2022-06-04 07:47:27,982 [shard 2] view - Error executing build step for base lwt_load_ks.table1: data_dictionary::no_such_column_family (Can't find a column family with UUID 9147cbe0-e3da-11ec-b64c-4064e47c502f)
WARN 2022-06-04 07:47:27,983 [shard 1] view - Error executing build step for base lwt_load_ks.table1: data_dictionary::no_such_column_family (Can't find a column family with UUID 9147cbe0-e3da-11ec-b64c-4064e47c502f)
ERROR 2022-06-04 07:47:27,983 [shard 0] view - Error applying view update to 127.0.54.4 (view: lwt_load_ks.table1_v_idx_index, base token: -3687370520174064539, view token: -3687370520174064539): data_dictionary::no_such_column_family (Can't find a column family with UUID 9147cbe0-e3da-11ec-b64c-4064e47c502f)
WARN 2022-06-04 07:47:27,983 [shard 0] view - Error executing build step for base lwt_load_ks.table1: data_dictionary::no_such_column_family (Can't find a column family with UUID 9147cbe0-e3da-11ec-b64c-4064e47c502f)
ERROR 2022-06-04 07:47:27,986 [shard 3] view - Error applying view update to 127.0.54.4 (view: lwt_load_ks.table1_v_idx_index, base token: 161534891121966484, view token: 161534891121966484): data_dictionary::no_such_column_family (Can't find a column family with UUID 9147cbe0-e3da-11ec-b64c-4064e47c502f)
WARN 2022-06-04 07:47:27,989 [shard 3] view - Error executing build step for base lwt_load_ks.table1: data_dictionary::no_such_column_family (Can't find a column family with UUID 9147cbe0-e3da-11ec-b64c-4064e47c502f)
INFO 2022-06-04 07:47:28,512 [shard 0] compaction - [Compact system_schema.tables 94a10900-e3da-11ec-9f1f-1727ebea2593] Compacted 3 sstables to [/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-qfr2ie8e/test/node4/data/system_schema/tables-afddfb9dbc1e30688056eed6c302ba09/me-52-big-Data.db:level=0]. 126kB to 45kB (~35% of original) in 628ms = 71kB/s. ~384 total partitions merged to 7.
INFO 2022-06-04 07:47:28,544 [shard 0] compaction - [Compact system_schema.scylla_tables 951882f0-e3da-11ec-9f1f-1727ebea2593] Compacting [/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-qfr2ie8e/test/node4/data/system_schema/scylla_tables-5d912ff1f7593665b2c88042ab5103dd/me-52-big-Data.db:level=0:origin=memtable,/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-qfr2ie8e/test/node4/data/system_schema/scylla_tables-5d912ff1f7593665b2c88042ab5103dd/me-48-big-Data.db:level=0:origin=compaction]
INFO 2022-06-04 07:47:28,908 [shard 0] compaction - [Compact system_schema.scylla_tables 951882f0-e3da-11ec-9f1f-1727ebea2593] Compacted 2 sstables to [/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-qfr2ie8e/test/node4/data/system_schema/scylla_tables-5d912ff1f7593665b2c88042ab5103dd/me-56-big-Data.db:level=0]. 81kB to 40kB (~50% of original) in 298ms = 137kB/s. ~256 total partitions merged to 7.
INFO 2022-06-04 07:47:28,917 [shard 0] compaction - [Compact system_schema.keyspaces 95514630-e3da-11ec-9f1f-1727ebea2593] Compacting [/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-qfr2ie8e/test/node4/data/system_schema/keyspaces-abac5682dea631c5b535b3d6cffd0fb6/me-56-big-Data.db:level=0:origin=memtable,/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-qfr2ie8e/test/node4/data/system_schema/keyspaces-abac5682dea631c5b535b3d6cffd0fb6/me-60-big-Data.db:level=0:origin=compaction]
ERROR 2022-06-04 07:47:29,005 [shard 2] view - Error applying view update to 127.0.54.4 (view: lwt_load_ks.table1_v_idx_index, base token: 2772720296986215918, view token: 2772720296986215918): data_dictionary::no_such_column_family (Can't find a column family with UUID 9147cbe0-e3da-11ec-b64c-4064e47c502f)
WARN 2022-06-04 07:47:29,006 [shard 2] view - Error executing build step for base lwt_load_ks.table1: data_dictionary::no_such_column_family (Can't find a column family with UUID 9147cbe0-e3da-11ec-b64c-4064e47c502f)
ERROR 2022-06-04 07:47:29,007 [shard 0] view - Error applying view update to 127.0.54.4 (view: lwt_load_ks.table1_v_idx_index, base token: -3687370520174064539, view token: -3687370520174064539): data_dictionary::no_such_column_family (Can't find a column family with UUID 9147cbe0-e3da-11ec-b64c-4064e47c502f)
ERROR 2022-06-04 07:47:29,007 [shard 1] view - Error applying view update to 127.0.54.4 (view: lwt_load_ks.table1_v_idx_index, base token: 3253377485216539567, view token: 3253377485216539567): data_dictionary::no_such_column_family (Can't find a column family with UUID 9147cbe0-e3da-11ec-b64c-4064e47c502f)
WARN 2022-06-04 07:47:29,008 [shard 0] view - Error executing build step for base lwt_load_ks.table1: data_dictionary::no_such_column_family (Can't find a column family with UUID 9147cbe0-e3da-11ec-b64c-4064e47c502f)
WARN 2022-06-04 07:47:29,008 [shard 1] view - Error executing build step for base lwt_load_ks.table1: data_dictionary::no_such_column_family (Can't find a column family with UUID 9147cbe0-e3da-11ec-b64c-4064e47c502f)
INFO 2022-06-04 07:47:29,011 [shard 0] compaction - [Compact system_schema.keyspaces 95514630-e3da-11ec-9f1f-1727ebea2593] Compacted 2 sstables to [/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-qfr2ie8e/test/node4/data/system_schema/keyspaces-abac5682dea631c5b535b3d6cffd0fb6/me-64-big-Data.db:level=0]. 81kB to 40kB (~50% of original) in 48ms = 853kB/s. ~256 total partitions merged to 7.
INFO 2022-06-04 07:47:29,015 [shard 0] compaction - [Compact system_schema.indexes 95603a50-e3da-11ec-9f1f-1727ebea2593] Compacting [/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-qfr2ie8e/test/node4/data/system_schema/indexes-0feb57ac311f382fba6d9024d305702f/me-32-big-Data.db:level=0:origin=memtable,/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-qfr2ie8e/test/node4/data/system_schema/indexes-0feb57ac311f382fba6d9024d305702f/me-28-big-Data.db:level=0:origin=compaction]
ERROR 2022-06-04 07:47:29,027 [shard 3] view - Error applying view update to 127.0.54.4 (view: lwt_load_ks.table1_v_idx_index, base token: 161534891121966484, view token: 161534891121966484): data_dictionary::no_such_column_family (Can't find a column family with UUID 9147cbe0-e3da-11ec-b64c-4064e47c502f)
WARN 2022-06-04 07:47:29,029 [shard 3] view - Error executing build step for base lwt_load_ks.table1: data_dictionary::no_such_column_family (Can't find a column family with UUID 9147cbe0-e3da-11ec-b64c-4064e47c502f)
INFO 2022-06-04 07:47:29,070 [shard 0] compaction - [Compact system_schema.indexes 95603a50-e3da-11ec-9f1f-1727ebea2593] Compacted 2 sstables to [/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-qfr2ie8e/test/node4/data/system_schema/indexes-0feb57ac311f382fba6d9024d305702f/me-36-big-Data.db:level=0]. 81kB to 40kB (~50% of original) in 40ms = 1MB/s. ~256 total partitions merged to 3.
INFO 2022-06-04 07:47:29,073 [shard 0] compaction - [Compact system_schema.computed_columns 95693b00-e3da-11ec-9f1f-1727ebea2593] Compacting [/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-qfr2ie8e/test/node4/data/system_schema/computed_columns-cc7c7069374033c192a4c3de78dbd2c4/me-32-big-Data.db:level=0:origin=memtable,/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-qfr2ie8e/test/node4/data/system_schema/computed_columns-cc7c7069374033c192a4c3de78dbd2c4/me-28-big-Data.db:level=0:origin=compaction]
INFO 2022-06-04 07:47:29,120 [shard 0] compaction - [Compact system_schema.computed_columns 95693b00-e3da-11ec-9f1f-1727ebea2593] Compacted 2 sstables to [/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-qfr2ie8e/test/node4/data/system_schema/computed_columns-cc7c7069374033c192a4c3de78dbd2c4/me-36-big-Data.db:level=0]. 81kB to 40kB (~50% of original) in 36ms = 1MB/s. ~256 total partitions merged to 3.
INFO 2022-06-04 07:47:29,125 [shard 0] compaction - [Compact system_schema.columns 95712a40-e3da-11ec-9f1f-1727ebea2593] Compacting [/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-qfr2ie8e/test/node4/data/system_schema/columns-24101c25a2ae3af787c1b40ee1aca33f/me-56-big-Data.db:level=0:origin=memtable,/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-qfr2ie8e/test/node4/data/system_schema/columns-24101c25a2ae3af787c1b40ee1aca33f/me-52-big-Data.db:level=0:origin=compaction]
INFO 2022-06-04 07:47:29,275 [shard 0] compaction - [Compact system_schema.columns 95712a40-e3da-11ec-9f1f-1727ebea2593] Compacted 2 sstables to [/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-qfr2ie8e/test/node4/data/system_schema/columns-24101c25a2ae3af787c1b40ee1aca33f/me-60-big-Data.db:level=0]. 94kB to 53kB (~56% of original) in 136ms = 391kB/s. ~256 total partitions merged to 7.
INFO 2022-06-04 07:47:29,286 [shard 0] compaction - [Compact system_schema.views 9589bb50-e3da-11ec-9f1f-1727ebea2593] Compacting [/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-qfr2ie8e/test/node4/data/system_schema/views-9786ac1cdd583201a7cdad556410c985/me-24-big-Data.db:level=0:origin=memtable,/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-qfr2ie8e/test/node4/data/system_schema/views-9786ac1cdd583201a7cdad556410c985/me-20-big-Data.db:level=0:origin=compaction]
INFO 2022-06-04 07:47:29,355 [shard 0] compaction - [Compact system_schema.views 9589bb50-e3da-11ec-9f1f-1727ebea2593] Compacted 2 sstables to [/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-qfr2ie8e/test/node4/data/system_schema/views-9786ac1cdd583201a7cdad556410c985/me-28-big-Data.db:level=0]. 81kB to 40kB (~50% of original) in 58ms = 706kB/s. ~256 total partitions merged to 3.
INFO 2022-06-04 07:47:31,005 [shard 2] compaction - [Compact system.truncated 969007c0-e3da-11ec-bb0a-1726ebea2593] Compacting [/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-qfr2ie8e/test/node4/data/system/truncated-38c19fd0fb863310a4b70d0cc66628aa/me-22-big-Data.db:level=0:origin=memtable,/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-qfr2ie8e/test/node4/data/system/truncated-38c19fd0fb863310a4b70d0cc66628aa/mc-18-big-Data.db:level=0:origin=compaction]
ERROR 2022-06-04 07:47:31,015 [shard 0] view - Error applying view update to 127.0.54.4 (view: lwt_load_ks.table1_v_idx_index, base token: -3687370520174064539, view token: -3687370520174064539): data_dictionary::no_such_column_family (Can't find a column family with UUID 9147cbe0-e3da-11ec-b64c-4064e47c502f)
ERROR 2022-06-04 07:47:31,015 [shard 1] view - Error applying view update to 127.0.54.4 (view: lwt_load_ks.table1_v_idx_index, base token: 3253377485216539567, view token: 3253377485216539567): data_dictionary::no_such_column_family (Can't find a column family with UUID 9147cbe0-e3da-11ec-b64c-4064e47c502f)
WARN 2022-06-04 07:47:31,015 [shard 1] view - Error executing build step for base lwt_load_ks.table1: data_dictionary::no_such_column_family (Can't find a column family with UUID 9147cbe0-e3da-11ec-b64c-4064e47c502f)
WARN 2022-06-04 07:47:31,016 [shard 0] view - Error executing build step for base lwt_load_ks.table1: data_dictionary::no_such_column_family (Can't find a column family with UUID 9147cbe0-e3da-11ec-b64c-4064e47c502f)
ERROR 2022-06-04 07:47:31,018 [shard 2] view - Error applying view update to 127.0.54.4 (view: lwt_load_ks.table1_v_idx_index, base token: 2772720296986215918, view token: 2772720296986215918): data_dictionary::no_such_column_family (Can't find a column family with UUID 9147cbe0-e3da-11ec-b64c-4064e47c502f)
WARN 2022-06-04 07:47:31,019 [shard 2] view - Error executing build step for base lwt_load_ks.table1: data_dictionary::no_such_column_family (Can't find a column family with UUID 9147cbe0-e3da-11ec-b64c-4064e47c502f)
INFO 2022-06-04 07:47:31,022 [shard 0] view - Stopping to build view lwt_load_ks.table1_v_idx_index
INFO 2022-06-04 07:47:31,023 [shard 0] query_processor - Column definitions for lwt_load_ks.table1 changed, invalidating related prepared statements
INFO 2022-06-04 07:47:31,027 [shard 1] compaction_manager - Stopping 1 tasks for 0 ongoing compactions for table lwt_load_ks.table1_v_idx_index due to table removal
INFO 2022-06-04 07:47:31,033 [shard 1] query_processor - Column definitions for lwt_load_ks.table1 changed, invalidating related prepared statements
INFO 2022-06-04 07:47:31,035 [shard 2] compaction_manager - Stopping 1 tasks for 0 ongoing compactions for table lwt_load_ks.table1_v_idx_index due to table removal
INFO 2022-06-04 07:47:31,043 [shard 2] query_processor - Column definitions for lwt_load_ks.table1 changed, invalidating related prepared statements
INFO 2022-06-04 07:47:31,046 [shard 3] query_processor - Column definitions for lwt_load_ks.table1 changed, invalidating related prepared statements
ERROR 2022-06-04 07:47:31,060 [shard 3] view - Schema version used for view updates (b2a9914b-43ff-3d4e-b9d5-dbf34536c11b) does not match the current base schema version of the view (00676167-6b3f-32ae-870a-fd2766d27641) for view lwt_load_ks.table1_v_idx_index of lwt_load_ks.table1, at: 0x13c6a550 0x1bdab90d 0x1bdab5cd 0x1bdadbc3 0x1bdaea52 0x1ae82beb 0x17692e8e 0x17692778 0x1769262c 0x176925b0 0x176924d6 0x17692143 0x17691d5d 0x17652994 0x176524b4 0x14412cfc 0x17702946 0x17700374 0x176ff94b 0x176ff346 0x177074fd 0x17707408 0x1770733e 0x176feee6 0x176fe779 0x176fe510 0x176fe4c3 0x176fe423 0x176fe3d3 0x176fe35c 0x176fe2d0 0x176fe22e 0x176fdd69 0x176fd7dd 0x176fd516 0x176f2d57 0x17660469 0x176d2117 0x176d1b98 0x176d1b78 0x176d1b58 0x176d1b2c 0x176d1ab0 0x176d198e 0x176d1852 0x143053ef 0x1b416a79
--------
seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::noncopyable_function<seastar::future<void> ()>, seastar::future<void>::then_impl_nrvo<seastar::noncopyable_function<seastar::future<void> ()>, seastar::future<void> >(seastar::noncopyable_function<seastar::future<void> ()>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> ()>&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>
--------
seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>, seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)> >(seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>
--------
seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>, seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)> >(seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>
--------
seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>, seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)> >(seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>
Aborting on shard 3.
Backtrace:
0x13c6a550
0x1b0f5f0d
0x1b0f5b0c
0x1af5a1f4
0x1af85d4e
0x1b06703c
0x1b06728d
0x1b0670ba
0x7fafdab36a1f
/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.ccm/scylla-repository/72f629c2b6e80e12ec69621b9d465ab9d4c5c6a9/scylla/libreloc/libc.so.6+0x3d2a1
/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.ccm/scylla-repository/72f629c2b6e80e12ec69621b9d465ab9d4c5c6a9/scylla/libreloc/libc.so.6+0x268a3
0x1ae82ccb
0x17692e8e
0x17692778
0x1769262c
0x176925b0
0x176924d6
0x17692143
0x17691d5d
0x17652994
0x176524b4
0x14412cfc
0x17702946
0x17700374
0x176ff94b
0x176ff346
0x177074fd
0x17707408
0x1770733e
0x176feee6
0x176fe779
0x176fe510
0x176fe4c3
0x176fe423
0x176fe3d3
0x176fe35c
0x176fe2d0
0x176fe22e
0x176fdd69
0x176fd7dd
0x176fd516
0x176f2d57
0x17660469
0x176d2117
0x176d1b98
0x176d1b78
0x176d1b58
0x176d1b2c
0x176d1ab0
0x176d198e
0x176d1852
0x143053ef
0x1b416a79Scylla version 5.1.dev-0.20220716.1cb64de8d8ff with build-id 7175e44a171872bd3305cf21f03f88ae14b03930 starting ...
INFO 2022-07-17 10:05:23,247 [shard 0] schema_tables - Altering keyspace1.standard1 id=e2a8c9e0-05b7-11ed-9cc1-5f835072f635 version=b22c48fc-a624-3f13-8e28-8478f191124f
INFO 2022-07-17 10:05:23,255 [shard 0] schema_tables - Altering keyspace1.idx_index id=f22ca6c0-05b7-11ed-8048-1c9fffdbc7a0 version=57ba5d73-9709-3572-86d0-db0511358234
INFO 2022-07-17 10:05:23,267 [shard 0] query_processor - Column definitions for keyspace1.standard1 changed, invalidating related prepared statements
INFO 2022-07-17 10:05:23,274 [shard 1] query_processor - Column definitions for keyspace1.standard1 changed, invalidating related prepared statements
INFO 2022-07-17 10:05:23,512 [shard 0] schema_tables - Schema version changed to c0a26e8d-a9bc-3cd4-8864-93cbc7d980db
INFO 2022-07-17 10:05:32,382 [shard 0] schema_tables - Schema version changed to c0a26e8d-a9bc-3cd4-8864-93cbc7d980db
INFO 2022-07-17 10:05:32,383 [shard 0] migration_manager - Schema merge with 127.0.24.2:0 completed
INFO 2022-07-17 10:05:32,383 [shard 0] migration_manager - Pulling schema from 127.0.24.2:0
INFO 2022-07-17 10:05:33,489 [shard 0] schema_tables - Schema version changed to c0a26e8d-a9bc-3cd4-8864-93cbc7d980db
INFO 2022-07-17 10:05:33,489 [shard 0] migration_manager - Schema merge with 127.0.24.2:0 completed
INFO 2022-07-17 10:05:33,542 [shard 0] schema_tables - Altering keyspace1.standard1 id=e2a8c9e0-05b7-11ed-9cc1-5f835072f635 version=08c0f5ed-c1ab-32ff-bb38-824e2c684efe
INFO 2022-07-17 10:05:33,546 [shard 0] schema_tables - Dropping keyspace1.idx_index id=f22ca6c0-05b7-11ed-8048-1c9fffdbc7a0 version=57ba5d73-9709-3572-86d0-db0511358234
ERROR 2022-07-17 10:05:33,549 [shard 1] view - Error applying view update to 127.0.24.4 (view: keyspace1.idx_index, base token: -8737480579144833199, view token: -161034028378084862): data_dictionary::no_such_column_family (Can't find a column family with UUID f22ca6c0-05b7-11ed-8048-1c9fffdbc7a0)
WARN 2022-07-17 10:05:33,550 [shard 1] view - Error executing build step for base keyspace1.standard1: data_dictionary::no_such_column_family (Can't find a column family with UUID f22ca6c0-05b7-11ed-8048-1c9fffdbc7a0)
INFO 2022-07-17 10:05:34,551 [shard 0] compaction_manager - Stopping 1 tasks for 0 ongoing compactions for table keyspace1.idx_index due to table removal
INFO 2022-07-17 10:05:34,560 [shard 0] compaction - [Compact system.truncated ffb37b20-05b7-11ed-9cc1-5f835072f635] Compacting [/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-mfy9ls6d/test/node4/data/system/truncated-38c19fd0fb863310a4b70d0cc66628aa/mc-2-big-Data.db:level=0:origin=memtable,/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-mfy9ls6d/test/node4/data/system/truncated-38c19fd0fb863310a4b70d0cc66628aa/me-4-big-Data.db:level=0:origin=memtable]
INFO 2022-07-17 10:05:34,562 [shard 0] view - Stopping to build view keyspace1.idx_index
INFO 2022-07-17 10:05:34,565 [shard 0] query_processor - Column definitions for keyspace1.standard1 changed, invalidating related prepared statements
INFO 2022-07-17 10:05:34,565 [shard 1] query_processor - Column definitions for keyspace1.standard1 changed, invalidating related prepared statements
ERROR 2022-07-17 10:05:34,568 [shard 1] view - Schema version used for view updates (08c0f5ed-c1ab-32ff-bb38-824e2c684efe) does not match the current base schema version of the view (b22c48fc-a624-3f13-8e28-8478f191124f) for view keyspace1.idx_index of keyspace1.standard1, at: 0x1413cd90 0x1c3381bd 0x1c337e7d 0x1c33a473 0x1c33b302 0x1b3eeafb 0x17beb31e 0x17beac08 0x17beaabc 0x17beaa40 0x17bea966 0x17bea5d3 0x17bea1ed 0x17baac44 0x17baa764 0x1492fbac 0x17c5c636 0x17c5a064 0x17c5958b 0x17c58f86 0x17c60e9d 0x17c60da8 0x17c60cde 0x17c58b26 0x17c583b9 0x17c58150 0x17c58103 0x17c58063 0x17c58013 0x17c57f9c 0x17c57f10 0x17c57e6e 0x17c579a9 0x17c5741d 0x17c57156 0x17c4b7d7 0x17bb8829 0x17c2a697 0x17c2a118 0x17c2a0f8 0x17c2a0d8 0x17c2a0ac 0x17c2a030 0x17c29f0e 0x17c29dd2 0x147f34ff 0x1b99d7d9
--------
seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::noncopyable_function<seastar::future<void> ()>, seastar::future<void>::then_impl_nrvo<seastar::noncopyable_function<seastar::future<void> ()>, seastar::future<void> >(seastar::noncopyable_function<seastar::future<void> ()>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> ()>&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>
--------
seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>, seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)> >(seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>
--------
seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>, seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)> >(seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>
--------
seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>, seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)> >(seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>
Aborting on shard 1.
Backtrace:
0x1413cd90
0x1b67ed5d
0x1b67e95c
0x1b4e14d4
0x1b50d8ae
0x1b5ef64c
0x1b5ef8ad
0x1b5ef6da
0x7f80f9279a1f
/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.ccm/scylla-repository/1cb64de8d8ff63b6eac3f014d46fd854414b6f6e/scylla/libreloc/libc.so.6+0x3d2a1
/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.ccm/scylla-repository/1cb64de8d8ff63b6eac3f014d46fd854414b6f6e/scylla/libreloc/libc.so.6+0x268a3
0x1b3eebf0
0x17beb31e
0x17beac08
0x17beaabc
0x17beaa40
0x17bea966
0x17bea5d3
0x17bea1ed
0x17baac44
0x17baa764
0x1492fbac
0x17c5c636
0x17c5a064
0x17c5958b
0x17c58f86
0x17c60e9d
0x17c60da8
0x17c60cde
0x17c58b26
0x17c583b9
0x17c58150
0x17c58103
0x17c58063
0x17c58013
0x17c57f9c
0x17c57f10
0x17c57e6e
0x17c579a9
0x17c5741d
0x17c57156
0x17c4b7d7
0x17bb8829
0x17c2a697
0x17c2a118
0x17c2a0f8
0x17c2a0d8
0x17c2a0ac
0x17c2a030
0x17c29f0e
0x17c29dd2
0x147f34ff
0x1b99d7d9Decoded:
[Backtrace #0]
__interceptor_backtrace.part.0 at crtstuff.c:?
void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) at ./build/debug/seastar/./seastar/include/seastar/util/backtrace.hh:59
seastar::backtrace_buffer::append_backtrace() at ./build/debug/seastar/./seastar/src/core/reactor.cc:758
seastar::print_with_backtrace(seastar::backtrace_buffer&, bool) at ./build/debug/seastar/./seastar/src/core/reactor.cc:788
seastar::print_with_backtrace(char const*, bool) at ./build/debug/seastar/./seastar/src/core/reactor.cc:800
seastar::sigabrt_action() at ./build/debug/seastar/./seastar/src/core/reactor.cc:3683
operator() at ./build/debug/seastar/./seastar/src/core/reactor.cc:3659
__invoke at ./build/debug/seastar/./seastar/src/core/reactor.cc:3655
?? ??:0
/data/scylla-s3-reloc.cache/by-build-id/7175e44a171872bd3305cf21f03f88ae14b03930/extracted/scylla/libexec/scylla/libreloc/libc.so.6 0x3d2a1
/data/scylla-s3-reloc.cache/by-build-id/7175e44a171872bd3305cf21f03f88ae14b03930/extracted/scylla/libexec/scylla/libreloc/libc.so.6 0x268a3
seastar::on_internal_error(seastar::logger&, std::basic_string_view<char, std::char_traits<char> >) at ./build/debug/seastar/./seastar/src/core/on_internal_error.cc:39
operator() at ./db/view/view.cc:1089
db::view::view_updates boost::range_detail::default_constructible_unary_fn_wrapper<db::view::make_view_update_builder(seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_29, db::view::view_updates>::operator()<db::view::view_and_base>(db::view::view_and_base&) const at /usr/include/boost/range/detail/default_constructible_unary_fn.hpp:62
boost::iterators::transform_iterator<boost::range_detail::default_constructible_unary_fn_wrapper<db::view::make_view_update_builder(seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_29, db::view::view_updates>, __gnu_cxx::__normal_iterator<db::view::view_and_base*, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > >, boost::use_default, boost::use_default>::dereference() const at /usr/include/boost/iterator/transform_iterator.hpp:126
boost::iterators::transform_iterator<boost::range_detail::default_constructible_unary_fn_wrapper<db::view::make_view_update_builder(seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_29, db::view::view_updates>, __gnu_cxx::__normal_iterator<db::view::view_and_base*, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > >, boost::use_default, boost::use_default>::reference boost::iterators::iterator_core_access::dereference<boost::iterators::transform_iterator<boost::range_detail::default_constructible_unary_fn_wrapper<db::view::make_view_update_builder(seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_29, db::view::view_updates>, __gnu_cxx::__normal_iterator<db::view::view_and_base*, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > >, boost::use_default, boost::use_default> >(boost::iterators::transform_iterator<boost::range_detail::default_constructible_unary_fn_wrapper<db::view::make_view_update_builder(seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_29, db::view::view_updates>, __gnu_cxx::__normal_iterator<db::view::view_and_base*, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > >, boost::use_default, boost::use_default> const&) at /usr/include/boost/iterator/iterator_facade.hpp:550
boost::iterators::detail::iterator_facade_base<boost::iterators::transform_iterator<boost::range_detail::default_constructible_unary_fn_wrapper<db::view::make_view_update_builder(seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_29, db::view::view_updates>, __gnu_cxx::__normal_iterator<db::view::view_and_base*, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > >, boost::use_default, boost::use_default>, db::view::view_updates, boost::iterators::random_access_traversal_tag, db::view::view_updates, long, false, false>::operator*() const at /usr/include/boost/iterator/iterator_facade.hpp:656
void std::vector<db::view::view_updates, std::allocator<db::view::view_updates> >::_M_range_initialize<boost::iterators::transform_iterator<boost::range_detail::default_constructible_unary_fn_wrapper<db::view::make_view_update_builder(seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_29, db::view::view_updates>, __gnu_cxx::__normal_iterator<db::view::view_and_base*, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > >, boost::use_default, boost::use_default> >(boost::iterators::transform_iterator<boost::range_detail::default_constructible_unary_fn_wrapper<db::view::make_view_update_builder(seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_29, db::view::view_updates>, __gnu_cxx::__normal_iterator<db::view::view_and_base*, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > >, boost::use_default, boost::use_default>, boost::iterators::transform_iterator<boost::range_detail::default_constructible_unary_fn_wrapper<db::view::make_view_update_builder(seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_29, db::view::view_updates>, __gnu_cxx::__normal_iterator<db::view::view_and_base*, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > >, boost::use_default, boost::use_default>, std::input_iterator_tag) at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/stl_vector.h:1564
vector<boost::iterators::transform_iterator<boost::range_detail::default_constructible_unary_fn_wrapper<(lambda at db/view/view.cc:1087:107), db::view::view_updates>, __gnu_cxx::__normal_iterator<db::view::view_and_base *, std::vector<db::view::view_and_base> > >, void> at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/stl_vector.h:657
std::vector<db::view::view_updates, std::allocator<db::view::view_updates> > boost::copy_range<std::vector<db::view::view_updates, std::allocator<db::view::view_updates> >, boost::range_detail::transformed_range<db::view::make_view_update_builder(seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_29, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > > >(boost::range_detail::transformed_range<db::view::make_view_update_builder(seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_29, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > > const&) at /usr/include/boost/range/iterator_range_core.hpp:842
db::view::make_view_update_builder(seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >) at ./db/view/view.cc:1087
replica::table::populate_views(std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >, dht::token, flat_mutation_reader_v2&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >) at ./replica/table.cc:1820
db::view::view_builder::consumer::flush_fragments() at ./db/view/view.cc:1996
db::view::view_builder::consumer::consume_end_of_partition() at ./db/view/view.cc:2009
seastar::bool_class<seastar::stop_iteration_tag> compact_mutation_state<(compact_for_sstables)0>::consume_end_of_partition<db::view::view_builder::consumer, noop_compacted_fragments_consumer>(db::view::view_builder::consumer&, noop_compacted_fragments_consumer&) at ././mutation_compactor.hh:450
compact_mutation_v2<(compact_for_sstables)0, db::view::view_builder::consumer, noop_compacted_fragments_consumer>::consume_end_of_partition() at ././mutation_compactor.hh:599
operator() at ././readers/flat_mutation_reader_v2.hh:292
seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > seastar::futurize<seastar::bool_class<seastar::stop_iteration_tag> >::invoke<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >::consume(partition_end&&)::{lambda()#1}>(flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >::consume(partition_end&&)::{lambda()#1}&&) at ././seastar/include/seastar/core/future.hh:2151
auto seastar::futurize_invoke<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >::consume(partition_end&&)::{lambda()#1}>(flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >::consume(partition_end&&)::{lambda()#1}&&) at ././seastar/include/seastar/core/future.hh:2180
flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >::consume(partition_end&&) at ././readers/flat_mutation_reader_v2.hh:291
decltype(auto) mutation_fragment_v2::consume<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >(flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >&) && at ././mutation_fragment_v2.hh:280
flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >::operator()(mutation_fragment_v2&&) at ././readers/flat_mutation_reader_v2.hh:271
seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > std::__invoke_impl<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >, flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >&, mutation_fragment_v2>(std::__invoke_other, flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >&, mutation_fragment_v2&&) at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/invoke.h:61
std::__invoke_result<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >&, mutation_fragment_v2>::type std::__invoke<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >&, mutation_fragment_v2>(flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >&, mutation_fragment_v2&&) at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/invoke.h:96
std::result_of<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >& (mutation_fragment_v2&&)>::type std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >::operator()<mutation_fragment_v2>(mutation_fragment_v2&&) const at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/refwrap.h:349
operator() at ././readers/flat_mutation_reader_v2.hh:252
seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > seastar::futurize<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > >::invoke<flat_mutation_reader_v2::impl::consume_pausable_in_thread<std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter>(std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter)::{lambda()#1}>(flat_mutation_reader_v2::impl::consume_pausable_in_thread<std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter>(std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter)::{lambda()#1}&&) at ././seastar/include/seastar/core/future.hh:2149
auto seastar::futurize_invoke<flat_mutation_reader_v2::impl::consume_pausable_in_thread<std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter>(std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter)::{lambda()#1}>(flat_mutation_reader_v2::impl::consume_pausable_in_thread<std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter>(std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter)::{lambda()#1}&&) at ././seastar/include/seastar/core/future.hh:2180
void flat_mutation_reader_v2::impl::consume_pausable_in_thread<std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter>(std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter) at ././readers/flat_mutation_reader_v2.hh:251
auto flat_mutation_reader_v2::impl::consume_in_thread<compact_for_query_v2<db::view::view_builder::consumer>, flat_mutation_reader_v2::no_filter>(compact_for_query_v2<db::view::view_builder::consumer>, flat_mutation_reader_v2::no_filter) at ././readers/flat_mutation_reader_v2.hh:338
auto flat_mutation_reader_v2::consume_in_thread<compact_for_query_v2<db::view::view_builder::consumer>, flat_mutation_reader_v2::no_filter>(compact_for_query_v2<db::view::view_builder::consumer>, flat_mutation_reader_v2::no_filter) at ././readers/flat_mutation_reader_v2.hh:478
auto flat_mutation_reader_v2::consume_in_thread<compact_for_query_v2<db::view::view_builder::consumer> >(compact_for_query_v2<db::view::view_builder::consumer>) at ././readers/flat_mutation_reader_v2.hh:484
db::view::view_builder::execute(db::view::view_builder::build_step&, exponential_backoff_retry) at ./db/view/view.cc:2047
operator() at ./db/view/view.cc:1823
void std::__invoke_impl<void, db::view::view_builder::do_build_step()::$_23>(std::__invoke_other, db::view::view_builder::do_build_step()::$_23&&) at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/invoke.h:61
std::__invoke_result<db::view::view_builder::do_build_step()::$_23>::type std::__invoke<db::view::view_builder::do_build_step()::$_23>(db::view::view_builder::do_build_step()::$_23&&) at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/invoke.h:96
decltype(auto) std::__apply_impl<db::view::view_builder::do_build_step()::$_23, std::tuple<>>(db::view::view_builder::do_build_step()::$_23&&, std::tuple<>&&, std::integer_sequence<unsigned long>) at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/tuple:1858
decltype(auto) std::apply<db::view::view_builder::do_build_step()::$_23, std::tuple<> >(db::view::view_builder::do_build_step()::$_23&&, std::tuple<>&&) at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/tuple:1869
seastar::future<void> seastar::futurize<void>::apply<db::view::view_builder::do_build_step()::$_23>(db::view::view_builder::do_build_step()::$_23&&, std::tuple<>&&) at ././seastar/include/seastar/core/future.hh:2113
operator() at ././seastar/include/seastar/core/thread.hh:258
seastar::noncopyable_function<void ()>::direct_vtable_for<seastar::async<db::view::view_builder::do_build_step()::$_23>(seastar::thread_attributes, db::view::view_builder::do_build_step()::$_23&&)::{lambda()#1}>::call(seastar::noncopyable_function<void ()> const*) at ././seastar/include/seastar/util/noncopyable_function.hh:124
seastar::noncopyable_function<void ()>::operator()() const at ././seastar/include/seastar/util/noncopyable_function.hh:209
seastar::thread_context::main() at ./build/debug/seastar/./seastar/src/core/thread.cc:299@eliransin @psarna ping
eliransin
commented
2 years ago It is probably a race between the view/base table being removed and being modified. I will take a look.
bhalevy
commented
1 year ago Scylla version 5.2.0~dev-0.20220831.4e413787d2dd with build-id 86547bea4eec3e3a118cd3400ce8900e879fcc49 starting ...
WARN 2022-08-31 12:17:27,855 [shard 0] compaction - DateTieredCompactionStrategy is deprecated. Usually cases for which it is used are better handled by TimeWindowCompactionStrategy. Please change your compaction strategy to TWCS as DTCS will be retired in the near future
ERROR 2022-08-31 12:17:27,856 [shard 1] view - Error applying view update to 127.0.74.2 (view: keyspace1.idx_index, base token: -7792875561242238461, view token: -7792875561242238461): data_dictionary::no_such_column_family (Can't find a column family with UUID d1aee6f0-2926-11ed-ba72-2088eeb54c9f)
INFO 2022-08-31 12:17:27,856 [shard 0] view - Stopping to build view keyspace1.idx_index
WARN 2022-08-31 12:17:27,856 [shard 1] view - Error executing build step for base keyspace1.standard1: data_dictionary::no_such_column_family (Can't find a column family with UUID d1aee6f0-2926-11ed-ba72-2088eeb54c9f)
INFO 2022-08-31 12:17:27,866 [shard 0] query_processor - Column definitions for keyspace1.standard1 changed, invalidating related prepared statements
WARN 2022-08-31 12:17:27,866 [shard 1] compaction - DateTieredCompactionStrategy is deprecated. Usually cases for which it is used are better handled by TimeWindowCompactionStrategy. Please change your compaction strategy to TWCS as DTCS will be retired in the near future
INFO 2022-08-31 12:17:27,869 [shard 1] query_processor - Column definitions for keyspace1.standard1 changed, invalidating related prepared statements
ERROR 2022-08-31 12:17:27,894 [shard 0] view - Schema version used for view updates (e71db5a2-302c-360b-a324-889efb76a6dc) does not match the current base schema version of the view (090b6967-8469-3996-bf20-0fe52aa616bd) for view keyspace1.idx_index of keyspace1.standard1, at: 0x14770e10 0x1cc42c1d 0x1cc428dd 0x1cc44ed3 0x1cc45d62 0x1bcd28eb 0x18b0c41e 0x18b0bd08 0x18b0bbbc 0x18b0bb40 0x18b0ba66 0x18b0b6d3 0x18b0b2ed 0x18ab7a24 0x18ab75c4 0x157ff9ff 0x18b829c6 0x18b803f4 0x18b7f91b 0x18b7f316 0x18b8733d 0x18b87248 0x18b8717e 0x18b7eeb6 0x18b7e749 0x18b7e4e0 0x18b7e493 0x18b7e3f3 0x18b7e3a3 0x18b7e32c 0x18b7e2a0 0x18b7e1fe 0x18b7dd39 0x18b7d7ad 0x18b7d4e6 0x18b6d797 0x18ac5609 0x18b4baa7 0x18b4b528 0x18b4b508 0x18b4b4e8 0x18b4b4bc 0x18b4b440 0x18b4b31e 0x18b4b1e2 0x156b2d5f 0x1c280119@eliransin and @tgrabiec probably understand this code better than me, but when I looked at it a year ago in https://github.com/scylladb/scylladb/issues/9059#issuecomment-882098740 my conclusion was that the test mentioned then really had concurrent view updates and view deletion and recreation, so it's not surprising that the view was "pulled under our feet", and the only bug is that we consider this an "internal error" (with a crash) instead of a simple failed write operation.
The new failure is in a different test, secondary_index_test.py, test_drop_local_index_while_building. In this test too we create a view (a secondary index) and then drop it in the middle of building, so it isn't suprising that code that is still working on the view table while this happens notices that the base table schema changed - and should just stop working on the view update instead of crashing.
If I'm right in my analysis, then it should be fairly easy to reproduce this bug in a fast test which repeatedly does view creation/deletion - instead of seeeing it once a year in a dtest. I'll try to do this.
nyh
commented
1 year ago After some effort, I was able to reproduce this bug in a single-node cql-pytest setup:
The test is very simple, and can reproduce the bug in less than a second on my machine. It creates a table with some data, and then loops creating a view and deleting it. The theory is that when the base table changes its schema during the view build, the view builder will get confused. The test code:
def test_loop_build_and_delete(cql, test_keyspace):
with new_test_table(cql, test_keyspace, 'p int primary key, v int') as table:
stmt = cql.prepare(f'INSERT INTO {table} (p,v) VALUES (?, ?)')
for i in range(3000):
cql.execute(stmt, [i, i])
mv = unique_name()
for i in range(10):
cql.execute(f"CREATE MATERIALIZED VIEW {test_keyspace}.{mv} AS SELECT * FROM {table} WHERE p IS NOT NULL AND v is NOT NULL PRIMARY KEY (v,p)")
cql.execute(f"DROP MATERIALIZED VIEW {test_keyspace}.{mv}")Acutally, with unmodified Scylla, I can't reproduce this bug - I only get the more "benign" error (that we also saw in the tests above as well) that a view build fails benignly when the table it is working on is deleted:
ERROR 2022-09-04 16:49:47,544 [shard 1] view - Error applying view update to 12
7.18.215.233 (view: cql_test_1662299387024.cql_test_1662299387514, base token:
3493305906249544994, view token: 3493305906249544994): data_dictionary::no_such
_column_family (Can't find a column family with UUID 70893da0-2c58-11ed-885e-06
7b7b9a4019)This is only a log message - it doesn't break the test which doesn't know this error happened. It's also not the on_internal_error reported above, and not a crash.
To reproduce the on_internal_error, I just need to change the timing of the code with the following patch:
diff --git a/db/view/view.cc b/db/view/view.cc
index 33eaa085a..f61a606ee 100644
--- a/db/view/view.cc
+++ b/db/view/view.cc
@@ -2204,6 +2204,7 @@ class view_builder::consumer {
_builder._as.check();
if (!_fragments.empty()) {
_fragments.emplace_front(*_step.reader.schema(), _builder._permit, partition_start(_step.current_key, tombstone()));
+ seastar::sleep(100ms).get(); // NYH
auto base_schema = _step.base->schema();
auto views = with_base_info_snapshot(_views_to_build);
auto reader = make_flat_mutation_reader_from_fragments(_step.reader.schema(), _builder._permit, std::move(_fragments));Then I get the error:
ERROR 2022-09-04 17:44:44,145 [shard 1] view - Schema version used for view upd
ates (7dbe0c89-6f47-3330-a98b-1e9421afbe80) does not match the current base sch
ema version of the view (43e69397-1983-3b66-babd-378a17a9c7b4) for view cql_tes
t_1662302683557.cql_test_1662302684037 of cql_test_1662302683557.cql_test_16623
02683564
WARN 2022-09-04 17:44:44,145 [shard 1] view - Error executing build step for base cql_test_1662302683557.cql_test_1662302683564: seastar::internal::backtraced<std::runtime_error> (Schema version used for view updates (7dbe0c89-6f47-3330-a98b-1e9421afbe80) does not match the current base schema version of the view (43e69397-1983-3b66-babd-378a17a9c7b4) for view cql_test_1662302683557.cql_test_1662302684037 of cql_test_1662302683557.cql_test_1662302683564 Backtrace: 0x35689ee 0x3568f90 0x35692a8 0x110ac02 0x110aa14 0x3411260 0x34110f4 0x3410f59 0x1f1482e 0x12d7108 0x1f3da90 0x1f3cf02 0x1f3cac6 0x1f3c724 0x1f3c50a 0x1f39c70 0x1f1e0dc 0x1f3182e 0x348f0a6My first observation is that in "normal" run, outside our test suite, the "internal" error doesn't look like a disastrous crash but rather like a "benign" error in executing a view build step - one of many types of such error. This reinforces my hunch from last year that maybe this isn't a real bug but just a over-hysterical error message turned from a normal error to an on_internal_error.
The second observation is about the cause of this mismatch - we have the code in flush_fragments():
auto base_schema = _step.base->schema();
auto views = with_base_info_snapshot(_views_to_build);Which takes _views_to_build, which was collected some time ago, and compares it to _step.base->schema(), which is refreshed when the base table's schema changes, so it's not surprising - and in my opinion not a fatal error - that those don't match.
The remaining question I have is this: Can we have this mismatch when the base table changed without deleting the view? E.g., perhaps some unrelated change like adding another view or adding a column? If this is the case, then we need to handle this case - even a non-fatal error would be wrong.
DoronArazii
commented
1 year ago Not a regression - pushing to 5.2
nyh
commented
1 year ago Although it looked like it might be relevant, I checked that unforunately, pull request https://github.com/scylladb/scylladb/pull/10040 for issue #10026 does not fix THIS issue. The test above continues to print ERRORs and adding a the sleep as mentioned above turns it into a more serious failure of the build step.
bhalevy
commented
1 year ago Scylla version 5.3.0~dev-0.20230121.0d64f327e1af with build-id 63874b7fb7fa4bd80ccf05b804f5d68f418a3404 starting ...
INFO 2023-01-21 06:52:25,993 [shard 0] view - Stopping to build view lwt_load_ks.table1_v_idx_index
INFO 2023-01-21 06:52:26,005 [shard 1] query_processor - Column definitions for lwt_load_ks.table1 changed, invalidating related prepared statements
INFO 2023-01-21 06:52:26,006 [shard 2] query_processor - Column definitions for lwt_load_ks.table1 changed, invalidating related prepared statements
INFO 2023-01-21 06:52:26,010 [shard 3] query_processor - Column definitions for lwt_load_ks.table1 changed, invalidating related prepared statements
INFO 2023-01-21 06:52:26,011 [shard 0] query_processor - Column definitions for lwt_load_ks.table1 changed, invalidating related prepared statements
INFO 2023-01-21 06:52:26,476 [shard 3] compaction - [Compact system.truncated 2a00a310-9958-11ed-98d5-fe3c26ed1f6f] Compacted 2 sstables to [/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-xec6z5wk/test/node3/data/system/truncated-38c19fd0fb863310a4b70d0cc66628aa/me-31-big-Data.db:level=0]. 139kB to 98kB (~70% of original) in 508ms = 193kB/s. ~256 total partitions merged to 3.
INFO 2023-01-21 06:52:26,479 [shard 3] compaction - [Compact system.truncated 2a5579d0-9958-11ed-98d5-fe3c26ed1f6f] Compacting [/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-xec6z5wk/test/node3/data/system/truncated-38c19fd0fb863310a4b70d0cc66628aa/me-23-big-Data.db:level=0:origin=memtable,/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-xec6z5wk/test/node3/data/system/truncated-38c19fd0fb863310a4b70d0cc66628aa/me-27-big-Data.db:level=0:origin=memtable,/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-xec6z5wk/test/node3/data/system/truncated-38c19fd0fb863310a4b70d0cc66628aa/me-31-big-Data.db:level=0:origin=compaction]
INFO 2023-01-21 06:52:26,610 [shard 0] schema_tables - Schema version changed to 3ecc2589-bec1-30fb-a499-08e520f02903
INFO 2023-01-21 06:52:27,001 [shard 3] compaction - [Compact system.truncated 2a5579d0-9958-11ed-98d5-fe3c26ed1f6f] Compacted 3 sstables to [/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-xec6z5wk/test/node3/data/system/truncated-38c19fd0fb863310a4b70d0cc66628aa/me-35-big-Data.db:level=0]. 180kB to 98kB (~54% of original) in 493ms = 199kB/s. ~384 total partitions merged to 3.
INFO 2023-01-21 06:52:31,453 [shard 0] schema_tables - Altering lwt_load_ks.table1 id=eb7596f0-9957-11ed-a3d7-fe3e26ed1f6f version=c0b91e9f-01ef-373c-89bf-53574e5d279a
INFO 2023-01-21 06:52:31,459 [shard 0] schema_tables - Creating lwt_load_ks.table1_v_idx_index id=2d384e70-9958-11ed-870b-57eb228a479f version=b70108c4-5098-3015-aead-eb49ce606609
INFO 2023-01-21 06:52:31,508 [shard 0] query_processor - Column definitions for lwt_load_ks.table1 changed, invalidating related prepared statements
INFO 2023-01-21 06:52:31,508 [shard 3] query_processor - Column definitions for lwt_load_ks.table1 changed, invalidating related prepared statements
INFO 2023-01-21 06:52:31,514 [shard 2] query_processor - Column definitions for lwt_load_ks.table1 changed, invalidating related prepared statements
INFO 2023-01-21 06:52:31,523 [shard 1] query_processor - Column definitions for lwt_load_ks.table1 changed, invalidating related prepared statements
INFO 2023-01-21 06:52:32,880 [shard 0] schema_tables - Schema version changed to c24c6fdd-005e-3285-82ff-c986440b42a6
INFO 2023-01-21 06:52:35,623 [shard 0] migration_manager - Update table 'lwt_load_ks.table1' From org.apache.cassandra.config.CFMetaData@0x619000678e80[cfId=eb7596f0-9957-11ed-a3d7-fe3e26ed1f6f,ksName=lwt_load_ks,cfName=table1,cfType=Standard,comparator=org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.UTF8Type),comment=,readRepairChance=0,dcLocalReadRepairChance=0,tombstoneGcOptions={"mode":"timeout","propagation_delay_in_seconds":"3600"},gcGraceSeconds=864000,keyValidator=org.apache.cassandra.db.marshal.Int32Type,minCompactionThreshold=4,maxCompactionThreshold=32,columnMetadata=[ColumnDefinition{name=pk, type=org.apache.cassandra.db.marshal.Int32Type, kind=PARTITION_KEY, componentIndex=0, droppedAt=-9223372036854775808}, ColumnDefinition{name=int_col, type=org.apache.cassandra.db.marshal.Int32Type, kind=REGULAR, componentIndex=null, droppedAt=-9223372036854775808}, ColumnDefinition{name=v, type=org.apache.cassandra.db.marshal.Int32Type, kind=REGULAR, componentIndex=null, droppedAt=-9223372036854775808}],compactionStrategyClass=class org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={enabled=true},compressionParameters={sstable_compression=org.apache.cassandra.io.compress.LZ4Compressor},bloomFilterFpChance=0.01,memtableFlushPeriod=0,caching={"keys":"ALL","rows_per_partition":"ALL"},cdc={},defaultTimeToLive=0,minIndexInterval=128,maxIndexInterval=2048,speculativeRetry=99.0PERCENTILE,triggers=[],isDense=false,version=c0b91e9f-01ef-373c-89bf-53574e5d279a,droppedColumns={},collections={},indices={table1_v_idx : 65b7e77e-3dbc-3af2-89da-9e355316e0a9}] To org.apache.cassandra.config.CFMetaData@0x619000450b80[cfId=eb7596f0-9957-11ed-a3d7-fe3e26ed1f6f,ksName=lwt_load_ks,cfName=table1,cfType=Standard,comparator=org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.UTF8Type),comment=,readRepairChance=0,dcLocalReadRepairChance=0,tombstoneGcOptions={"mode":"timeout","propagation_delay_in_seconds":"3600"},gcGraceSeconds=864000,keyValidator=org.apache.cassandra.db.marshal.Int32Type,minCompactionThreshold=4,maxCompactionThreshold=32,columnMetadata=[ColumnDefinition{name=pk, type=org.apache.cassandra.db.marshal.Int32Type, kind=PARTITION_KEY, componentIndex=0, droppedAt=-9223372036854775808}, ColumnDefinition{name=int_col, type=org.apache.cassandra.db.marshal.Int32Type, kind=REGULAR, componentIndex=null, droppedAt=-9223372036854775808}, ColumnDefinition{name=v, type=org.apache.cassandra.db.marshal.Int32Type, kind=REGULAR, componentIndex=null, droppedAt=-9223372036854775808}],compactionStrategyClass=class org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={enabled=true},compressionParameters={sstable_compression=org.apache.cassandra.io.compress.LZ4Compressor},bloomFilterFpChance=0.01,memtableFlushPeriod=0,caching={"keys":"ALL","rows_per_partition":"ALL"},cdc={},defaultTimeToLive=0,minIndexInterval=128,maxIndexInterval=2048,speculativeRetry=99.0PERCENTILE,triggers=[],isDense=false,version=2fc90b70-9958-11ed-a3d7-fe3e26ed1f6f,droppedColumns={},collections={},indices={}]
INFO 2023-01-21 06:52:35,854 [shard 0] schema_tables - Altering lwt_load_ks.table1 id=eb7596f0-9957-11ed-a3d7-fe3e26ed1f6f version=e7c037cd-8c7c-3e2a-b1c4-bc3686cbd7d3
INFO 2023-01-21 06:52:35,860 [shard 0] schema_tables - Dropping lwt_load_ks.table1_v_idx_index id=2d384e70-9958-11ed-870b-57eb228a479f version=b70108c4-5098-3015-aead-eb49ce606609
INFO 2023-01-21 06:52:35,867 [shard 0] database - Dropping lwt_load_ks.table1_v_idx_index with auto-snapshot
ERROR 2023-01-21 06:52:35,871 [shard 2] view - Error applying view update to 127.0.22.3 (view: lwt_load_ks.table1_v_idx_index, base token: -5982998750134182192, view token: -5982998750134182192): data_dictionary::no_such_column_family (Can't find a column family with UUID 2d384e70-9958-11ed-870b-57eb228a479f)
ERROR 2023-01-21 06:52:35,872 [shard 1] view - Error applying view update to 127.0.22.3 (view: lwt_load_ks.table1_v_idx_index, base token: -5488018160383005717, view token: -5488018160383005717): data_dictionary::no_such_column_family (Can't find a column family with UUID 2d384e70-9958-11ed-870b-57eb228a479f)
WARN 2023-01-21 06:52:35,872 [shard 2] view - Error executing build step for base lwt_load_ks.table1: data_dictionary::no_such_column_family (Can't find a column family with UUID 2d384e70-9958-11ed-870b-57eb228a479f)
WARN 2023-01-21 06:52:35,873 [shard 1] view - Error executing build step for base lwt_load_ks.table1: data_dictionary::no_such_column_family (Can't find a column family with UUID 2d384e70-9958-11ed-870b-57eb228a479f)
INFO 2023-01-21 06:52:35,884 [shard 0] database - Truncating lwt_load_ks.table1_v_idx_index with auto-snapshot
ERROR 2023-01-21 06:52:36,243 [shard 3] view - Error applying view update to 127.0.22.3 (view: lwt_load_ks.table1_v_idx_index, base token: -9219848578536890599, view token: -9219848578536890599): data_dictionary::no_such_column_family (Can't find a column family with UUID 2d384e70-9958-11ed-870b-57eb228a479f)
WARN 2023-01-21 06:52:36,245 [shard 3] view - Error executing build step for base lwt_load_ks.table1: data_dictionary::no_such_column_family (Can't find a column family with UUID 2d384e70-9958-11ed-870b-57eb228a479f)
ERROR 2023-01-21 06:52:36,919 [shard 2] view - Error applying view update to 127.0.22.3 (view: lwt_load_ks.table1_v_idx_index, base token: -5982998750134182192, view token: -5982998750134182192): data_dictionary::no_such_column_family (Can't find a column family with UUID 2d384e70-9958-11ed-870b-57eb228a479f)
WARN 2023-01-21 06:52:36,920 [shard 2] view - Error executing build step for base lwt_load_ks.table1: data_dictionary::no_such_column_family (Can't find a column family with UUID 2d384e70-9958-11ed-870b-57eb228a479f)
ERROR 2023-01-21 06:52:36,970 [shard 1] view - Error applying view update to 127.0.22.3 (view: lwt_load_ks.table1_v_idx_index, base token: -5488018160383005717, view token: -5488018160383005717): data_dictionary::no_such_column_family (Can't find a column family with UUID 2d384e70-9958-11ed-870b-57eb228a479f)
WARN 2023-01-21 06:52:36,974 [shard 1] view - Error executing build step for base lwt_load_ks.table1: data_dictionary::no_such_column_family (Can't find a column family with UUID 2d384e70-9958-11ed-870b-57eb228a479f)
ERROR 2023-01-21 06:52:37,271 [shard 3] view - Error applying view update to 127.0.22.3 (view: lwt_load_ks.table1_v_idx_index, base token: -9219848578536890599, view token: -9219848578536890599): data_dictionary::no_such_column_family (Can't find a column family with UUID 2d384e70-9958-11ed-870b-57eb228a479f)
WARN 2023-01-21 06:52:37,272 [shard 3] view - Error executing build step for base lwt_load_ks.table1: data_dictionary::no_such_column_family (Can't find a column family with UUID 2d384e70-9958-11ed-870b-57eb228a479f)
INFO 2023-01-21 06:52:38,542 [shard 0] view - Building view lwt_load_ks.table1_v_idx_index, starting at token minimum token
ERROR 2023-01-21 06:52:38,597 [shard 0] view - Error applying view update to 127.0.22.3 (view: lwt_load_ks.table1_v_idx_index, base token: -9223297786983086897, view token: -9223297786983086897): data_dictionary::no_such_column_family (Can't find a column family with UUID 2d384e70-9958-11ed-870b-57eb228a479f)
WARN 2023-01-21 06:52:38,600 [shard 0] view - Error executing build step for base lwt_load_ks.table1: data_dictionary::no_such_column_family (Can't find a column family with UUID 2d384e70-9958-11ed-870b-57eb228a479f)
ERROR 2023-01-21 06:52:39,120 [shard 1] view - Error applying view update to 127.0.22.3 (view: lwt_load_ks.table1_v_idx_index, base token: -5488018160383005717, view token: -5488018160383005717): data_dictionary::no_such_column_family (Can't find a column family with UUID 2d384e70-9958-11ed-870b-57eb228a479f)
WARN 2023-01-21 06:52:39,126 [shard 1] view - Error executing build step for base lwt_load_ks.table1: data_dictionary::no_such_column_family (Can't find a column family with UUID 2d384e70-9958-11ed-870b-57eb228a479f)
ERROR 2023-01-21 06:52:39,130 [shard 2] view - Error applying view update to 127.0.22.3 (view: lwt_load_ks.table1_v_idx_index, base token: -5982998750134182192, view token: -5982998750134182192): data_dictionary::no_such_column_family (Can't find a column family with UUID 2d384e70-9958-11ed-870b-57eb228a479f)
WARN 2023-01-21 06:52:39,135 [shard 2] view - Error executing build step for base lwt_load_ks.table1: data_dictionary::no_such_column_family (Can't find a column family with UUID 2d384e70-9958-11ed-870b-57eb228a479f)
INFO 2023-01-21 06:52:39,192 [shard 3] compaction - [Compact system.truncated 31e8de30-9958-11ed-98d5-fe3c26ed1f6f] Compacting [/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-xec6z5wk/test/node3/data/system/truncated-38c19fd0fb863310a4b70d0cc66628aa/me-39-big-Data.db:level=0:origin=memtable,/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-xec6z5wk/test/node3/data/system/truncated-38c19fd0fb863310a4b70d0cc66628aa/me-35-big-Data.db:level=0:origin=compaction]
INFO 2023-01-21 06:52:39,198 [shard 3] compaction_manager - Stopping 1 tasks for 0 ongoing compactions for table lwt_load_ks.table1_v_idx_index due to table removal
WARN 2023-01-21 06:52:39,275 [shard 0] sstable - Could not remove table directory "/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-xec6z5wk/test/node3/data/lwt_load_ks/table1_v_idx_index-2d384e70995811ed870b57eb228a479f/snapshots": std::filesystem::__cxx11::filesystem_error (error system:39, filesystem error: remove failed: Directory not empty [/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-xec6z5wk/test/node3/data/lwt_load_ks/table1_v_idx_index-2d384e70995811ed870b57eb228a479f/snapshots]). Ignored.
INFO 2023-01-21 06:52:39,289 [shard 0] view - Stopping to build view lwt_load_ks.table1_v_idx_index
INFO 2023-01-21 06:52:39,296 [shard 2] query_processor - Column definitions for lwt_load_ks.table1 changed, invalidating related prepared statements
INFO 2023-01-21 06:52:39,297 [shard 0] query_processor - Column definitions for lwt_load_ks.table1 changed, invalidating related prepared statements
INFO 2023-01-21 06:52:39,298 [shard 3] query_processor - Column definitions for lwt_load_ks.table1 changed, invalidating related prepared statements
INFO 2023-01-21 06:52:39,313 [shard 1] query_processor - Column definitions for lwt_load_ks.table1 changed, invalidating related prepared statements
ERROR 2023-01-21 06:52:39,322 [shard 3] view - Schema version used for view updates (e7c037cd-8c7c-3e2a-b1c4-bc3686cbd7d3) does not match the current base schema version of the view (c0b91e9f-01ef-373c-89bf-53574e5d279a) for view lwt_load_ks.table1_v_idx_index of lwt_load_ks.table1, at: 0xe388010 0x14aa8d45 0x14aa8a9d 0x14aaaeb3 0x14aab93f 0x13b1d5ed 0x13b1cee4 0x112f5fdf 0xef51498 0x1136f23d 0x1136dc70 0x1136d13b 0x11372bcd 0x1136c322 0x1136c159 0x1136c117 0x1136c094 0x1136bd0b 0x1136b78e 0x1136b503 0x1135f573 0x11306ee6 0x1134b387 0x141572b9
--------
seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::noncopyable_function<seastar::future<void> ()>, seastar::future<void>::then_impl_nrvo<seastar::noncopyable_function<seastar::future<void> ()>, seastar::future<void> >(seastar::noncopyable_function<seastar::future<void> ()>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> ()>&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>
--------
seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>, seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)> >(seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>
--------
seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>, seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)> >(seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>
--------
seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>, seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)> >(seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>
Aborting on shard 3.
Backtrace:
0xe388010
0x13dfe015
0x13dfdc4c
0x13c58354
0x13c8784e
0x13d70fdc
0x13d7123d
0x13d7106a
/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.ccm/scylla-repository/0d64f327e1af9bcbb711ee217eda6df16e517c42/libreloc/libc.so.6+0x3cb4f
/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.ccm/scylla-repository/0d64f327e1af9bcbb711ee217eda6df16e517c42/libreloc/libc.so.6+0x8ce7b
/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.ccm/scylla-repository/0d64f327e1af9bcbb711ee217eda6df16e517c42/libreloc/libc.so.6+0x3caa5
/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.ccm/scylla-repository/0d64f327e1af9bcbb711ee217eda6df16e517c42/libreloc/libc.so.6+0x267fb
0x13b1ceee
0x112f5fdf
0xef51498
0x1136f23d
0x1136dc70
0x1136d13b
0x11372bcd
0x1136c322
0x1136c159
0x1136c117
0x1136c094
0x1136bd0b
0x1136b78e
0x1136b503
0x1135f573
0x11306ee6
0x1134b387
0x141572b9Decoded:
[Backtrace #0]
__interceptor_backtrace.part.0 at ??:?
void seastar::backtrace<seastar::current_backtrace_tasklocal()::$_3>(seastar::current_backtrace_tasklocal()::$_3&&) at ./build/debug/seastar/./seastar/include/seastar/util/backtrace.hh:59
seastar::current_backtrace_tasklocal() at ./build/debug/seastar/./seastar/src/util/backtrace.cc:86
seastar::current_tasktrace() at ./build/debug/seastar/./seastar/src/util/backtrace.cc:137
seastar::current_backtrace() at ./build/debug/seastar/./seastar/src/util/backtrace.cc:170
log_error_and_backtrace(seastar::logger&, std::basic_string_view<char, std::char_traits<char> >) at ./build/debug/seastar/./seastar/src/core/on_internal_error.cc:37
seastar::on_internal_error(seastar::logger&, std::basic_string_view<char, std::char_traits<char> >) at ./build/debug/seastar/./seastar/src/core/on_internal_error.cc:42
operator() at ./db/view/view.cc:1448
(inlined by) db::view::view_updates boost::range_detail::default_constructible_unary_fn_wrapper<db::view::make_view_update_builder(replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_33, db::view::view_updates>::operator()<db::view::view_and_base>(db::view::view_and_base&) const at /usr/include/boost/range/detail/default_constructible_unary_fn.hpp:62
(inlined by) boost::iterators::transform_iterator<boost::range_detail::default_constructible_unary_fn_wrapper<db::view::make_view_update_builder(replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_33, db::view::view_updates>, __gnu_cxx::__normal_iterator<db::view::view_and_base*, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > >, boost::use_default, boost::use_default>::dereference() const at /usr/include/boost/iterator/transform_iterator.hpp:126
(inlined by) boost::iterators::transform_iterator<boost::range_detail::default_constructible_unary_fn_wrapper<db::view::make_view_update_builder(replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_33, db::view::view_updates>, __gnu_cxx::__normal_iterator<db::view::view_and_base*, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > >, boost::use_default, boost::use_default>::reference boost::iterators::iterator_core_access::dereference<boost::iterators::transform_iterator<boost::range_detail::default_constructible_unary_fn_wrapper<db::view::make_view_update_builder(replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_33, db::view::view_updates>, __gnu_cxx::__normal_iterator<db::view::view_and_base*, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > >, boost::use_default, boost::use_default> >(boost::iterators::transform_iterator<boost::range_detail::default_constructible_unary_fn_wrapper<db::view::make_view_update_builder(replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_33, db::view::view_updates>, __gnu_cxx::__normal_iterator<db::view::view_and_base*, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > >, boost::use_default, boost::use_default> const&) at /usr/include/boost/iterator/iterator_facade.hpp:550
(inlined by) boost::iterators::detail::iterator_facade_base<boost::iterators::transform_iterator<boost::range_detail::default_constructible_unary_fn_wrapper<db::view::make_view_update_builder(replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_33, db::view::view_updates>, __gnu_cxx::__normal_iterator<db::view::view_and_base*, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > >, boost::use_default, boost::use_default>, db::view::view_updates, boost::iterators::random_access_traversal_tag, db::view::view_updates, long, false, false>::operator*() const at /usr/include/boost/iterator/iterator_facade.hpp:656
(inlined by) void std::vector<db::view::view_updates, std::allocator<db::view::view_updates> >::_M_range_initialize<boost::iterators::transform_iterator<boost::range_detail::default_constructible_unary_fn_wrapper<db::view::make_view_update_builder(replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_33, db::view::view_updates>, __gnu_cxx::__normal_iterator<db::view::view_and_base*, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > >, boost::use_default, boost::use_default> >(boost::iterators::transform_iterator<boost::range_detail::default_constructible_unary_fn_wrapper<db::view::make_view_update_builder(replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_33, db::view::view_updates>, __gnu_cxx::__normal_iterator<db::view::view_and_base*, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > >, boost::use_default, boost::use_default>, boost::iterators::transform_iterator<boost::range_detail::default_constructible_unary_fn_wrapper<db::view::make_view_update_builder(replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_33, db::view::view_updates>, __gnu_cxx::__normal_iterator<db::view::view_and_base*, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > >, boost::use_default, boost::use_default>, std::input_iterator_tag) at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/stl_vector.h:1668
(inlined by) vector<boost::iterators::transform_iterator<boost::range_detail::default_constructible_unary_fn_wrapper<(lambda at db/view/view.cc:1446:107), db::view::view_updates>, __gnu_cxx::__normal_iterator<db::view::view_and_base *, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > >, boost::use_default, boost::use_default>, void> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/stl_vector.h:706
(inlined by) std::vector<db::view::view_updates, std::allocator<db::view::view_updates> > boost::copy_range<std::vector<db::view::view_updates, std::allocator<db::view::view_updates> >, boost::range_detail::transformed_range<db::view::make_view_update_builder(replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_33, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > > >(boost::range_detail::transformed_range<db::view::make_view_update_builder(replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_33, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > > const&) at /usr/include/boost/range/iterator_range_core.hpp:842
(inlined by) db::view::make_view_update_builder(replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >) at ./db/view/view.cc:1446
replica::table::populate_views(std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >, dht::token, flat_mutation_reader_v2&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >) at ./replica/table.cc:2043
db::view::view_builder::consumer::flush_fragments() at ./db/view/view.cc:2388
db::view::view_builder::consumer::consume_end_of_partition() at ./db/view/view.cc:2401
seastar::bool_class<seastar::stop_iteration_tag> compact_mutation_state<(compact_for_sstables)0>::consume_end_of_partition<db::view::view_builder::consumer, noop_compacted_fragments_consumer>(db::view::view_builder::consumer&, noop_compacted_fragments_consumer&) at ././mutation_compactor.hh:469
operator() at ././readers/flat_mutation_reader_v2.hh:292
(inlined by) seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > seastar::futurize<seastar::bool_class<seastar::stop_iteration_tag> >::invoke<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >::consume(partition_end&&)::{lambda()#1}>(flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >::consume(partition_end&&)::{lambda()#1}&&) at ././seastar/include/seastar/core/future.hh:2149
auto seastar::futurize_invoke<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >::consume(partition_end&&)::{lambda()#1}>(flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >::consume(partition_end&&)::{lambda()#1}&&) at ././seastar/include/seastar/core/future.hh:2178
(inlined by) flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >::consume(partition_end&&) at ././readers/flat_mutation_reader_v2.hh:291
(inlined by) decltype(auto) mutation_fragment_v2::consume<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >(flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >&) && at ././mutation_fragment_v2.hh:280
flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >::operator()(mutation_fragment_v2&&) at ././readers/flat_mutation_reader_v2.hh:271
(inlined by) seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > std::__invoke_impl<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >, flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >&, mutation_fragment_v2>(std::__invoke_other, flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >&, mutation_fragment_v2&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:61
(inlined by) std::__invoke_result<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >&, mutation_fragment_v2>::type std::__invoke<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >&, mutation_fragment_v2>(flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >&, mutation_fragment_v2&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:96
std::result_of<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >& (mutation_fragment_v2&&)>::type std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >::operator()<mutation_fragment_v2>(mutation_fragment_v2&&) const at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/refwrap.h:358
operator() at ././readers/flat_mutation_reader_v2.hh:252
(inlined by) seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > seastar::futurize<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > >::invoke<flat_mutation_reader_v2::impl::consume_pausable_in_thread<std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter>(std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter)::{lambda()#1}>(flat_mutation_reader_v2::impl::consume_pausable_in_thread<std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter>(std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter)::{lambda()#1}&&) at ././seastar/include/seastar/core/future.hh:2147
auto seastar::futurize_invoke<flat_mutation_reader_v2::impl::consume_pausable_in_thread<std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter>(std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter)::{lambda()#1}>(flat_mutation_reader_v2::impl::consume_pausable_in_thread<std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter>(std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter)::{lambda()#1}&&) at ././seastar/include/seastar/core/future.hh:2178
(inlined by) void flat_mutation_reader_v2::impl::consume_pausable_in_thread<std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter>(std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter) at ././readers/flat_mutation_reader_v2.hh:251
auto flat_mutation_reader_v2::impl::consume_in_thread<compact_for_query_v2<db::view::view_builder::consumer>, flat_mutation_reader_v2::no_filter>(compact_for_query_v2<db::view::view_builder::consumer>, flat_mutation_reader_v2::no_filter) at ././readers/flat_mutation_reader_v2.hh:338
auto flat_mutation_reader_v2::consume_in_thread<compact_for_query_v2<db::view::view_builder::consumer>, flat_mutation_reader_v2::no_filter>(compact_for_query_v2<db::view::view_builder::consumer>, flat_mutation_reader_v2::no_filter) at ././readers/flat_mutation_reader_v2.hh:478
auto flat_mutation_reader_v2::consume_in_thread<compact_for_query_v2<db::view::view_builder::consumer> >(compact_for_query_v2<db::view::view_builder::consumer>) at ././readers/flat_mutation_reader_v2.hh:484
db::view::view_builder::execute(db::view::view_builder::build_step&, exponential_backoff_retry) at ./db/view/view.cc:2438
operator() at ./db/view/view.cc:2206
(inlined by) void std::__invoke_impl<void, db::view::view_builder::do_build_step()::$_26>(std::__invoke_other, db::view::view_builder::do_build_step()::$_26&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:61
(inlined by) std::__invoke_result<db::view::view_builder::do_build_step()::$_26>::type std::__invoke<db::view::view_builder::do_build_step()::$_26>(db::view::view_builder::do_build_step()::$_26&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:96
(inlined by) decltype(auto) std::__apply_impl<db::view::view_builder::do_build_step()::$_26, std::tuple<>>(db::view::view_builder::do_build_step()::$_26&&, std::tuple<>&&, std::integer_sequence<unsigned long>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/tuple:1852
(inlined by) decltype(auto) std::apply<db::view::view_builder::do_build_step()::$_26, std::tuple<> >(db::view::view_builder::do_build_step()::$_26&&, std::tuple<>&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/tuple:1863
(inlined by) seastar::future<void> seastar::futurize<void>::apply<db::view::view_builder::do_build_step()::$_26>(db::view::view_builder::do_build_step()::$_26&&, std::tuple<>&&) at ././seastar/include/seastar/core/future.hh:2111
(inlined by) operator() at ././seastar/include/seastar/core/thread.hh:258
(inlined by) seastar::noncopyable_function<void ()>::direct_vtable_for<seastar::async<db::view::view_builder::do_build_step()::$_26>(seastar::thread_attributes, db::view::view_builder::do_build_step()::$_26&&)::{lambda()#1}>::call(seastar::noncopyable_function<void ()> const*) at ././seastar/include/seastar/util/noncopyable_function.hh:124
seastar::thread_context::main() at ./build/debug/seastar/./seastar/src/core/thread.cc:299@nyh / @eliransin are we planning to deal with it in 5.2?
nyh
commented
1 year ago @DoronArazii, Since @bhalevy noted above this is still happening in 5.3 so I guess also master, why are you asking specifically on 5.2?
DoronArazii
commented
1 year ago @nyh i'm trying to close the content for milestone 5.2 - and this case is under this milestone. If we can't commit to solve it, it's fine, I can remove it to the large backlog of 5.x (it means - sometime during 5.x versions we will have it, but it's unknown when). *maybe I made a mistake by adding this to 5.2 milestone on October 2022 (too optimistic...)
mykaul
commented
1 year ago @nyh i'm trying to close the content for milestone 5.2 - and this case is under this milestone. If we can't commit to solve it, it's fine, I can remove it to the large backlog of 5.x (it means - sometime during 5.x versions we will have it, but it's unknown when). *maybe I made a mistake by adding this to 5.2 milestone on October 2022 (too optimistic...)
Being optimistic is good! Nothing magical is going to happen right now that will make this issue fixed - let's defer to 5.3 (to signal we DO wish to see it fixed, rather than shelf it on 5.x). If it gets fixed soonish and we can backport - even better.
bhalevy
commented
1 year ago Scylla version 5.3.0~dev-0.20230308.cee597560a58 with build-id c6ebc8368d86040f1ade7ff0365c5801453709d3 starting ...
INFO 2023-03-08 07:06:11,517 [shard 0] view - Building view lwt_load_ks.table1_v_idx_index, starting at token minimum token
INFO 2023-03-08 07:06:11,581 [shard 0] compaction - [Compact lwt_load_ks.table1 b5182c70-bd7f-11ed-a9b6-db3566b700c2] Compacting [/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-25jb799a/test/node3/data/lwt_load_ks/table1-866bc760bd7f11ed94eee9187aff873c/me-8-big-Data.db:level=0:origin=memtable,/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-25jb799a/test/node3/data/lwt_load_ks/table1-866bc760bd7f11ed94eee9187aff873c/me-4-big-Data.db:level=0:origin=memtable]
INFO 2023-03-08 07:06:14,668 [shard 0] schema_tables - Altering lwt_load_ks.table1 id=866bc760-bd7f-11ed-94ee-e9187aff873c version=193d4f19-1b0b-355d-b447-94cb5f11bc12
INFO 2023-03-08 07:06:14,671 [shard 0] schema_tables - Dropping lwt_load_ks.table1_v_idx_index id=b498ea50-bd7f-11ed-a9b6-db3566b700c2 version=0240784d-847f-3700-8752-e7e9a1f63679
INFO 2023-03-08 07:06:14,675 [shard 0] database - Dropping lwt_load_ks.table1_v_idx_index with auto-snapshot
ERROR 2023-03-08 07:06:14,678 [shard 3] view - Error applying view update to 127.0.75.3 (view: lwt_load_ks.table1_v_idx_index, base token: 3480750085395428140, view token: 3480750085395428140): data_dictionary::no_such_column_family (Can't find a column family with UUID b498ea50-bd7f-11ed-a9b6-db3566b700c2)
ERROR 2023-03-08 07:06:14,678 [shard 2] view - Error applying view update to 127.0.75.3 (view: lwt_load_ks.table1_v_idx_index, base token: 6740312425489355467, view token: 6740312425489355467): data_dictionary::no_such_column_family (Can't find a column family with UUID b498ea50-bd7f-11ed-a9b6-db3566b700c2)
ERROR 2023-03-08 07:06:14,678 [shard 0] view - Error applying view update to 127.0.75.3 (view: lwt_load_ks.table1_v_idx_index, base token: 1509400826364173356, view token: 1509400826364173356): data_dictionary::no_such_column_family (Can't find a column family with UUID b498ea50-bd7f-11ed-a9b6-db3566b700c2)
WARN 2023-03-08 07:06:14,679 [shard 3] view - Error executing build step for base lwt_load_ks.table1: data_dictionary::no_such_column_family (Can't find a column family with UUID b498ea50-bd7f-11ed-a9b6-db3566b700c2)
INFO 2023-03-08 07:06:14,679 [shard 0] database - Truncating lwt_load_ks.table1_v_idx_index with auto-snapshot
WARN 2023-03-08 07:06:14,679 [shard 2] view - Error executing build step for base lwt_load_ks.table1: data_dictionary::no_such_column_family (Can't find a column family with UUID b498ea50-bd7f-11ed-a9b6-db3566b700c2)
WARN 2023-03-08 07:06:14,680 [shard 0] view - Error executing build step for base lwt_load_ks.table1: data_dictionary::no_such_column_family (Can't find a column family with UUID b498ea50-bd7f-11ed-a9b6-db3566b700c2)
ERROR 2023-03-08 07:06:14,691 [shard 1] view - Error applying view update to 127.0.75.3 (view: lwt_load_ks.table1_v_idx_index, base token: 407052670363649979, view token: 407052670363649979): data_dictionary::no_such_column_family (Can't find a column family with UUID b498ea50-bd7f-11ed-a9b6-db3566b700c2)
WARN 2023-03-08 07:06:14,692 [shard 1] view - Error executing build step for base lwt_load_ks.table1: data_dictionary::no_such_column_family (Can't find a column family with UUID b498ea50-bd7f-11ed-a9b6-db3566b700c2)
ERROR 2023-03-08 07:06:15,710 [shard 2] view - Error applying view update to 127.0.75.3 (view: lwt_load_ks.table1_v_idx_index, base token: 6740312425489355467, view token: 6740312425489355467): data_dictionary::no_such_column_family (Can't find a column family with UUID b498ea50-bd7f-11ed-a9b6-db3566b700c2)
WARN 2023-03-08 07:06:15,712 [shard 2] view - Error executing build step for base lwt_load_ks.table1: data_dictionary::no_such_column_family (Can't find a column family with UUID b498ea50-bd7f-11ed-a9b6-db3566b700c2)
ERROR 2023-03-08 07:06:15,715 [shard 3] view - Error applying view update to 127.0.75.3 (view: lwt_load_ks.table1_v_idx_index, base token: 3480750085395428140, view token: 3480750085395428140): data_dictionary::no_such_column_family (Can't find a column family with UUID b498ea50-bd7f-11ed-a9b6-db3566b700c2)
WARN 2023-03-08 07:06:15,716 [shard 3] view - Error executing build step for base lwt_load_ks.table1: data_dictionary::no_such_column_family (Can't find a column family with UUID b498ea50-bd7f-11ed-a9b6-db3566b700c2)
ERROR 2023-03-08 07:06:15,736 [shard 0] view - Error applying view update to 127.0.75.3 (view: lwt_load_ks.table1_v_idx_index, base token: 1509400826364173356, view token: 1509400826364173356): data_dictionary::no_such_column_family (Can't find a column family with UUID b498ea50-bd7f-11ed-a9b6-db3566b700c2)
WARN 2023-03-08 07:06:15,737 [shard 0] view - Error executing build step for base lwt_load_ks.table1: data_dictionary::no_such_column_family (Can't find a column family with UUID b498ea50-bd7f-11ed-a9b6-db3566b700c2)
ERROR 2023-03-08 07:06:15,856 [shard 1] view - Error applying view update to 127.0.75.3 (view: lwt_load_ks.table1_v_idx_index, base token: 407052670363649979, view token: 407052670363649979): data_dictionary::no_such_column_family (Can't find a column family with UUID b498ea50-bd7f-11ed-a9b6-db3566b700c2)
WARN 2023-03-08 07:06:15,858 [shard 1] view - Error executing build step for base lwt_load_ks.table1: data_dictionary::no_such_column_family (Can't find a column family with UUID b498ea50-bd7f-11ed-a9b6-db3566b700c2)
ERROR 2023-03-08 07:06:17,737 [shard 2] view - Error applying view update to 127.0.75.3 (view: lwt_load_ks.table1_v_idx_index, base token: 6740312425489355467, view token: 6740312425489355467): data_dictionary::no_such_column_family (Can't find a column family with UUID b498ea50-bd7f-11ed-a9b6-db3566b700c2)
WARN 2023-03-08 07:06:17,738 [shard 2] view - Error executing build step for base lwt_load_ks.table1: data_dictionary::no_such_column_family (Can't find a column family with UUID b498ea50-bd7f-11ed-a9b6-db3566b700c2)
ERROR 2023-03-08 07:06:17,767 [shard 3] view - Error applying view update to 127.0.75.3 (view: lwt_load_ks.table1_v_idx_index, base token: 3480750085395428140, view token: 3480750085395428140): data_dictionary::no_such_column_family (Can't find a column family with UUID b498ea50-bd7f-11ed-a9b6-db3566b700c2)
WARN 2023-03-08 07:06:17,768 [shard 3] view - Error executing build step for base lwt_load_ks.table1: data_dictionary::no_such_column_family (Can't find a column family with UUID b498ea50-bd7f-11ed-a9b6-db3566b700c2)
ERROR 2023-03-08 07:06:17,780 [shard 0] view - Error applying view update to 127.0.75.3 (view: lwt_load_ks.table1_v_idx_index, base token: 1509400826364173356, view token: 1509400826364173356): data_dictionary::no_such_column_family (Can't find a column family with UUID b498ea50-bd7f-11ed-a9b6-db3566b700c2)
WARN 2023-03-08 07:06:17,781 [shard 0] view - Error executing build step for base lwt_load_ks.table1: data_dictionary::no_such_column_family (Can't find a column family with UUID b498ea50-bd7f-11ed-a9b6-db3566b700c2)
ERROR 2023-03-08 07:06:18,141 [shard 1] view - Error applying view update to 127.0.75.3 (view: lwt_load_ks.table1_v_idx_index, base token: 407052670363649979, view token: 407052670363649979): data_dictionary::no_such_column_family (Can't find a column family with UUID b498ea50-bd7f-11ed-a9b6-db3566b700c2)
WARN 2023-03-08 07:06:18,146 [shard 1] view - Error executing build step for base lwt_load_ks.table1: data_dictionary::no_such_column_family (Can't find a column family with UUID b498ea50-bd7f-11ed-a9b6-db3566b700c2)
INFO 2023-03-08 07:06:18,888 [shard 2] compaction - [Compact lwt_load_ks.table1 b4b41370-bd7f-11ed-b5a4-db3366b700c2] Compacted 2 sstables to [/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-25jb799a/test/node3/data/lwt_load_ks/table1-866bc760bd7f11ed94eee9187aff873c/me-14-big-Data.db:level=0]. 221kB to 180kB (~81% of original) in 8010ms = 22kB/s. ~2048 total partitions merged to 1816.
INFO 2023-03-08 07:06:19,006 [shard 0] compaction_manager - Stopping 1 tasks for 0 ongoing compactions for table lwt_load_ks.table1_v_idx_index due to table removal
INFO 2023-03-08 07:06:19,006 [shard 1] compaction_manager - Stopping 1 tasks for 0 ongoing compactions for table lwt_load_ks.table1_v_idx_index due to table removal
INFO 2023-03-08 07:06:19,006 [shard 3] compaction_manager - Stopping 1 tasks for 0 ongoing compactions for table lwt_load_ks.table1_v_idx_index due to table removal
INFO 2023-03-08 07:06:19,007 [shard 2] compaction_manager - Stopping 1 tasks for 0 ongoing compactions for table lwt_load_ks.table1_v_idx_index due to table removal
WARN 2023-03-08 07:06:19,029 [shard 0] sstable - Could not remove table directory "/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-25jb799a/test/node3/data/lwt_load_ks/table1_v_idx_index-b498ea50bd7f11eda9b6db3566b700c2/snapshots": std::filesystem::__cxx11::filesystem_error (error system:39, filesystem error: remove failed: Directory not empty [/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-25jb799a/test/node3/data/lwt_load_ks/table1_v_idx_index-b498ea50bd7f11eda9b6db3566b700c2/snapshots]). Ignored.
INFO 2023-03-08 07:06:19,036 [shard 0] view - Stopping to build view lwt_load_ks.table1_v_idx_index
INFO 2023-03-08 07:06:19,039 [shard 0] query_processor - Column definitions for lwt_load_ks.table1 changed, invalidating related prepared statements
INFO 2023-03-08 07:06:19,040 [shard 3] query_processor - Column definitions for lwt_load_ks.table1 changed, invalidating related prepared statements
INFO 2023-03-08 07:06:19,040 [shard 1] query_processor - Column definitions for lwt_load_ks.table1 changed, invalidating related prepared statements
INFO 2023-03-08 07:06:19,042 [shard 2] query_processor - Column definitions for lwt_load_ks.table1 changed, invalidating related prepared statements
INFO 2023-03-08 07:06:19,347 [shard 0] schema_tables - Schema version changed to d6304e3b-60f2-3980-8019-30f556b88be5
INFO 2023-03-08 07:06:19,608 [shard 3] compaction - [Compact lwt_load_ks.table1 b4d927a0-bd7f-11ed-8d91-db3466b700c2] Compacted 2 sstables to [/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-25jb799a/test/node3/data/lwt_load_ks/table1-866bc760bd7f11ed94eee9187aff873c/me-15-big-Data.db:level=0]. 278kB to 184kB (~66% of original) in 8417ms = 21kB/s. ~2176 total partitions merged to 1922.
INFO 2023-03-08 07:06:19,611 [shard 3] compaction - [Compact system.truncated b9ebd490-bd7f-11ed-8d91-db3466b700c2] Compacting [/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-25jb799a/test/node3/data/system/truncated-38c19fd0fb863310a4b70d0cc66628aa/me-43-big-Data.db:level=0:origin=memtable,/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-25jb799a/test/node3/data/system/truncated-38c19fd0fb863310a4b70d0cc66628aa/me-47-big-Data.db:level=0:origin=memtable,/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-25jb799a/test/node3/data/system/truncated-38c19fd0fb863310a4b70d0cc66628aa/me-39-big-Data.db:level=0:origin=compaction]
INFO 2023-03-08 07:06:19,648 [shard 0] compaction - [Compact lwt_load_ks.table1 b5182c70-bd7f-11ed-a9b6-db3566b700c2] Compacted 2 sstables to [/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-25jb799a/test/node3/data/lwt_load_ks/table1-866bc760bd7f11ed94eee9187aff873c/me-12-big-Data.db:level=0]. 266kB to 184kB (~69% of original) in 8027ms = 22kB/s. ~2048 total partitions merged to 1903.
INFO 2023-03-08 07:06:19,651 [shard 0] compaction - [Compact system_schema.columns b9f1ef10-bd7f-11ed-a9b6-db3566b700c2] Compacting [/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-25jb799a/test/node3/data/system_schema/columns-24101c25a2ae3af787c1b40ee1aca33f/me-16-big-Data.db:level=0:origin=memtable,/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-25jb799a/test/node3/data/system_schema/columns-24101c25a2ae3af787c1b40ee1aca33f/md-12-big-Data.db:level=0:origin=compaction]
INFO 2023-03-08 07:06:19,852 [shard 3] compaction - [Compact system.truncated b9ebd490-bd7f-11ed-8d91-db3466b700c2] Compacted 3 sstables to [/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-25jb799a/test/node3/data/system/truncated-38c19fd0fb863310a4b70d0cc66628aa/me-51-big-Data.db:level=0]. 180kB to 98kB (~54% of original) in 220ms = 446kB/s. ~384 total partitions merged to 4.
INFO 2023-03-08 07:06:19,898 [shard 0] compaction - [Compact system_schema.columns b9f1ef10-bd7f-11ed-a9b6-db3566b700c2] Compacted 2 sstables to [/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-25jb799a/test/node3/data/system_schema/columns-24101c25a2ae3af787c1b40ee1aca33f/me-20-big-Data.db:level=0]. 94kB to 53kB (~56% of original) in 232ms = 229kB/s. ~256 total partitions merged to 7.
INFO 2023-03-08 07:06:19,901 [shard 0] compaction - [Compact system_schema.tables ba1814b0-bd7f-11ed-a9b6-db3566b700c2] Compacting [/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-25jb799a/test/node3/data/system_schema/tables-afddfb9dbc1e30688056eed6c302ba09/me-8-big-Data.db:level=0:origin=memtable,/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-25jb799a/test/node3/data/system_schema/tables-afddfb9dbc1e30688056eed6c302ba09/mc-4-big-Data.db:level=0:origin=memtable]
INFO 2023-03-08 07:06:20,007 [shard 0] compaction - [Compact system_schema.tables ba1814b0-bd7f-11ed-a9b6-db3566b700c2] Compacted 2 sstables to [/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-25jb799a/test/node3/data/system_schema/tables-afddfb9dbc1e30688056eed6c302ba09/me-12-big-Data.db:level=0]. 86kB to 45kB (~52% of original) in 91ms = 495kB/s. ~256 total partitions merged to 7.
INFO 2023-03-08 07:06:20,729 [shard 1] compaction - [Compact lwt_load_ks.table1 b4e8df10-bd7f-11ed-853e-db3266b700c2] Compacted 2 sstables to [/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-25jb799a/test/node3/data/lwt_load_ks/table1-866bc760bd7f11ed94eee9187aff873c/me-13-big-Data.db:level=0]. 299kB to 184kB (~61% of original) in 9506ms = 19kB/s. ~2048 total partitions merged to 1880.
INFO 2023-03-08 07:06:21,586 [shard 0] migration_manager - Update table 'lwt_load_ks.table1' From org.apache.cassandra.config.CFMetaData@0x619000a4f680[cfId=866bc760-bd7f-11ed-94ee-e9187aff873c,ksName=lwt_load_ks,cfName=table1,cfType=Standard,comparator=org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.UTF8Type),comment=,readRepairChance=0,dcLocalReadRepairChance=0,tombstoneGcOptions={"mode":"timeout","propagation_delay_in_seconds":"3600"},gcGraceSeconds=864000,keyValidator=org.apache.cassandra.db.marshal.Int32Type,minCompactionThreshold=4,maxCompactionThreshold=32,columnMetadata=[ColumnDefinition{name=pk, type=org.apache.cassandra.db.marshal.Int32Type, kind=PARTITION_KEY, componentIndex=0, droppedAt=-9223372036854775808}, ColumnDefinition{name=int_col, type=org.apache.cassandra.db.marshal.Int32Type, kind=REGULAR, componentIndex=null, droppedAt=-9223372036854775808}, ColumnDefinition{name=v, type=org.apache.cassandra.db.marshal.Int32Type, kind=REGULAR, componentIndex=null, droppedAt=-9223372036854775808}],compactionStrategyClass=class org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={enabled=true},compressionParameters={sstable_compression=org.apache.cassandra.io.compress.LZ4Compressor},bloomFilterFpChance=0.01,memtableFlushPeriod=0,caching={"keys":"ALL","rows_per_partition":"ALL"},cdc={},defaultTimeToLive=0,minIndexInterval=128,maxIndexInterval=2048,speculativeRetry=99.0PERCENTILE,triggers=[],isDense=false,version=193d4f19-1b0b-355d-b447-94cb5f11bc12,droppedColumns={},collections={},indices={}] To org.apache.cassandra.config.CFMetaData@0x619000638d80[cfId=866bc760-bd7f-11ed-94ee-e9187aff873c,ksName=lwt_load_ks,cfName=table1,cfType=Standard,comparator=org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.UTF8Type),comment=,readRepairChance=0,dcLocalReadRepairChance=0,tombstoneGcOptions={"mode":"timeout","propagation_delay_in_seconds":"3600"},gcGraceSeconds=864000,keyValidator=org.apache.cassandra.db.marshal.Int32Type,minCompactionThreshold=4,maxCompactionThreshold=32,columnMetadata=[ColumnDefinition{name=pk, type=org.apache.cassandra.db.marshal.Int32Type, kind=PARTITION_KEY, componentIndex=0, droppedAt=-9223372036854775808}, ColumnDefinition{name=int_col, type=org.apache.cassandra.db.marshal.Int32Type, kind=REGULAR, componentIndex=null, droppedAt=-9223372036854775808}, ColumnDefinition{name=v, type=org.apache.cassandra.db.marshal.Int32Type, kind=REGULAR, componentIndex=null, droppedAt=-9223372036854775808}],compactionStrategyClass=class org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={enabled=true},compressionParameters={sstable_compression=org.apache.cassandra.io.compress.LZ4Compressor},bloomFilterFpChance=0.01,memtableFlushPeriod=0,caching={"keys":"ALL","rows_per_partition":"ALL"},cdc={},defaultTimeToLive=0,minIndexInterval=128,maxIndexInterval=2048,speculativeRetry=99.0PERCENTILE,triggers=[],isDense=false,version=bb197f20-bd7f-11ed-a9b6-db3566b700c2,droppedColumns={},collections={},indices={table1_v_idx : 65b7e77e-3dbc-3af2-89da-9e355316e0a9}]
INFO 2023-03-08 07:06:21,691 [shard 0] schema_tables - Altering lwt_load_ks.table1 id=866bc760-bd7f-11ed-94ee-e9187aff873c version=e5278e1b-64e6-38f1-8900-c578796bb1eb
INFO 2023-03-08 07:06:21,695 [shard 0] schema_tables - Creating lwt_load_ks.table1_v_idx_index id=bb19a630-bd7f-11ed-a9b6-db3566b700c2 version=df250033-3fee-33bd-adc8-aa6add48e69b
INFO 2023-03-08 07:06:21,711 [shard 0] query_processor - Column definitions for lwt_load_ks.table1 changed, invalidating related prepared statements
INFO 2023-03-08 07:06:21,719 [shard 3] query_processor - Column definitions for lwt_load_ks.table1 changed, invalidating related prepared statements
INFO 2023-03-08 07:06:21,719 [shard 1] query_processor - Column definitions for lwt_load_ks.table1 changed, invalidating related prepared statements
INFO 2023-03-08 07:06:21,724 [shard 2] query_processor - Column definitions for lwt_load_ks.table1 changed, invalidating related prepared statements
INFO 2023-03-08 07:06:21,832 [shard 0] view - Building view lwt_load_ks.table1_v_idx_index, starting at token 1509400826364173356
INFO 2023-03-08 07:06:22,278 [shard 0] schema_tables - Schema version changed to 438a7a69-893c-3003-9047-b0c82349197b
INFO 2023-03-08 07:06:25,802 [shard 0] schema_tables - Altering lwt_load_ks.table1 id=866bc760-bd7f-11ed-94ee-e9187aff873c version=dd78b246-57da-380f-bf1f-566f055419e5
INFO 2023-03-08 07:06:25,805 [shard 0] schema_tables - Dropping lwt_load_ks.table1_v_idx_index id=bb19a630-bd7f-11ed-a9b6-db3566b700c2 version=df250033-3fee-33bd-adc8-aa6add48e69b
INFO 2023-03-08 07:06:25,805 [shard 0] database - Dropping lwt_load_ks.table1_v_idx_index with auto-snapshot
ERROR 2023-03-08 07:06:25,807 [shard 1] view - Error applying view update to 127.0.75.3 (view: lwt_load_ks.table1_v_idx_index, base token: -1871298252625020920, view token: -1871298252625020920): data_dictionary::no_such_column_family (Can't find a column family with UUID bb19a630-bd7f-11ed-a9b6-db3566b700c2)
ERROR 2023-03-08 07:06:25,808 [shard 0] view - Error applying view update to 127.0.75.3 (view: lwt_load_ks.table1_v_idx_index, base token: -4358577383040064436, view token: -4358577383040064436): data_dictionary::no_such_column_family (Can't find a column family with UUID bb19a630-bd7f-11ed-a9b6-db3566b700c2)
WARN 2023-03-08 07:06:25,808 [shard 1] view - Error executing build step for base lwt_load_ks.table1: data_dictionary::no_such_column_family (Can't find a column family with UUID bb19a630-bd7f-11ed-a9b6-db3566b700c2)
INFO 2023-03-08 07:06:25,809 [shard 0] database - Truncating lwt_load_ks.table1_v_idx_index with auto-snapshot
WARN 2023-03-08 07:06:25,811 [shard 0] view - Error executing build step for base lwt_load_ks.table1: data_dictionary::no_such_column_family (Can't find a column family with UUID bb19a630-bd7f-11ed-a9b6-db3566b700c2)
ERROR 2023-03-08 07:06:25,820 [shard 3] view - Error applying view update to 127.0.75.3 (view: lwt_load_ks.table1_v_idx_index, base token: 3345660330269217793, view token: 3345660330269217793): data_dictionary::no_such_column_family (Can't find a column family with UUID bb19a630-bd7f-11ed-a9b6-db3566b700c2)
WARN 2023-03-08 07:06:25,822 [shard 3] view - Error executing build step for base lwt_load_ks.table1: data_dictionary::no_such_column_family (Can't find a column family with UUID bb19a630-bd7f-11ed-a9b6-db3566b700c2)
ERROR 2023-03-08 07:06:26,830 [shard 1] view - Error applying view update to 127.0.75.3 (view: lwt_load_ks.table1_v_idx_index, base token: -1871298252625020920, view token: -1871298252625020920): data_dictionary::no_such_column_family (Can't find a column family with UUID bb19a630-bd7f-11ed-a9b6-db3566b700c2)
WARN 2023-03-08 07:06:26,831 [shard 1] view - Error executing build step for base lwt_load_ks.table1: data_dictionary::no_such_column_family (Can't find a column family with UUID bb19a630-bd7f-11ed-a9b6-db3566b700c2)
ERROR 2023-03-08 07:06:26,838 [shard 0] view - Error applying view update to 127.0.75.3 (view: lwt_load_ks.table1_v_idx_index, base token: -4358577383040064436, view token: -4358577383040064436): data_dictionary::no_such_column_family (Can't find a column family with UUID bb19a630-bd7f-11ed-a9b6-db3566b700c2)
WARN 2023-03-08 07:06:26,840 [shard 0] view - Error executing build step for base lwt_load_ks.table1: data_dictionary::no_such_column_family (Can't find a column family with UUID bb19a630-bd7f-11ed-a9b6-db3566b700c2)
ERROR 2023-03-08 07:06:26,857 [shard 3] view - Error applying view update to 127.0.75.3 (view: lwt_load_ks.table1_v_idx_index, base token: 3345660330269217793, view token: 3345660330269217793): data_dictionary::no_such_column_family (Can't find a column family with UUID bb19a630-bd7f-11ed-a9b6-db3566b700c2)
WARN 2023-03-08 07:06:26,858 [shard 3] view - Error executing build step for base lwt_load_ks.table1: data_dictionary::no_such_column_family (Can't find a column family with UUID bb19a630-bd7f-11ed-a9b6-db3566b700c2)
INFO 2023-03-08 07:06:28,777 [shard 2] compaction - [Compact system.truncated bf5f8d40-bd7f-11ed-b5a4-db3366b700c2] Compacting [/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-25jb799a/test/node3/data/system/truncated-38c19fd0fb863310a4b70d0cc66628aa/me-10-big-Data.db:level=0:origin=memtable,/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-25jb799a/test/node3/data/system/truncated-38c19fd0fb863310a4b70d0cc66628aa/mc-6-big-Data.db:level=0:origin=memtable]
INFO 2023-03-08 07:06:28,817 [shard 3] compaction_manager - Stopping 1 tasks for 0 ongoing compactions for table lwt_load_ks.table1_v_idx_index due to table removal
ERROR 2023-03-08 07:06:28,867 [shard 0] view - Error applying view update to 127.0.75.3 (view: lwt_load_ks.table1_v_idx_index, base token: -4358577383040064436, view token: -4358577383040064436): data_dictionary::no_such_column_family (Can't find a column family with UUID bb19a630-bd7f-11ed-a9b6-db3566b700c2)
WARN 2023-03-08 07:06:28,869 [shard 0] view - Error executing build step for base lwt_load_ks.table1: data_dictionary::no_such_column_family (Can't find a column family with UUID bb19a630-bd7f-11ed-a9b6-db3566b700c2)
ERROR 2023-03-08 07:06:28,880 [shard 3] view - Error applying view update to 127.0.75.3 (view: lwt_load_ks.table1_v_idx_index, base token: 3345660330269217793, view token: 3345660330269217793): data_dictionary::no_such_column_family (Can't find a column family with UUID bb19a630-bd7f-11ed-a9b6-db3566b700c2)
WARN 2023-03-08 07:06:28,880 [shard 3] view - Error executing build step for base lwt_load_ks.table1: data_dictionary::no_such_column_family (Can't find a column family with UUID bb19a630-bd7f-11ed-a9b6-db3566b700c2)
WARN 2023-03-08 07:06:28,920 [shard 0] sstable - Could not remove table directory "/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-25jb799a/test/node3/data/lwt_load_ks/table1_v_idx_index-bb19a630bd7f11eda9b6db3566b700c2/snapshots": std::filesystem::__cxx11::filesystem_error (error system:39, filesystem error: remove failed: Directory not empty [/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-25jb799a/test/node3/data/lwt_load_ks/table1_v_idx_index-bb19a630bd7f11eda9b6db3566b700c2/snapshots]). Ignored.
INFO 2023-03-08 07:06:28,925 [shard 0] view - Stopping to build view lwt_load_ks.table1_v_idx_index
INFO 2023-03-08 07:06:28,927 [shard 0] query_processor - Column definitions for lwt_load_ks.table1 changed, invalidating related prepared statements
INFO 2023-03-08 07:06:28,932 [shard 3] query_processor - Column definitions for lwt_load_ks.table1 changed, invalidating related prepared statements
INFO 2023-03-08 07:06:28,932 [shard 2] query_processor - Column definitions for lwt_load_ks.table1 changed, invalidating related prepared statements
INFO 2023-03-08 07:06:29,010 [shard 1] query_processor - Column definitions for lwt_load_ks.table1 changed, invalidating related prepared statements
INFO 2023-03-08 07:06:29,053 [shard 2] compaction - [Compact system.truncated bf5f8d40-bd7f-11ed-b5a4-db3366b700c2] Compacted 2 sstables to [/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-25jb799a/test/node3/data/system/truncated-38c19fd0fb863310a4b70d0cc66628aa/me-22-big-Data.db:level=0]. 131kB to 94kB (~71% of original) in 256ms = 367kB/s. ~256 total partitions merged to 5.
INFO 2023-03-08 07:06:29,056 [shard 2] compaction - [Compact system.truncated bf8d05e0-bd7f-11ed-b5a4-db3366b700c2] Compacting [/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-25jb799a/test/node3/data/system/truncated-38c19fd0fb863310a4b70d0cc66628aa/me-14-big-Data.db:level=0:origin=memtable,/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-25jb799a/test/node3/data/system/truncated-38c19fd0fb863310a4b70d0cc66628aa/me-18-big-Data.db:level=0:origin=memtable,/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-25jb799a/test/node3/data/system/truncated-38c19fd0fb863310a4b70d0cc66628aa/me-22-big-Data.db:level=0:origin=compaction]
ERROR 2023-03-08 07:06:29,273 [shard 1] view - Schema version used for view updates (dd78b246-57da-380f-bf1f-566f055419e5) does not match the current base schema version of the view (e5278e1b-64e6-38f1-8900-c578796bb1eb) for view lwt_load_ks.table1_v_idx_index of lwt_load_ks.table1, at: 0xcf55950 0x7f0851d47f7c 0x7f0851d47cd7 0x7f0851d4a029 0x7f0851d4aa9f 0x7f0850c5822b 0x7f0850c57b48 0xff81d1f 0xdb3c1e7 0xfffbf5d 0xfffa990 0xfff9ebb 0xffff91d 0xfff90a2 0xfff8ed9 0xfff8e97 0xfff8e14 0xfff8a8b 0xfff850e 0xfff8283 0xffec2b3 0xff92f7b 0xffd7d88 0x7f0850b5560d 0x7f0851346827
--------
seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::noncopyable_function<seastar::future<void> ()>, seastar::future<void>::then_impl_nrvo<seastar::noncopyable_function<seastar::future<void> ()>, seastar::future<void> >(seastar::noncopyable_function<seastar::future<void> ()>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> ()>&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>
--------
seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>, seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)> >(seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>
--------
seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>, seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)> >(seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>
--------
seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>, seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)> >(seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>
Aborting on shard 1.
Backtrace:
0xcf55950
0x7f085100b45c
0x7f085100b0b3
0x7f0850e6ad09
0x7f0850e996c5
0x7f0850f7e5b9
0x7f0850f7e7ee
0x7f0850f7e63a
/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.ccm/scylla-repository/cee597560a5854befdb7ac17303d828bc0a1a334/libreloc/libc.so.6+0x3cb1f
/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.ccm/scylla-repository/cee597560a5854befdb7ac17303d828bc0a1a334/libreloc/libc.so.6+0x8ce5b
/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.ccm/scylla-repository/cee597560a5854befdb7ac17303d828bc0a1a334/libreloc/libc.so.6+0x3ca75
/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.ccm/scylla-repository/cee597560a5854befdb7ac17303d828bc0a1a334/libreloc/libc.so.6+0x267fb
0x7f0850c57b52
0xff81d1f
0xdb3c1e7
0xfffbf5d
0xfffa990
0xfff9ebb
0xffff91d
0xfff90a2
0xfff8ed9
0xfff8e97
0xfff8e14
0xfff8a8b
0xfff850e
0xfff8283
0xffec2b3
0xff92f7b
0xffd7d88
0x7f0850b5560d
0x7f0851346827Decoded:
operator() at ./db/view/view.cc:1448
(inlined by) db::view::view_updates boost::range_detail::default_constructible_unary_fn_wrapper<db::view::make_view_update_builder(replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_33, db::view::view_updates>::operator()<db::view::view_and_base>(db::view::view_and_base&) const at /usr/include/boost/range/detail/default_constructible_unary_fn.hpp:62
(inlined by) boost::iterators::transform_iterator<boost::range_detail::default_constructible_unary_fn_wrapper<db::view::make_view_update_builder(replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_33, db::view::view_updates>, __gnu_cxx::__normal_iterator<db::view::view_and_base*, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > >, boost::use_default, boost::use_default>::dereference() const at /usr/include/boost/iterator/transform_iterator.hpp:126
(inlined by) boost::iterators::transform_iterator<boost::range_detail::default_constructible_unary_fn_wrapper<db::view::make_view_update_builder(replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_33, db::view::view_updates>, __gnu_cxx::__normal_iterator<db::view::view_and_base*, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > >, boost::use_default, boost::use_default>::reference boost::iterators::iterator_core_access::dereference<boost::iterators::transform_iterator<boost::range_detail::default_constructible_unary_fn_wrapper<db::view::make_view_update_builder(replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_33, db::view::view_updates>, __gnu_cxx::__normal_iterator<db::view::view_and_base*, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > >, boost::use_default, boost::use_default> >(boost::iterators::transform_iterator<boost::range_detail::default_constructible_unary_fn_wrapper<db::view::make_view_update_builder(replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_33, db::view::view_updates>, __gnu_cxx::__normal_iterator<db::view::view_and_base*, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > >, boost::use_default, boost::use_default> const&) at /usr/include/boost/iterator/iterator_facade.hpp:550
(inlined by) boost::iterators::detail::iterator_facade_base<boost::iterators::transform_iterator<boost::range_detail::default_constructible_unary_fn_wrapper<db::view::make_view_update_builder(replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_33, db::view::view_updates>, __gnu_cxx::__normal_iterator<db::view::view_and_base*, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > >, boost::use_default, boost::use_default>, db::view::view_updates, boost::iterators::random_access_traversal_tag, db::view::view_updates, long, false, false>::operator*() const at /usr/include/boost/iterator/iterator_facade.hpp:656
(inlined by) void std::vector<db::view::view_updates, std::allocator<db::view::view_updates> >::_M_range_initialize<boost::iterators::transform_iterator<boost::range_detail::default_constructible_unary_fn_wrapper<db::view::make_view_update_builder(replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_33, db::view::view_updates>, __gnu_cxx::__normal_iterator<db::view::view_and_base*, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > >, boost::use_default, boost::use_default> >(boost::iterators::transform_iterator<boost::range_detail::default_constructible_unary_fn_wrapper<db::view::make_view_update_builder(replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_33, db::view::view_updates>, __gnu_cxx::__normal_iterator<db::view::view_and_base*, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > >, boost::use_default, boost::use_default>, boost::iterators::transform_iterator<boost::range_detail::default_constructible_unary_fn_wrapper<db::view::make_view_update_builder(replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_33, db::view::view_updates>, __gnu_cxx::__normal_iterator<db::view::view_and_base*, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > >, boost::use_default, boost::use_default>, std::input_iterator_tag) at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/stl_vector.h:1668
(inlined by) vector<boost::iterators::transform_iterator<boost::range_detail::default_constructible_unary_fn_wrapper<(lambda at db/view/view.cc:1446:107), db::view::view_updates>, __gnu_cxx::__normal_iterator<db::view::view_and_base *, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > >, boost::use_default, boost::use_default>, void> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/stl_vector.h:706
(inlined by) std::vector<db::view::view_updates, std::allocator<db::view::view_updates> > boost::copy_range<std::vector<db::view::view_updates, std::allocator<db::view::view_updates> >, boost::range_detail::transformed_range<db::view::make_view_update_builder(replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_33, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > > >(boost::range_detail::transformed_range<db::view::make_view_update_builder(replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_33, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > > const&) at /usr/include/boost/range/iterator_range_core.hpp:842
(inlined by) db::view::make_view_update_builder(replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >) at ./db/view/view.cc:1446
replica::table::populate_views(std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >, dht::token, flat_mutation_reader_v2&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >) at ./replica/table.cc:2113
db::view::view_builder::consumer::flush_fragments() at ./db/view/view.cc:2389
db::view::view_builder::consumer::consume_end_of_partition() at ./db/view/view.cc:2402
seastar::bool_class<seastar::stop_iteration_tag> compact_mutation_state<(compact_for_sstables)0>::consume_end_of_partition<db::view::view_builder::consumer, noop_compacted_fragments_consumer>(db::view::view_builder::consumer&, noop_compacted_fragments_consumer&) at ././mutation/mutation_compactor.hh:469
operator() at ././readers/flat_mutation_reader_v2.hh:291
(inlined by) seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > seastar::futurize<seastar::bool_class<seastar::stop_iteration_tag> >::invoke<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >::consume(partition_end&&)::{lambda()#1}>(flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >::consume(partition_end&&)::{lambda()#1}&&) at ././seastar/include/seastar/core/future.hh:2147
auto seastar::futurize_invoke<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >::consume(partition_end&&)::{lambda()#1}>(flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >::consume(partition_end&&)::{lambda()#1}&&) at ././seastar/include/seastar/core/future.hh:2176
(inlined by) flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >::consume(partition_end&&) at ././readers/flat_mutation_reader_v2.hh:290
(inlined by) decltype(auto) mutation_fragment_v2::consume<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >(flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >&) && at ././mutation/mutation_fragment_v2.hh:281
flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >::operator()(mutation_fragment_v2&&) at ././readers/flat_mutation_reader_v2.hh:270
(inlined by) seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > std::__invoke_impl<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >, flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >&, mutation_fragment_v2>(std::__invoke_other, flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >&, mutation_fragment_v2&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:61
(inlined by) std::__invoke_result<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >&, mutation_fragment_v2>::type std::__invoke<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >&, mutation_fragment_v2>(flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >&, mutation_fragment_v2&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:96
std::result_of<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >& (mutation_fragment_v2&&)>::type std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >::operator()<mutation_fragment_v2>(mutation_fragment_v2&&) const at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/refwrap.h:358
operator() at ././readers/flat_mutation_reader_v2.hh:251
(inlined by) seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > seastar::futurize<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > >::invoke<flat_mutation_reader_v2::impl::consume_pausable_in_thread<std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter>(std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter)::{lambda()#1}>(flat_mutation_reader_v2::impl::consume_pausable_in_thread<std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter>(std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter)::{lambda()#1}&&) at ././seastar/include/seastar/core/future.hh:2145
auto seastar::futurize_invoke<flat_mutation_reader_v2::impl::consume_pausable_in_thread<std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter>(std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter)::{lambda()#1}>(flat_mutation_reader_v2::impl::consume_pausable_in_thread<std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter>(std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter)::{lambda()#1}&&) at ././seastar/include/seastar/core/future.hh:2176
(inlined by) void flat_mutation_reader_v2::impl::consume_pausable_in_thread<std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter>(std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter) at ././readers/flat_mutation_reader_v2.hh:250
auto flat_mutation_reader_v2::impl::consume_in_thread<compact_for_query_v2<db::view::view_builder::consumer>, flat_mutation_reader_v2::no_filter>(compact_for_query_v2<db::view::view_builder::consumer>, flat_mutation_reader_v2::no_filter) at ././readers/flat_mutation_reader_v2.hh:337
auto flat_mutation_reader_v2::consume_in_thread<compact_for_query_v2<db::view::view_builder::consumer>, flat_mutation_reader_v2::no_filter>(compact_for_query_v2<db::view::view_builder::consumer>, flat_mutation_reader_v2::no_filter) at ././readers/flat_mutation_reader_v2.hh:477
auto flat_mutation_reader_v2::consume_in_thread<compact_for_query_v2<db::view::view_builder::consumer> >(compact_for_query_v2<db::view::view_builder::consumer>) at ././readers/flat_mutation_reader_v2.hh:483
db::view::view_builder::execute(db::view::view_builder::build_step&, exponential_backoff_retry) at ./db/view/view.cc:2439
operator() at ./db/view/view.cc:2207
(inlined by) void std::__invoke_impl<void, db::view::view_builder::do_build_step()::$_26>(std::__invoke_other, db::view::view_builder::do_build_step()::$_26&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:61
(inlined by) std::__invoke_result<db::view::view_builder::do_build_step()::$_26>::type std::__invoke<db::view::view_builder::do_build_step()::$_26>(db::view::view_builder::do_build_step()::$_26&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:96
(inlined by) decltype(auto) std::__apply_impl<db::view::view_builder::do_build_step()::$_26, std::tuple<>>(db::view::view_builder::do_build_step()::$_26&&, std::tuple<>&&, std::integer_sequence<unsigned long>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/tuple:1852
(inlined by) decltype(auto) std::apply<db::view::view_builder::do_build_step()::$_26, std::tuple<> >(db::view::view_builder::do_build_step()::$_26&&, std::tuple<>&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/tuple:1863
(inlined by) seastar::future<void> seastar::futurize<void>::apply<db::view::view_builder::do_build_step()::$_26>(db::view::view_builder::do_build_step()::$_26&&, std::tuple<>&&) at ././seastar/include/seastar/core/future.hh:2109
(inlined by) operator() at ././seastar/include/seastar/core/thread.hh:258
(inlined by) seastar::noncopyable_function<void ()>::direct_vtable_for<seastar::async<db::view::view_builder::do_build_step()::$_26>(seastar::thread_attributes, db::view::view_builder::do_build_step()::$_26&&)::{lambda()#1}>::call(seastar::noncopyable_function<void ()> const*) at ././seastar/include/seastar/util/noncopyable_function.hh:124Reloc package here: http://downloads.scylladb.com/unstable/scylla/master/relocatable/2023-03-08T03:11:57Z/scylla-debug-unstripped-5.3.0~dev-0.20230308.cee597560a58.x86_64.tar.gz Core dump here: https://jenkins.scylladb.com/view/master/job/scylla-master/job/dtest-daily-debug/159/artifact/logs-full.debug.020/1678259330093_lwt_schema_modification_test.py%3A%3ATestLWTSchemaModification%3A%3Atest_index_drop_add/node3-reactor-1.2241.1678259189.core.gz
DoronArazii
commented
1 year ago @eliransin / @nyh there's a high volume around CI-Stability issues, can you please prioritize this?
eliransin
commented
1 year ago @DoronArazii , prioritizing it means making it P1, is this what you meant?
DoronArazii
commented
1 year ago Yes @eliransin , and also assigning someone to handle this. thanks
bhalevy
commented
1 year ago INFO 2023-04-27 19:57:09,658 [shard 0] schema_tables - Schema version changed to 971c25b4-c01c-3a6d-aa0d-4abdcb316831
INFO 2023-04-27 19:57:11,999 [shard 0] view - Building view lwt_load_ks.table1_v_idx_index, starting at token minimum token
INFO 2023-04-27 19:57:12,165 [shard 0] migration_manager - Update table 'lwt_load_ks.table1' From org.apache.cassandra.config.CFMetaData@0x6190003d6880[cfId=7dbbcb00-e535-11ed-99ba-ef90a5a861c8,ksName=lwt_load_ks,cfName=table1,cfType=Standard,comparator=org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.UTF8Type),comment=,readRepairChance=0,dcLocalReadRepairChance=0,tombstoneGcOptions={"mode":"timeout","propagation_delay_in_seconds":"3600"},gcGraceSeconds=864000,keyValidator=org.apache.cassandra.db.marshal.Int32Type,minCompactionThreshold=4,maxCompactionThreshold=32,columnMetadata=[ColumnDefinition{name=pk, type=org.apache.cassandra.db.marshal.Int32Type, kind=PARTITION_KEY, componentIndex=0, droppedAt=-9223372036854775808}, ColumnDefinition{name=int_col, type=org.apache.cassandra.db.marshal.Int32Type, kind=REGULAR, componentIndex=null, droppedAt=-9223372036854775808}, ColumnDefinition{name=v, type=org.apache.cassandra.db.marshal.Int32Type, kind=REGULAR, componentIndex=null, droppedAt=-9223372036854775808}],compactionStrategyClass=class org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={enabled=true},compressionParameters={sstable_compression=org.apache.cassandra.io.compress.LZ4Compressor},bloomFilterFpChance=0.01,memtableFlushPeriod=0,caching={"keys":"ALL","rows_per_partition":"ALL"},cdc={},defaultTimeToLive=0,minIndexInterval=128,maxIndexInterval=2048,speculativeRetry=99.0PERCENTILE,triggers=[],isDense=false,version=d38a8099-c0e5-3f55-a460-6aa9406d5cc3,droppedColumns={},collections={},indices={table1_v_idx : 65b7e77e-3dbc-3af2-89da-9e355316e0a9}] To org.apache.cassandra.config.CFMetaData@0x619000113f80[cfId=7dbbcb00-e535-11ed-99ba-ef90a5a861c8,ksName=lwt_load_ks,cfName=table1,cfType=Standard,comparator=org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.UTF8Type),comment=,readRepairChance=0,dcLocalReadRepairChance=0,tombstoneGcOptions={"mode":"timeout","propagation_delay_in_seconds":"3600"},gcGraceSeconds=864000,keyValidator=org.apache.cassandra.db.marshal.Int32Type,minCompactionThreshold=4,maxCompactionThreshold=32,columnMetadata=[ColumnDefinition{name=pk, type=org.apache.cassandra.db.marshal.Int32Type, kind=PARTITION_KEY, componentIndex=0, droppedAt=-9223372036854775808}, ColumnDefinition{name=int_col, type=org.apache.cassandra.db.marshal.Int32Type, kind=REGULAR, componentIndex=null, droppedAt=-9223372036854775808}, ColumnDefinition{name=v, type=org.apache.cassandra.db.marshal.Int32Type, kind=REGULAR, componentIndex=null, droppedAt=-9223372036854775808}],compactionStrategyClass=class org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={enabled=true},compressionParameters={sstable_compression=org.apache.cassandra.io.compress.LZ4Compressor},bloomFilterFpChance=0.01,memtableFlushPeriod=0,caching={"keys":"ALL","rows_per_partition":"ALL"},cdc={},defaultTimeToLive=0,minIndexInterval=128,maxIndexInterval=2048,speculativeRetry=99.0PERCENTILE,triggers=[],isDense=false,version=b33f2240-e535-11ed-99ba-ef90a5a861c8,droppedColumns={},collections={},indices={}]
INFO 2023-04-27 19:57:12,324 [shard 0] schema_tables - Altering lwt_load_ks.table1 id=7dbbcb00-e535-11ed-99ba-ef90a5a861c8 version=dfefdb47-3691-3dc7-affe-e4ec6c2b7ebf
INFO 2023-04-27 19:57:12,328 [shard 0] schema_tables - Dropping lwt_load_ks.table1_v_idx_index id=b1461160-e535-11ed-80f6-573e062d77b9 version=7365b1d5-6869-3f17-88e3-4c5665887ffc
INFO 2023-04-27 19:57:12,330 [shard 0] database - Dropping lwt_load_ks.table1_v_idx_index with auto-snapshot
ERROR 2023-04-27 19:57:12,333 [shard 1] view - Error applying view update to 127.0.33.4 (view: lwt_load_ks.table1_v_idx_index, base token: -6258651536561445896, view token: -6258651536561445896): data_dictionary::no_such_column_family (Can't find a column family with UUID b1461160-e535-11ed-80f6-573e062d77b9)
ERROR 2023-04-27 19:57:12,334 [shard 2] view - Error applying view update to 127.0.33.4 (view: lwt_load_ks.table1_v_idx_index, base token: -7523144313183920711, view token: -7523144313183920711): data_dictionary::no_such_column_family (Can't find a column family with UUID b1461160-e535-11ed-80f6-573e062d77b9)
WARN 2023-04-27 19:57:12,334 [shard 1] view - Error executing build step for base lwt_load_ks.table1: data_dictionary::no_such_column_family (Can't find a column family with UUID b1461160-e535-11ed-80f6-573e062d77b9)
WARN 2023-04-27 19:57:12,335 [shard 2] view - Error executing build step for base lwt_load_ks.table1: data_dictionary::no_such_column_family (Can't find a column family with UUID b1461160-e535-11ed-80f6-573e062d77b9)
ERROR 2023-04-27 19:57:12,335 [shard 0] view - Error applying view update to 127.0.33.4 (view: lwt_load_ks.table1_v_idx_index, base token: -8777160030288591033, view token: -8777160030288591033): data_dictionary::no_such_column_family (Can't find a column family with UUID b1461160-e535-11ed-80f6-573e062d77b9)
INFO 2023-04-27 19:57:12,335 [shard 0] database - Truncating lwt_load_ks.table1_v_idx_index with auto-snapshot
WARN 2023-04-27 19:57:12,337 [shard 0] view - Error executing build step for base lwt_load_ks.table1: data_dictionary::no_such_column_family (Can't find a column family with UUID b1461160-e535-11ed-80f6-573e062d77b9)
ERROR 2023-04-27 19:57:12,337 [shard 3] view - Error applying view update to 127.0.33.4 (view: lwt_load_ks.table1_v_idx_index, base token: -7035101017347541615, view token: -7035101017347541615): data_dictionary::no_such_column_family (Can't find a column family with UUID b1461160-e535-11ed-80f6-573e062d77b9)
WARN 2023-04-27 19:57:12,340 [shard 3] view - Error executing build step for base lwt_load_ks.table1: data_dictionary::no_such_column_family (Can't find a column family with UUID b1461160-e535-11ed-80f6-573e062d77b9)
INFO 2023-04-27 19:57:13,312 [shard 1] compaction - [Compact system.truncated b3edb1c0-e535-11ed-be49-ef8ea5a861c8] Compacting [/jenkins/workspace/scylla-5.2/dtest-daily-debug/scylla/.dtest/dtest-j_7bjner/test/node4/data/system/truncated-38c19fd0fb863310a4b70d0cc66628aa/me-5-big-Data.db:level=0:origin=memtable,/jenkins/workspace/scylla-5.2/dtest-daily-debug/scylla/.dtest/dtest-j_7bjner/test/node4/data/system/truncated-38c19fd0fb863310a4b70d0cc66628aa/me-9-big-Data.db:level=0:origin=memtable]
INFO 2023-04-27 19:57:13,316 [shard 1] compaction_manager - Stopping 1 tasks for 0 ongoing compactions for table lwt_load_ks.table1_v_idx_index due to table removal
WARN 2023-04-27 19:57:13,327 [shard 0] sstable - Could not remove table directory "/jenkins/workspace/scylla-5.2/dtest-daily-debug/scylla/.dtest/dtest-j_7bjner/test/node4/data/lwt_load_ks/table1_v_idx_index-b1461160e53511ed80f6573e062d77b9/snapshots": std::filesystem::__cxx11::filesystem_error (error system:39, filesystem error: remove failed: Directory not empty [/jenkins/workspace/scylla-5.2/dtest-daily-debug/scylla/.dtest/dtest-j_7bjner/test/node4/data/lwt_load_ks/table1_v_idx_index-b1461160e53511ed80f6573e062d77b9/snapshots]). Ignored.
INFO 2023-04-27 19:57:13,346 [shard 0] view - Stopping to build view lwt_load_ks.table1_v_idx_index
INFO 2023-04-27 19:57:13,357 [shard 3] query_processor - Column definitions for lwt_load_ks.table1 changed, invalidating related prepared statements
INFO 2023-04-27 19:57:13,381 [shard 2] query_processor - Column definitions for lwt_load_ks.table1 changed, invalidating related prepared statements
INFO 2023-04-27 19:57:13,385 [shard 1] query_processor - Column definitions for lwt_load_ks.table1 changed, invalidating related prepared statements
INFO 2023-04-27 19:57:13,386 [shard 0] query_processor - Column definitions for lwt_load_ks.table1 changed, invalidating related prepared statements
ERROR 2023-04-27 19:57:13,396 [shard 1] view - Schema version used for view updates (dfefdb47-3691-3dc7-affe-e4ec6c2b7ebf) does not match the current base schema version of the view (d38a8099-c0e5-3f55-a460-6aa9406d5cc3) for view lwt_load_ks.table1_v_idx_index of lwt_load_ks.table1, at: 0xdd11850 0x141f9175 0x141f8ecd 0x141fb2e3 0x141fbd6f 0x13265bed 0x132654e4 0x10c7cddf 0xe8db7f8 0x10cf626d 0x10cf4ca0 0x10cf40f7 0x10cf9bfd 0x10cf32e2 0x10cf3119 0x10cf30d7 0x10cf3054 0x10cf2ccb 0x10cf274e 0x10cf24c3 0x10ce6533 0x10c8dce6 0x10cd22b7 0x138a8329
--------
seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::noncopyable_function<seastar::future<void> ()>, seastar::future<void>::then_impl_nrvo<seastar::noncopyable_function<seastar::future<void> ()>, seastar::future<void> >(seastar::noncopyable_function<seastar::future<void> ()>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> ()>&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>
--------
seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>, seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)> >(seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>
--------
seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>, seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)> >(seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>
--------
seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>, seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)> >(seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>
Aborting on shard 1.
Backtrace:
0xdd11850
0x1354c705
0x1354c33c
0x133a69a4
0x133d5d7e
0x134bf50c
0x134bf76d
0x134bf59a
/jenkins/workspace/scylla-5.2/dtest-daily-debug/scylla/.ccm/scylla-repository/429b696bbc1bf46942eea459e5e05d1941d7e94b/libreloc/libc.so.6+0x3cb4f
/jenkins/workspace/scylla-5.2/dtest-daily-debug/scylla/.ccm/scylla-repository/429b696bbc1bf46942eea459e5e05d1941d7e94b/libreloc/libc.so.6+0x8ce7b
/jenkins/workspace/scylla-5.2/dtest-daily-debug/scylla/.ccm/scylla-repository/429b696bbc1bf46942eea459e5e05d1941d7e94b/libreloc/libc.so.6+0x3caa5
/jenkins/workspace/scylla-5.2/dtest-daily-debug/scylla/.ccm/scylla-repository/429b696bbc1bf46942eea459e5e05d1941d7e94b/libreloc/libc.so.6+0x267fb
0x132654ee
0x10c7cddf
0xe8db7f8
0x10cf626d
0x10cf4ca0
0x10cf40f7
0x10cf9bfd
0x10cf32e2
0x10cf3119
0x10cf30d7
0x10cf3054
0x10cf2ccb
0x10cf274e
0x10cf24c3
0x10ce6533
0x10c8dce6
0x10cd22b7
0x138a8329Decoded:
[Backtrace #0]
__interceptor_backtrace.part.0 at ??:?
void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) at ./build/debug/seastar/./seastar/include/seastar/util/backtrace.hh:59
seastar::backtrace_buffer::append_backtrace() at ./build/debug/seastar/./seastar/src/core/reactor.cc:783
seastar::print_with_backtrace(seastar::backtrace_buffer&, bool) at ./build/debug/seastar/./seastar/src/core/reactor.cc:813
seastar::print_with_backtrace(char const*, bool) at ./build/debug/seastar/./seastar/src/core/reactor.cc:825
seastar::sigabrt_action() at ./build/debug/seastar/./seastar/src/core/reactor.cc:3864
operator() at ./build/debug/seastar/./seastar/src/core/reactor.cc:3840
__invoke at ./build/debug/seastar/./seastar/src/core/reactor.cc:3836
/data/scylla-s3-reloc.cache/by-build-id/c4e44bb157560934d05cb561bea8e745603c4af6/extracted/scylla/libreloc/libc.so.6: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=765237b0355c030ff41d969eedcb87bfccb43595, for GNU/Linux 3.2.0, not stripped
__GI___sigaction at :?
__pthread_kill_implementation at ??:?
__GI_raise at :?
__GI_abort at :?
seastar::on_internal_error(seastar::logger&, std::basic_string_view<char, std::char_traits<char> >) at ./build/debug/seastar/./seastar/src/core/on_internal_error.cc:43
operator() at ./db/view/view.cc:1448
(inlined by) db::view::view_updates boost::range_detail::default_constructible_unary_fn_wrapper<db::view::make_view_update_builder(replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_33, db::view::view_updates>::operator()<db::view::view_and_base>(db::view::view_and_base&) const at /usr/include/boost/range/detail/default_constructible_unary_fn.hpp:62
(inlined by) boost::iterators::transform_iterator<boost::range_detail::default_constructible_unary_fn_wrapper<db::view::make_view_update_builder(replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_33, db::view::view_updates>, __gnu_cxx::__normal_iterator<db::view::view_and_base*, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > >, boost::use_default, boost::use_default>::dereference() const at /usr/include/boost/iterator/transform_iterator.hpp:126
(inlined by) boost::iterators::transform_iterator<boost::range_detail::default_constructible_unary_fn_wrapper<db::view::make_view_update_builder(replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_33, db::view::view_updates>, __gnu_cxx::__normal_iterator<db::view::view_and_base*, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > >, boost::use_default, boost::use_default>::reference boost::iterators::iterator_core_access::dereference<boost::iterators::transform_iterator<boost::range_detail::default_constructible_unary_fn_wrapper<db::view::make_view_update_builder(replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_33, db::view::view_updates>, __gnu_cxx::__normal_iterator<db::view::view_and_base*, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > >, boost::use_default, boost::use_default> >(boost::iterators::transform_iterator<boost::range_detail::default_constructible_unary_fn_wrapper<db::view::make_view_update_builder(replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_33, db::view::view_updates>, __gnu_cxx::__normal_iterator<db::view::view_and_base*, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > >, boost::use_default, boost::use_default> const&) at /usr/include/boost/iterator/iterator_facade.hpp:550
(inlined by) boost::iterators::detail::iterator_facade_base<boost::iterators::transform_iterator<boost::range_detail::default_constructible_unary_fn_wrapper<db::view::make_view_update_builder(replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_33, db::view::view_updates>, __gnu_cxx::__normal_iterator<db::view::view_and_base*, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > >, boost::use_default, boost::use_default>, db::view::view_updates, boost::iterators::random_access_traversal_tag, db::view::view_updates, long, false, false>::operator*() const at /usr/include/boost/iterator/iterator_facade.hpp:656
(inlined by) void std::vector<db::view::view_updates, std::allocator<db::view::view_updates> >::_M_range_initialize<boost::iterators::transform_iterator<boost::range_detail::default_constructible_unary_fn_wrapper<db::view::make_view_update_builder(replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_33, db::view::view_updates>, __gnu_cxx::__normal_iterator<db::view::view_and_base*, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > >, boost::use_default, boost::use_default> >(boost::iterators::transform_iterator<boost::range_detail::default_constructible_unary_fn_wrapper<db::view::make_view_update_builder(replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_33, db::view::view_updates>, __gnu_cxx::__normal_iterator<db::view::view_and_base*, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > >, boost::use_default, boost::use_default>, boost::iterators::transform_iterator<boost::range_detail::default_constructible_unary_fn_wrapper<db::view::make_view_update_builder(replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_33, db::view::view_updates>, __gnu_cxx::__normal_iterator<db::view::view_and_base*, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > >, boost::use_default, boost::use_default>, std::input_iterator_tag) at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/stl_vector.h:1668
(inlined by) vector<boost::iterators::transform_iterator<boost::range_detail::default_constructible_unary_fn_wrapper<(lambda at db/view/view.cc:1446:107), db::view::view_updates>, __gnu_cxx::__normal_iterator<db::view::view_and_base *, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > >, boost::use_default, boost::use_default>, void> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/stl_vector.h:706
(inlined by) std::vector<db::view::view_updates, std::allocator<db::view::view_updates> > boost::copy_range<std::vector<db::view::view_updates, std::allocator<db::view::view_updates> >, boost::range_detail::transformed_range<db::view::make_view_update_builder(replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_33, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > > >(boost::range_detail::transformed_range<db::view::make_view_update_builder(replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_33, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > > const&) at /usr/include/boost/range/iterator_range_core.hpp:842
(inlined by) db::view::make_view_update_builder(replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >) at ./db/view/view.cc:1446
replica::table::populate_views(std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >, dht::token, flat_mutation_reader_v2&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >) at ./replica/table.cc:2045
db::view::view_builder::consumer::flush_fragments() at ./db/view/view.cc:2388
db::view::view_builder::consumer::consume_end_of_partition() at ./db/view/view.cc:2401
seastar::bool_class<seastar::stop_iteration_tag> compact_mutation_state<(compact_for_sstables)0>::consume_end_of_partition<db::view::view_builder::consumer, noop_compacted_fragments_consumer>(db::view::view_builder::consumer&, noop_compacted_fragments_consumer&) at ././mutation_compactor.hh:469
operator() at ././readers/flat_mutation_reader_v2.hh:291
(inlined by) seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > seastar::futurize<seastar::bool_class<seastar::stop_iteration_tag> >::invoke<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >::consume(partition_end&&)::{lambda()#1}>(flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >::consume(partition_end&&)::{lambda()#1}&&) at ././seastar/include/seastar/core/future.hh:2149
auto seastar::futurize_invoke<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >::consume(partition_end&&)::{lambda()#1}>(flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >::consume(partition_end&&)::{lambda()#1}&&) at ././seastar/include/seastar/core/future.hh:2178
(inlined by) flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >::consume(partition_end&&) at ././readers/flat_mutation_reader_v2.hh:290
(inlined by) decltype(auto) mutation_fragment_v2::consume<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >(flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >&) && at ././mutation_fragment_v2.hh:280
flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >::operator()(mutation_fragment_v2&&) at ././readers/flat_mutation_reader_v2.hh:270
(inlined by) seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > std::__invoke_impl<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >, flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >&, mutation_fragment_v2>(std::__invoke_other, flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >&, mutation_fragment_v2&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:61
(inlined by) std::__invoke_result<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >&, mutation_fragment_v2>::type std::__invoke<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >&, mutation_fragment_v2>(flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >&, mutation_fragment_v2&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:96
std::result_of<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >& (mutation_fragment_v2&&)>::type std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >::operator()<mutation_fragment_v2>(mutation_fragment_v2&&) const at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/refwrap.h:358
operator() at ././readers/flat_mutation_reader_v2.hh:251
(inlined by) seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > seastar::futurize<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > >::invoke<flat_mutation_reader_v2::impl::consume_pausable_in_thread<std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter>(std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter)::{lambda()#1}>(flat_mutation_reader_v2::impl::consume_pausable_in_thread<std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter>(std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter)::{lambda()#1}&&) at ././seastar/include/seastar/core/future.hh:2147
auto seastar::futurize_invoke<flat_mutation_reader_v2::impl::consume_pausable_in_thread<std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter>(std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter)::{lambda()#1}>(flat_mutation_reader_v2::impl::consume_pausable_in_thread<std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter>(std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter)::{lambda()#1}&&) at ././seastar/include/seastar/core/future.hh:2178
(inlined by) void flat_mutation_reader_v2::impl::consume_pausable_in_thread<std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter>(std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter) at ././readers/flat_mutation_reader_v2.hh:250
auto flat_mutation_reader_v2::impl::consume_in_thread<compact_for_query_v2<db::view::view_builder::consumer>, flat_mutation_reader_v2::no_filter>(compact_for_query_v2<db::view::view_builder::consumer>, flat_mutation_reader_v2::no_filter) at ././readers/flat_mutation_reader_v2.hh:337
auto flat_mutation_reader_v2::consume_in_thread<compact_for_query_v2<db::view::view_builder::consumer>, flat_mutation_reader_v2::no_filter>(compact_for_query_v2<db::view::view_builder::consumer>, flat_mutation_reader_v2::no_filter) at ././readers/flat_mutation_reader_v2.hh:477
auto flat_mutation_reader_v2::consume_in_thread<compact_for_query_v2<db::view::view_builder::consumer> >(compact_for_query_v2<db::view::view_builder::consumer>) at ././readers/flat_mutation_reader_v2.hh:483
db::view::view_builder::execute(db::view::view_builder::build_step&, exponential_backoff_retry) at ./db/view/view.cc:2438
operator() at ./db/view/view.cc:2206
(inlined by) void std::__invoke_impl<void, db::view::view_builder::do_build_step()::$_26>(std::__invoke_other, db::view::view_builder::do_build_step()::$_26&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:61
(inlined by) std::__invoke_result<db::view::view_builder::do_build_step()::$_26>::type std::__invoke<db::view::view_builder::do_build_step()::$_26>(db::view::view_builder::do_build_step()::$_26&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:96
(inlined by) decltype(auto) std::__apply_impl<db::view::view_builder::do_build_step()::$_26, std::tuple<>>(db::view::view_builder::do_build_step()::$_26&&, std::tuple<>&&, std::integer_sequence<unsigned long>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/tuple:1852
(inlined by) decltype(auto) std::apply<db::view::view_builder::do_build_step()::$_26, std::tuple<> >(db::view::view_builder::do_build_step()::$_26&&, std::tuple<>&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/tuple:1863
(inlined by) seastar::future<void> seastar::futurize<void>::apply<db::view::view_builder::do_build_step()::$_26>(db::view::view_builder::do_build_step()::$_26&&, std::tuple<>&&) at ././seastar/include/seastar/core/future.hh:2111
(inlined by) operator() at ././seastar/include/seastar/core/thread.hh:258
(inlined by) seastar::noncopyable_function<void ()>::direct_vtable_for<seastar::async<db::view::view_builder::do_build_step()::$_26>(seastar::thread_attributes, db::view::view_builder::do_build_step()::$_26&&)::{lambda()#1}>::call(seastar::noncopyable_function<void ()> const*) at ././seastar/include/seastar/util/noncopyable_function.hh:124
seastar::thread_context::main() at ./build/debug/seastar/./seastar/src/core/thread.cc:299@eliransin / @nyh it's a CI-Stability case without any movement. Can you please assign someone to handle
bhalevy
commented
1 year ago Still seeing similar issues. https://jenkins.scylladb.com/view/master/job/scylla-master/job/dtest-daily-debug/198/artifact/logs-full.debug.056/1684389028644_secondary_indexes_test.py%3A%3ATestLocalIndexes%3A%3Atest_drop_local_index_while_building/node1.log
Scylla version 5.3.0~dev-0.20230518.6f4a68175b54 with build-id ff34d927919817aaa8551e8536056fa0040a379f starting ...
...
INFO 2023-05-18 05:49:25,585 [shard 0] view - Building view keyspace1.idx_index, starting at token minimum token
INFO 2023-05-18 05:49:30,280 [shard 0] schema_tables - Altering keyspace1.standard1 id=75355e50-f53f-11ed-822d-50c3a07d061a version=f8ddacc9-b93e-31db-8759-5d5cc6bffb10
INFO 2023-05-18 05:49:30,285 [shard 0] schema_tables - Dropping keyspace1.idx_index id=9a2b7f00-f53f-11ed-8b8d-e981fef795da version=f8a0206c-eba1-3a04-b869-d70e863a5417
INFO 2023-05-18 05:49:30,303 [shard 0] database - Dropping keyspace1.idx_index with auto-snapshot
ERROR 2023-05-18 05:49:30,310 [shard 1] view - Error applying view update to 127.0.23.1 (view: keyspace1.idx_index, base token: -8815261609143315375, view token: -8815261609143315375): data_dictionary::no_such_column_family (Can't find a column family with UUID 9a2b7f00-f53f-11ed-8b8d-e981fef795da)
WARN 2023-05-18 05:49:30,311 [shard 1] view - Error executing build step for base keyspace1.standard1: data_dictionary::no_such_column_family (Can't find a column family with UUID 9a2b7f00-f53f-11ed-8b8d-e981fef795da)
INFO 2023-05-18 05:49:30,317 [shard 0] database - Truncating keyspace1.idx_index with auto-snapshot
ERROR 2023-05-18 05:49:30,345 [shard 0] view - Error applying view update to 127.0.23.1 (view: keyspace1.idx_index, base token: -8389030939554696063, view token: -8389030939554696063): data_dictionary::no_such_column_family (Can't find a column family with UUID 9a2b7f00-f53f-11ed-8b8d-e981fef795da)
WARN 2023-05-18 05:49:30,350 [shard 0] view - Error executing build step for base keyspace1.standard1: data_dictionary::no_such_column_family (Can't find a column family with UUID 9a2b7f00-f53f-11ed-8b8d-e981fef795da)
ERROR 2023-05-18 05:49:31,334 [shard 1] view - Error applying view update to 127.0.23.1 (view: keyspace1.idx_index, base token: -8815261609143315375, view token: -8815261609143315375): data_dictionary::no_such_column_family (Can't find a column family with UUID 9a2b7f00-f53f-11ed-8b8d-e981fef795da)
WARN 2023-05-18 05:49:31,336 [shard 1] view - Error executing build step for base keyspace1.standard1: data_dictionary::no_such_column_family (Can't find a column family with UUID 9a2b7f00-f53f-11ed-8b8d-e981fef795da)
ERROR 2023-05-18 05:49:31,371 [shard 0] view - Error applying view update to 127.0.23.1 (view: keyspace1.idx_index, base token: -8389030939554696063, view token: -8389030939554696063): data_dictionary::no_such_column_family (Can't find a column family with UUID 9a2b7f00-f53f-11ed-8b8d-e981fef795da)
WARN 2023-05-18 05:49:31,373 [shard 0] view - Error executing build step for base keyspace1.standard1: data_dictionary::no_such_column_family (Can't find a column family with UUID 9a2b7f00-f53f-11ed-8b8d-e981fef795da)
INFO 2023-05-18 05:49:33,353 [shard 0] compaction_manager - Stopping 1 tasks for 0 ongoing compactions for table keyspace1.idx_index compaction_group=0/1 due to table removal
INFO 2023-05-18 05:49:33,354 [shard 1] compaction_manager - Stopping 1 tasks for 0 ongoing compactions for table keyspace1.idx_index compaction_group=0/1 due to table removal
INFO 2023-05-18 05:49:33,358 [shard 0] compaction - [Compact system.truncated c3b383e0-f53f-11ed-aced-b7661fffc361] Compacting [/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-5nq5smkl/test/node1/data/system/truncated-38c19fd0fb863310a4b70d0cc66628aa/me-4-big-Data.db:level=0:origin=memtable,/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-5nq5smkl/test/node1/data/system/truncated-38c19fd0fb863310a4b70d0cc66628aa/mc-2-big-Data.db:level=0:origin=memtable]
ERROR 2023-05-18 05:49:33,365 [shard 1] view - Error applying view update to 127.0.23.1 (view: keyspace1.idx_index, base token: -8815261609143315375, view token: -8815261609143315375): data_dictionary::no_such_column_family (Can't find a column family with UUID 9a2b7f00-f53f-11ed-8b8d-e981fef795da)
WARN 2023-05-18 05:49:33,367 [shard 1] view - Error executing build step for base keyspace1.standard1: data_dictionary::no_such_column_family (Can't find a column family with UUID 9a2b7f00-f53f-11ed-8b8d-e981fef795da)
WARN 2023-05-18 05:49:33,372 [shard 0] sstable - Could not remove table directory "/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-5nq5smkl/test/node1/data/keyspace1/idx_index-9a2b7f00f53f11ed8b8de981fef795da/snapshots": std::filesystem::__cxx11::filesystem_error (error system:39, filesystem error: remove failed: Directory not empty [/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.dtest/dtest-5nq5smkl/test/node1/data/keyspace1/idx_index-9a2b7f00f53f11ed8b8de981fef795da/snapshots]). Ignored.
INFO 2023-05-18 05:49:33,394 [shard 0] view - Stopping to build view keyspace1.idx_index
INFO 2023-05-18 05:49:33,403 [shard 1] query_processor - Column definitions for keyspace1.standard1 changed, invalidating related prepared statements
INFO 2023-05-18 05:49:33,403 [shard 0] query_processor - Column definitions for keyspace1.standard1 changed, invalidating related prepared statements
ERROR 2023-05-18 05:49:33,424 [shard 0] view - Schema version used for view updates (f8ddacc9-b93e-31db-8759-5d5cc6bffb10) does not match the current base schema version of the view (8a6cb391-7c23-34eb-9910-f9871f7dee4c) for view keyspace1.idx_index of keyspace1.standard1, at: 0xd4d2f90 /jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.ccm/scylla-repository/6f4a68175b5440829c506a5dce3e48edd2c662b8/libreloc/libseastar.so+0x554ff0c /jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.ccm/scylla-repository/6f4a68175b5440829c506a5dce3e48edd2c662b8/libreloc/libseastar.so+0x554fc67 /jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.ccm/scylla-repository/6f4a68175b5440829c506a5dce3e48edd2c662b8/libreloc/libseastar.so+0x5551fb9 /jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.ccm/scylla-repository/6f4a68175b5440829c506a5dce3e48edd2c662b8/libreloc/libseastar.so+0x5552a2f /jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.ccm/scylla-repository/6f4a68175b5440829c506a5dce3e48edd2c662b8/libreloc/libseastar.so+0x443a9b0 /jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.ccm/scylla-repository/6f4a68175b5440829c506a5dce3e48edd2c662b8/libreloc/libseastar.so+0x443a55b 0x107025d7 0xe1257ff 0x1077d8b9 0x1077c2d0 0x1077b707 0x1078128d 0x1077a8ee 0x1077a729 0x1077a6e7 0x1077a664 0x1077a2db 0x10779d4a 0x10779aae 0x1076d741 0x1071405e 0x107586e8 /jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.ccm/scylla-repository/6f4a68175b5440829c506a5dce3e48edd2c662b8/libreloc/libseastar.so+0x43385ad /jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.ccm/scylla-repository/6f4a68175b5440829c506a5dce3e48edd2c662b8/libreloc/libseastar.so+0x4b3ff87
--------
seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::noncopyable_function<seastar::future<void> ()>, seastar::future<void>::then_impl_nrvo<seastar::noncopyable_function<seastar::future<void> ()>, seastar::future<void> >(seastar::noncopyable_function<seastar::future<void> ()>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> ()>&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>
--------
seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>, seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)> >(seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>
--------
seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>, seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)> >(seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>
--------
seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>, seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)> >(seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>
Aborting on shard 0.
Backtrace:
0xd4d2f90
/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.ccm/scylla-repository/6f4a68175b5440829c506a5dce3e48edd2c662b8/libreloc/libseastar.so+0x47fa75c
/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.ccm/scylla-repository/6f4a68175b5440829c506a5dce3e48edd2c662b8/libreloc/libseastar.so+0x47fa3b3
/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.ccm/scylla-repository/6f4a68175b5440829c506a5dce3e48edd2c662b8/libreloc/libseastar.so+0x464cc59
/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.ccm/scylla-repository/6f4a68175b5440829c506a5dce3e48edd2c662b8/libreloc/libseastar.so+0x467c495
/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.ccm/scylla-repository/6f4a68175b5440829c506a5dce3e48edd2c662b8/libreloc/libseastar.so+0x476d849
/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.ccm/scylla-repository/6f4a68175b5440829c506a5dce3e48edd2c662b8/libreloc/libseastar.so+0x476da7e
/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.ccm/scylla-repository/6f4a68175b5440829c506a5dce3e48edd2c662b8/libreloc/libseastar.so+0x476d8ca
/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.ccm/scylla-repository/6f4a68175b5440829c506a5dce3e48edd2c662b8/libreloc/libc.so.6+0x3cb1f
/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.ccm/scylla-repository/6f4a68175b5440829c506a5dce3e48edd2c662b8/libreloc/libc.so.6+0x8ce5b
/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.ccm/scylla-repository/6f4a68175b5440829c506a5dce3e48edd2c662b8/libreloc/libc.so.6+0x3ca75
/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.ccm/scylla-repository/6f4a68175b5440829c506a5dce3e48edd2c662b8/libreloc/libc.so.6+0x267fb
/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.ccm/scylla-repository/6f4a68175b5440829c506a5dce3e48edd2c662b8/libreloc/libseastar.so+0x443a583
0x107025d7
0xe1257ff
0x1077d8b9
0x1077c2d0
0x1077b707
0x1078128d
0x1077a8ee
0x1077a729
0x1077a6e7
0x1077a664
0x1077a2db
0x10779d4a
0x10779aae
0x1076d741
0x1071405e
0x107586e8
/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.ccm/scylla-repository/6f4a68175b5440829c506a5dce3e48edd2c662b8/libreloc/libseastar.so+0x43385ad
/jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.ccm/scylla-repository/6f4a68175b5440829c506a5dce3e48edd2c662b8/libreloc/libseastar.so+0x4b3ff87Decoded:
[Backtrace #0]
__interceptor_backtrace.part.0 at ??:?
void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) at ./build/debug/seastar/./seastar/include/seastar/util/backtrace.hh:60
seastar::backtrace_buffer::append_backtrace() at ./build/debug/seastar/./seastar/src/core/reactor.cc:788
seastar::print_with_backtrace(seastar::backtrace_buffer&, bool) at ./build/debug/seastar/./seastar/src/core/reactor.cc:818
seastar::print_with_backtrace(char const*, bool) at ./build/debug/seastar/./seastar/src/core/reactor.cc:830
seastar::sigabrt_action() at ./build/debug/seastar/./seastar/src/core/reactor.cc:3946
operator() at ./build/debug/seastar/./seastar/src/core/reactor.cc:3922
__invoke at ./build/debug/seastar/./seastar/src/core/reactor.cc:3918
/data/scylla-s3-reloc.cache/by-build-id/ff34d927919817aaa8551e8536056fa0040a379f/extracted/scylla/libreloc/libc.so.6: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=81daba31ee66dbd63efdc4252a872949d874d136, for GNU/Linux 3.2.0, not stripped
__GI___sigaction at :?
__pthread_kill_implementation at ??:?
__GI_raise at :?
__GI_abort at :?
seastar::on_internal_error(seastar::logger&, std::basic_string_view<char, std::char_traits<char> >) at ./build/debug/seastar/./seastar/src/core/on_internal_error.cc:44
operator() at ./db/view/view.cc:1455
(inlined by) db::view::view_updates boost::range_detail::default_constructible_unary_fn_wrapper<db::view::make_view_update_builder(data_dictionary::database, replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_33, db::view::view_updates>::operator()<db::view::view_and_base>(db::view::view_and_base&) const at /usr/include/boost/range/detail/default_constructible_unary_fn.hpp:62
(inlined by) boost::iterators::transform_iterator<boost::range_detail::default_constructible_unary_fn_wrapper<db::view::make_view_update_builder(data_dictionary::database, replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_33, db::view::view_updates>, __gnu_cxx::__normal_iterator<db::view::view_and_base*, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > >, boost::use_default, boost::use_default>::dereference() const at /usr/include/boost/iterator/transform_iterator.hpp:126
(inlined by) boost::iterators::transform_iterator<boost::range_detail::default_constructible_unary_fn_wrapper<db::view::make_view_update_builder(data_dictionary::database, replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_33, db::view::view_updates>, __gnu_cxx::__normal_iterator<db::view::view_and_base*, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > >, boost::use_default, boost::use_default>::reference boost::iterators::iterator_core_access::dereference<boost::iterators::transform_iterator<boost::range_detail::default_constructible_unary_fn_wrapper<db::view::make_view_update_builder(data_dictionary::database, replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_33, db::view::view_updates>, __gnu_cxx::__normal_iterator<db::view::view_and_base*, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > >, boost::use_default, boost::use_default> >(boost::iterators::transform_iterator<boost::range_detail::default_constructible_unary_fn_wrapper<db::view::make_view_update_builder(data_dictionary::database, replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_33, db::view::view_updates>, __gnu_cxx::__normal_iterator<db::view::view_and_base*, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > >, boost::use_default, boost::use_default> const&) at /usr/include/boost/iterator/iterator_facade.hpp:550
(inlined by) boost::iterators::detail::iterator_facade_base<boost::iterators::transform_iterator<boost::range_detail::default_constructible_unary_fn_wrapper<db::view::make_view_update_builder(data_dictionary::database, replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_33, db::view::view_updates>, __gnu_cxx::__normal_iterator<db::view::view_and_base*, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > >, boost::use_default, boost::use_default>, db::view::view_updates, boost::iterators::random_access_traversal_tag, db::view::view_updates, long, false, false>::operator*() const at /usr/include/boost/iterator/iterator_facade.hpp:656
(inlined by) void std::vector<db::view::view_updates, std::allocator<db::view::view_updates> >::_M_range_initialize<boost::iterators::transform_iterator<boost::range_detail::default_constructible_unary_fn_wrapper<db::view::make_view_update_builder(data_dictionary::database, replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_33, db::view::view_updates>, __gnu_cxx::__normal_iterator<db::view::view_and_base*, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > >, boost::use_default, boost::use_default> >(boost::iterators::transform_iterator<boost::range_detail::default_constructible_unary_fn_wrapper<db::view::make_view_update_builder(data_dictionary::database, replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_33, db::view::view_updates>, __gnu_cxx::__normal_iterator<db::view::view_and_base*, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > >, boost::use_default, boost::use_default>, boost::iterators::transform_iterator<boost::range_detail::default_constructible_unary_fn_wrapper<db::view::make_view_update_builder(data_dictionary::database, replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_33, db::view::view_updates>, __gnu_cxx::__normal_iterator<db::view::view_and_base*, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > >, boost::use_default, boost::use_default>, std::input_iterator_tag) at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/stl_vector.h:1668
(inlined by) vector<boost::iterators::transform_iterator<boost::range_detail::default_constructible_unary_fn_wrapper<(lambda at db/view/view.cc:1453:107), db::view::view_updates>, __gnu_cxx::__normal_iterator<db::view::view_and_base *, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > >, boost::use_default, boost::use_default>, void> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/stl_vector.h:706
(inlined by) std::vector<db::view::view_updates, std::allocator<db::view::view_updates> > boost::copy_range<std::vector<db::view::view_updates, std::allocator<db::view::view_updates> >, boost::range_detail::transformed_range<db::view::make_view_update_builder(data_dictionary::database, replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_33, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > > >(boost::range_detail::transformed_range<db::view::make_view_update_builder(data_dictionary::database, replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::$_33, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> > > const&) at /usr/include/boost/range/iterator_range_core.hpp:842
(inlined by) db::view::make_view_update_builder(data_dictionary::database, replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, flat_mutation_reader_v2&&, seastar::optimized_optional<flat_mutation_reader_v2>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >) at ./db/view/view.cc:1453
replica::table::populate_views(seastar::shared_ptr<db::view::view_update_generator>, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >, dht::token, flat_mutation_reader_v2&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >) at ./replica/table.cc:2157
db::view::view_builder::consumer::flush_fragments() at ./db/view/view.cc:2406
db::view::view_builder::consumer::consume_end_of_partition() at ./db/view/view.cc:2420
seastar::bool_class<seastar::stop_iteration_tag> compact_mutation_state<(compact_for_sstables)0>::consume_end_of_partition<db::view::view_builder::consumer, noop_compacted_fragments_consumer>(db::view::view_builder::consumer&, noop_compacted_fragments_consumer&) at ././mutation/mutation_compactor.hh:469
operator() at ././readers/flat_mutation_reader_v2.hh:289
(inlined by) seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > seastar::futurize<seastar::bool_class<seastar::stop_iteration_tag> >::invoke<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >::consume(partition_end&&)::{lambda()#1}>(flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >::consume(partition_end&&)::{lambda()#1}&&) at ././seastar/include/seastar/core/future.hh:1992
auto seastar::futurize_invoke<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >::consume(partition_end&&)::{lambda()#1}>(flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >::consume(partition_end&&)::{lambda()#1}&&) at ././seastar/include/seastar/core/future.hh:2021
(inlined by) flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >::consume(partition_end&&) at ././readers/flat_mutation_reader_v2.hh:288
(inlined by) decltype(auto) mutation_fragment_v2::consume<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >(flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >&) && at ././mutation/mutation_fragment_v2.hh:289
flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >::operator()(mutation_fragment_v2&&) at ././readers/flat_mutation_reader_v2.hh:268
(inlined by) seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > std::__invoke_impl<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >, flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >&, mutation_fragment_v2>(std::__invoke_other, flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >&, mutation_fragment_v2&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:61
(inlined by) std::__invoke_result<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >&, mutation_fragment_v2>::type std::__invoke<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >&, mutation_fragment_v2>(flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >&, mutation_fragment_v2&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:96
std::result_of<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >& (mutation_fragment_v2&&)>::type std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >::operator()<mutation_fragment_v2>(mutation_fragment_v2&&) const at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/refwrap.h:358
operator() at ././readers/flat_mutation_reader_v2.hh:249
(inlined by) seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > seastar::futurize<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > >::invoke<flat_mutation_reader_v2::impl::consume_pausable_in_thread<std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter>(std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter)::{lambda()#1}>(flat_mutation_reader_v2::impl::consume_pausable_in_thread<std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter>(std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter)::{lambda()#1}&&) at ././seastar/include/seastar/core/future.hh:1990
auto seastar::futurize_invoke<flat_mutation_reader_v2::impl::consume_pausable_in_thread<std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter>(std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter)::{lambda()#1}>(flat_mutation_reader_v2::impl::consume_pausable_in_thread<std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter>(std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter)::{lambda()#1}&&) at ././seastar/include/seastar/core/future.hh:2021
(inlined by) void flat_mutation_reader_v2::impl::consume_pausable_in_thread<std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter>(std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, flat_mutation_reader_v2::no_filter) at ././readers/flat_mutation_reader_v2.hh:248
auto flat_mutation_reader_v2::impl::consume_in_thread<compact_for_query_v2<db::view::view_builder::consumer>, flat_mutation_reader_v2::no_filter>(compact_for_query_v2<db::view::view_builder::consumer>, flat_mutation_reader_v2::no_filter) at ././readers/flat_mutation_reader_v2.hh:335
auto flat_mutation_reader_v2::consume_in_thread<compact_for_query_v2<db::view::view_builder::consumer>, flat_mutation_reader_v2::no_filter>(compact_for_query_v2<db::view::view_builder::consumer>, flat_mutation_reader_v2::no_filter) at ././readers/flat_mutation_reader_v2.hh:473
auto flat_mutation_reader_v2::consume_in_thread<compact_for_query_v2<db::view::view_builder::consumer> >(compact_for_query_v2<db::view::view_builder::consumer>) at ././readers/flat_mutation_reader_v2.hh:479
db::view::view_builder::execute(db::view::view_builder::build_step&, exponential_backoff_retry) at ./db/view/view.cc:2457
operator() at ./db/view/view.cc:2222
(inlined by) void std::__invoke_impl<void, db::view::view_builder::do_build_step()::$_26>(std::__invoke_other, db::view::view_builder::do_build_step()::$_26&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:61
(inlined by) std::__invoke_result<db::view::view_builder::do_build_step()::$_26>::type std::__invoke<db::view::view_builder::do_build_step()::$_26>(db::view::view_builder::do_build_step()::$_26&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:96
(inlined by) decltype(auto) std::__apply_impl<db::view::view_builder::do_build_step()::$_26, std::tuple<>>(db::view::view_builder::do_build_step()::$_26&&, std::tuple<>&&, std::integer_sequence<unsigned long>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/tuple:1852
(inlined by) decltype(auto) std::apply<db::view::view_builder::do_build_step()::$_26, std::tuple<> >(db::view::view_builder::do_build_step()::$_26&&, std::tuple<>&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/tuple:1863
(inlined by) seastar::future<void> seastar::futurize<void>::apply<db::view::view_builder::do_build_step()::$_26>(db::view::view_builder::do_build_step()::$_26&&, std::tuple<>&&) at ././seastar/include/seastar/core/future.hh:1954
(inlined by) operator() at ././seastar/include/seastar/core/thread.hh:258
(inlined by) seastar::noncopyable_function<void ()>::direct_vtable_for<seastar::async<db::view::view_builder::do_build_step()::$_26>(seastar::thread_attributes, db::view::view_builder::do_build_step()::$_26&&)::{lambda()#1}>::call(seastar::noncopyable_function<void ()> const*) at ././seastar/include/seastar/util/noncopyable_function.hh:124Coredump is here: https://jenkins.scylladb.com/view/master/job/scylla-master/job/dtest-daily-debug/198/artifact/logs-full.debug.056/1684389028644_secondary_indexes_test.py%3A%3ATestLocalIndexes%3A%3Atest_drop_local_index_while_building/node1-scylla.6497.1684388973.core.gz Reloc package here: http://downloads.scylladb.com/unstable/scylla/master/relocatable/2023-05-18T03:04:07Z/scylla-debug-unstripped-5.3.0~dev-0.20230518.6f4a68175b54.x86_64.tar.gz
DoronArazii
commented
1 year ago @nyh / @cvybhu does anyone is actively working on it?
nyh
commented
1 year ago No, as you can see in the comments I last worked on this in September 2022. I was actually able to reproduce this issue then, but didn't find an easy fix - and haven't looked at it since.
I'll try to find time to handle this one too, but I was assigned to do other stuff, not this.
nyh
commented
1 year ago Happened again in SCT, see https://github.com/scylladb/scylladb/issues/13708
kostja
commented
3 months ago @kbr-scylla the duplicate issue #13708 is using consistent_topology_changes: true, so apparently it's reproducible with consistent schema changes. @temichus could you please try to reproduce it? The dtest failure has not been reproducible for a long time, but the longevity test failure by @aleksbykov could still be possible. If it doesn't reproduce with 6.0/consistent cluster management, please close.
juliayakovlev
commented
3 months ago Reproduced with 2023.1.7
Scylla version: 2023.1.7-20240402.1f84904a7100 with build-id d8d3f80469204be472563634963c19f86b295af0
Kernel Version: 5.15.0-1056-aws
Describe your issue in detail and steps it took to produce it.
Describe the impact this issue causes to the user.
Describe the frequency with how this issue can be reproduced.
Cluster size: 12 nodes (i3en.2xlarge)
Scylla Nodes used in this run:
OS / Image: ami-022c98b5b8d709845 ami-0c12e2ac711a25fd8 (aws: undefined_region)
Test: longevity-multidc-schema-topology-changes-12h-test
Test id: 077a3e30-7906-4a43-95b8-0f9aa83eeb40
Test name: enterprise-2023.1/longevity/longevity-multidc-schema-topology-changes-12h-test
Test config file(s):
mykaul
commented
3 months ago @juliayakovlev - did you run with raft for schema enabled or disabled?
juliayakovlev
commented
3 months ago @juliayakovlev - did you run with raft for schema enabled or disabled?
raft disabled (default)
mykaul
commented
3 months ago @juliayakovlev - did you run with raft for schema enabled or disabled?
raft disabled (default)
Thanks. Would be good to test if it happens with raft enabled.
Isn't raft for schema enabled for new clusters?
kostja
commented
3 months ago @mykaul https://github.com/scylladb/scylladb/issues/13708 is with raft enabled.
kbr-scylla
commented
1 month ago Let's not pretend that an issue which wasn't fixed for 3 years is urgent, obviously it isn't or it would have been fixed in the past 3 years (... right?) --- turning it from P1 to P2.
kbr-scylla
commented
1 month ago but didn't find an easy fix
@nyh I read your comments and please confirm if I understand correctly --- this is a non-critical transient failure because we're trying to write to MV that was dropped? If so --- then turning on_internal_error into logger.error(...); return; would be a fix?
nyh
commented
1 month ago but didn't find an easy fix
@nyh I read your comments and please confirm if I understand correctly --- this is a non-critical transient failure because we're trying to write to MV that was dropped? If so --- then turning
on_internal_errorintologger.error(...); return;would be a fix?
Last time I looked at this issue was two years ago, and my conclusions are in https://github.com/scylladb/scylladb/issues/9059#issuecomment-1236357999 and I don't remember more than that.
I noted that outside our test environment that converts on_internal_error into crashes, this on_internal_error is benign - it is converted into an exception, which causes a view building step to be aborted, and in my experience this leads to the step being immediately started again and completed normally. You're right that this indeed means that there is no real need for this "on_internal_error" - a normal exception would have been good enough. Maybe even returning nothing like you suggested would also work - but we'll need to check. And in any case, it seems to me that if an exception also works, it's better (I think it will be logged somewhere).
But the question remains is why the person who wrote this code thought it was an "on_internal_error" situation. Maybe there is really a bug here that is reproduced by the test, and if we ignore it, we lose the opportunity to fix this bug? I don't know. @tgrabiec wrote this code in 3a6ec9933c768aa1e4c84a38bf5c5101cd226115, maybe he remembers something.
tgrabiec
commented
1 month ago But the question remains is why the person who wrote this code thought it was an "on_internal_error" situation. Maybe there is really a bug here that is reproduced by the test, and if we ignore it, we lose the opportunity to fix this bug? I don't know. @tgrabiec wrote this code in 3a6ec99, maybe he remembers something.
It's on_internal_error() because it should never happen, it means invariants are violated. Maybe the situation is transient, maybe not. We use on_internal_error() for such cases to allow us to get the core dump and debug the problem.
yarongilor
commented
2 weeks ago might be reproduced in dtest debug mode - https://jenkins.scylladb.com/view/master/job/scylla-master/job/dtest-debug/308/testReport/secondary_indexes_test/TestLocalIndexes/Run_Dtest_Parallel_Cloud_Machines___FullDtest___full_split018___test_drop_local_index_while_building/
Scylla version 6.1.0~dev-0.20240706.407274e828a9 with build-id 3698e2b372406a68eafc2c8b89a6d7294b3d7c85
The scenario failure log is:
06:30:15,177 700 tools.data DEBUG data.py :324 | test_drop_local_index_while_building: Create index: CREATE INDEX idx ON standard1 ((key), "C0")
06:30:18,926 700 tools.data DEBUG data.py :329 | test_drop_local_index_while_building: Index idx has been created
06:30:18,936 700 secondary_indexes_test DEBUG secondary_indexes_test.py:2181 | test_drop_local_index_while_building: View ID: 5f102000-3c2a-11ef-aef1-623323b4d2b7
06:30:24,140 700 cassandra.cluster WARNING libevreactor.py :295 | test_drop_local_index_while_building: Host 127.0.32.3:9042 has been marked downnode-3 had:
view - Schema version used for view updates (613e98c0-3c2a-11ef-a831-8c00daa1ac03) does not match the current base schema version of the view (5f0ff8f0-3c2a-11ef-aef1-623323b4d2b7) for view keyspace1.idx_index of keyspace1.standard1fuller log:
WARN 2024-07-07 06:30:22,125 [shard 1:stmt] storage_proxy - Failed to apply mutation from 127.0.32.4#1: data_dictionary::no_such_column_family (Can't find a column family with UUID 5f102000-3c2a-11ef-aef1-623323b4d2b7)
INFO 2024-07-07 06:30:22,125 [shard 1:comp] compaction - [Compact system.truncated 63129570-3c2a-11ef-b4c8-8bffdaa1ac03] Compacted 2 sstables to [/jenkins/workspace/scylla-master/dtest-debug/scylla/.dtest/dtest-yg3zr0sm/test/node3/data/system/truncated-38c19fd0fb863310a4b70d0cc66628aa/me-3ghm_0i2m_05sc02qyz8tmunbqwz-big-Data.db:level=0]. 80kB to 75kB (~93% of original) in 95ms = 848kB/s. ~256 total partitions merged to 7.
WARN 2024-07-07 06:30:22,273 [shard 1: gms] view - Error executing build step for base keyspace1.standard1: seastar::internal::backtraced<data_dictionary::no_such_column_family> (Can't find a column family with UUID 5f102000-3c2a-11ef-aef1-623323b4d2b7 Backtrace: 0x2db1439 /jenkins/workspace/scylla-master/dtest-debug/scylla/.ccm/scylla-repository/407274e828a9b78b0b339b52dc5dd793d3dd789c/libreloc/libseastar.so+0x42d98a7 /jenkins/workspace/scylla-master/dtest-debug/scylla/.ccm/scylla-repository/407274e828a9b78b0b339b52dc5dd793d3dd789c/libreloc/libseastar.so+0x42d94f1 /jenkins/workspace/scylla-master/dtest-debug/scylla/.ccm/scylla-repository/407274e828a9b78b0b339b52dc5dd793d3dd789c/libreloc/libseastar.so+0x42dd720 /jenkins/workspace/scylla-master/dtest-debug/scylla/.ccm/scylla-repository/407274e828a9b78b0b339b52dc5dd793d3dd789c/libreloc/libseastar.so+0x42dff6f 0x44a0939 0x44a02c6 0x44a0175 0x4481142 0x444f0a6 0x6b41819 0x6b40a07 0x2f7133d 0x2f710cc 0x2f70e2c /jenkins/workspace/scylla-master/dtest-debug/scylla/.ccm/scylla-repository/407274e828a9b78b0b339b52dc5dd793d3dd789c/libreloc/libseastar.so+0x347f3be /jenkins/workspace/scylla-master/dtest-debug/scylla/.ccm/scylla-repository/407274e828a9b78b0b339b52dc5dd793d3dd789c/libreloc/libseastar.so+0x348b0d5 /jenkins/workspace/scylla-master/dtest-debug/scylla/.ccm/scylla-repository/407274e828a9b78b0b339b52dc5dd793d3dd789c/libreloc/libseastar.so+0x3490439 /jenkins/workspace/scylla-master/dtest-debug/scylla/.ccm/scylla-repository/407274e828a9b78b0b339b52dc5dd793d3dd789c/libreloc/libseastar.so+0x357529f /jenkins/workspace/scylla-master/dtest-debug/scylla/.ccm/scylla-repository/407274e828a9b78b0b339b52dc5dd793d3dd789c/libreloc/libseastar.so+0x35738a0 /jenkins/workspace/scylla-master/dtest-debug/scylla/.ccm/scylla-repository/407274e828a9b78b0b339b52dc5dd793d3dd789c/libreloc/libseastar.so+0x3573790 /jenkins/workspace/scylla-master/dtest-debug/scylla/.ccm/scylla-repository/407274e828a9b78b0b339b52dc5dd793d3dd789c/libreloc/libseastar.so+0x3572efc /jenkins/workspace/scylla-master/dtest-debug/scylla/.ccm/scylla-repository/407274e828a9b78b0b339b52dc5dd793d3dd789c/libreloc/libseastar.so+0x3260688 0x2e03b2c /jenkins/workspace/scylla-master/dtest-debug/scylla/.ccm/scylla-repository/407274e828a9b78b0b339b52dc5dd793d3dd789c/libreloc/libc.so.6+0x961b6 /jenkins/workspace/scylla-master/dtest-debug/scylla/.ccm/scylla-repository/407274e828a9b78b0b339b52dc5dd793d3dd789c/libreloc/libc.so.6+0x1183cb
--------
seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::noncopyable_function<seastar::future<void> ()>, seastar::future<void>::then_impl_nrvo<seastar::noncopyable_function<seastar::future<void> ()>, seastar::future<void> >(seastar::noncopyable_function<seastar::future<void> ()>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> ()>&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>
--------
N7seastar8internal14do_until_stateIZZNS_23max_concurrent_for_eachIN5utils14chunked_vectorI26frozen_mutation_and_schemaLm131072EE13iterator_typeIS5_EES8_ZN2db4view21view_update_generator9mutate_MVENS_13lw_shared_ptrIK6schemaEEN3dht5tokenES6_RNSA_5statsERN7replica8cf_statsEN7tracing15trace_state_ptrENS_15semaphore_unitsINS_33default_timeout_exception_factoryENS_12lowres_clockEEENS_10bool_classIN7service15allow_hints_tagEEENST_INSA_24wait_for_all_updates_tagEEEE3$_0QaarQT1_T__XclfL1p_defL1p0_ERSt7same_asINS_6futureIvEEEXppfL1p0_Eoosr3stdE7same_asIT0_S11_Esr3stdE12sentinel_forIS15_S11_EEES14_S11_S15_mOS10_ENKUlRZNS2_IS8_S8_SZ_QaarQS10_S11__XclfL2p_defL2p0_ERS12_IS14_EXppfL2p0_Eoosr3stdE7same_asIS15_S11_Esr3stdE12sentinel_forIS15_S11_EEES14_S11_S15_mS16_E5stateE_clES18_EUlvE_ZZNS2_IS8_S8_SZ_QaarQS10_S11__XclfL1p_defL1p0_ERS12_IS14_EXppfL1p0_Eoosr3stdE7same_asIS15_S11_Esr3stdE12sentinel_forIS15_S11_EEES14_S11_S15_mS16_ENKS19_clES18_EUlvE0_EE
--------
seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::noncopyable_function<seastar::future<void> ()>, seastar::future<void>::then_impl_nrvo<seastar::noncopyable_function<seastar::future<void> ()>, seastar::future<void> >(seastar::noncopyable_function<seastar::future<void> ()>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> ()>&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>
--------
seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::noncopyable_function<seastar::future<void> ()>, seastar::future<void>::then_impl_nrvo<seastar::noncopyable_function<seastar::future<void> ()>, seastar::future<void> >(seastar::noncopyable_function<seastar::future<void> ()>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> ()>&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>
--------
N7seastar8internal13do_with_stateISt5tupleIJZNS_23max_concurrent_for_eachIN5utils14chunked_vectorI26frozen_mutation_and_schemaLm131072EE13iterator_typeIS6_EES9_ZN2db4view21view_update_generator9mutate_MVENS_13lw_shared_ptrIK6schemaEEN3dht5tokenES7_RNSB_5statsERN7replica8cf_statsEN7tracing15trace_state_ptrENS_15semaphore_unitsINS_33default_timeout_exception_factoryENS_12lowres_clockEEENS_10bool_classIN7service15allow_hints_tagEEENSU_INSB_24wait_for_all_updates_tagEEEE3$_0QaarQT1_T__XclfL0p_defL0p0_ERSt7same_asINS_6futureIvEEEXppfL0p0_Eoosr3stdE7same_asIT0_S12_Esr3stdE12sentinel_forIS16_S12_EEES15_S12_S16_mOS11_E5stateEES15_EE
--------
seastar::internal::coroutine_traits_base<void>::promise_type
--------
seastar::internal::coroutine_traits_base<void>::promise_type
--------
seastar::(anonymous namespace)::thread_wake_task
--------
seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::noncopyable_function<seastar::future<void> ()>, seastar::future<void>::then_impl_nrvo<seastar::noncopyable_function<seastar::future<void> ()>, seastar::future<void> >(seastar::noncopyable_function<seastar::future<void> ()>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> ()>&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>
--------
seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>, seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)> >(seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>
--------
seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>, seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)> >(seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>
--------
seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>, seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)> >(seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>)
INFO 2024-07-07 06:30:22,400 [shard 0: gms] view - Building view keyspace1.idx_index, starting at token minimum token
INFO 2024-07-07 06:30:22,405 [shard 0: gms] view - Stopping to build view keyspace1.idx_index
INFO 2024-07-07 06:30:22,406 [shard 0: gms] query_processor - Column definitions for keyspace1.standard1 changed, invalidating related prepared statements
INFO 2024-07-07 06:30:22,408 [shard 1: gms] query_processor - Column definitions for keyspace1.standard1 changed, invalidating related prepared statements
ERROR 2024-07-07 06:30:22,421 [shard 0: gms] view - Schema version used for view updates (613e98c0-3c2a-11ef-a831-8c00daa1ac03) does not match the current base schema version of the view (5f0ff8f0-3c2a-11ef-aef1-623323b4d2b7) for view keyspace1.idx_index of keyspace1.standard1, at: 0x2db1439 /jenkins/workspace/scylla-master/dtest-debug/scylla/.ccm/scylla-repository/407274e828a9b78b0b339b52dc5dd793d3dd789c/libreloc/libseastar.so+0x42d98a7 /jenkins/workspace/scylla-master/dtest-debug/scylla/.ccm/scylla-repository/407274e828a9b78b0b339b52dc5dd793d3dd789c/libreloc/libseastar.so+0x42d94f1 /jenkins/workspace/scylla-master/dtest-debug/scylla/.ccm/scylla-repository/407274e828a9b78b0b339b52dc5dd793d3dd789c/libreloc/libseastar.so+0x42dd720 /jenkins/workspace/scylla-master/dtest-debug/scylla/.ccm/scylla-repository/407274e828a9b78b0b339b52dc5dd793d3dd789c/libreloc/libseastar.so+0x42dff6f /jenkins/workspace/scylla-master/dtest-debug/scylla/.ccm/scylla-repository/407274e828a9b78b0b339b52dc5dd793d3dd789c/libreloc/libseastar.so+0x3257ec1 /jenkins/workspace/scylla-master/dtest-debug/scylla/.ccm/scylla-repository/407274e828a9b78b0b339b52dc5dd793d3dd789c/libreloc/libseastar.so+0x3257ad7 0x6afce31 0x6bd9831 0x6ba1500 0x6b9ed81 0x6ba61ae 0x6b9d6e9 0x6b9d56a 0x6b9ccd5 0x6b9c48f 0x6b9c166 0x6b8dac9 0x6b130fe 0x6b647e0 /jenkins/workspace/scylla-master/dtest-debug/scylla/.ccm/scylla-repository/407274e828a9b78b0b339b52dc5dd793d3dd789c/libreloc/libseastar.so+0x3146962 /jenkins/workspace/scylla-master/dtest-debug/scylla/.ccm/scylla-repository/407274e828a9b78b0b339b52dc5dd793d3dd789c/libreloc/libseastar.so+0x392c1c8
--------
seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::noncopyable_function<seastar::future<void> ()>, seastar::future<void>::then_impl_nrvo<seastar::noncopyable_function<seastar::future<void> ()>, seastar::future<void> >(seastar::noncopyable_function<seastar::future<void> ()>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> ()>&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>
--------
seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>, seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)> >(seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>
--------
seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>, seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)> >(seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>
--------
seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>, seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)> >(seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>
Aborting on shard 0, in scheduling group gossip.
Backtrace:
0x2db1439
/jenkins/workspace/scylla-master/dtest-debug/scylla/.ccm/scylla-repository/407274e828a9b78b0b339b52dc5dd793d3dd789c/libreloc/libseastar.so+0x35feb57
/jenkins/workspace/scylla-master/dtest-debug/scylla/.ccm/scylla-repository/407274e828a9b78b0b339b52dc5dd793d3dd789c/libreloc/libseastar.so+0x35fe7e5
/jenkins/workspace/scylla-master/dtest-debug/scylla/.ccm/scylla-repository/407274e828a9b78b0b339b52dc5dd793d3dd789c/libreloc/libseastar.so+0x345f96b
/jenkins/workspace/scylla-master/dtest-debug/scylla/.ccm/scylla-repository/407274e828a9b78b0b339b52dc5dd793d3dd789c/libreloc/libseastar.so+0x3491bb5
/jenkins/workspace/scylla-master/dtest-debug/scylla/.ccm/scylla-repository/407274e828a9b78b0b339b52dc5dd793d3dd789c/libreloc/libseastar.so+0x3572a19
/jenkins/workspace/scylla-master/dtest-debug/scylla/.ccm/scylla-repository/407274e828a9b78b0b339b52dc5dd793d3dd789c/libreloc/libseastar.so+0x3572d3b
/jenkins/workspace/scylla-master/dtest-debug/scylla/.ccm/scylla-repository/407274e828a9b78b0b339b52dc5dd793d3dd789c/libreloc/libseastar.so+0x3572b6e
/jenkins/workspace/scylla-master/dtest-debug/scylla/.ccm/scylla-repository/407274e828a9b78b0b339b52dc5dd793d3dd789c/libreloc/libc.so.6+0x4070f
/jenkins/workspace/scylla-master/dtest-debug/scylla/.ccm/scylla-repository/407274e828a9b78b0b339b52dc5dd793d3dd789c/libreloc/libc.so.6+0x98143
/jenkins/workspace/scylla-master/dtest-debug/scylla/.ccm/scylla-repository/407274e828a9b78b0b339b52dc5dd793d3dd789c/libreloc/libc.so.6+0x4065d
/jenkins/workspace/scylla-master/dtest-debug/scylla/.ccm/scylla-repository/407274e828a9b78b0b339b52dc5dd793d3dd789c/libreloc/libc.so.6+0x28901
/jenkins/workspace/scylla-master/dtest-debug/scylla/.ccm/scylla-repository/407274e828a9b78b0b339b52dc5dd793d3dd789c/libreloc/libseastar.so+0x3257aff
0x6afce31
0x6bd9831
0x6ba1500
0x6b9ed81
0x6ba61ae
0x6b9d6e9
0x6b9d56a
0x6b9ccd5
0x6b9c48f
0x6b9c166
0x6b8dac9
0x6b130fe
0x6b647e0
/jenkins/workspace/scylla-master/dtest-debug/scylla/.ccm/scylla-repository/407274e828a9b78b0b339b52dc5dd793d3dd789c/libreloc/libseastar.so+0x3146962
/jenkins/workspace/scylla-master/dtest-debug/scylla/.ccm/scylla-repository/407274e828a9b78b0b339b52dc5dd793d3dd789c/libreloc/libseastar.so+0x392c1c8The dtest failed for:
failed on teardown with "AssertionError: Critical errors found: [('node1', ['Aborting on shard 0, in scheduling group gossip.']), ('node2', ['Aborting on shard 0, in scheduling group gossip.']), ('node3', ['Aborting on shard 0, in scheduling group gossip.'])]@yarongilor - please decode the backtrace, let's see if it's a known one.
yarongilor
commented
2 weeks ago @yarongilor - please decode the backtrace, let's see if it's a known one.
looks similar to the above.
[Backtrace #0]
__interceptor_backtrace at ??:?
void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) at ./build/debug/seastar/./seastar/include/seastar/util/backtrace.hh:68
seastar::backtrace_buffer::append_backtrace() at ./build/debug/seastar/./build/debug/seastar/./seastar/src/core/reactor.cc:826
seastar::print_with_backtrace(seastar::backtrace_buffer&, bool) at ./build/debug/seastar/./build/debug/seastar/./seastar/src/core/reactor.cc:859
seastar::print_with_backtrace(char const*, bool) at ./build/debug/seastar/./build/debug/seastar/./seastar/src/core/reactor.cc:871
seastar::sigabrt_action() at ./build/debug/seastar/./build/debug/seastar/./seastar/src/core/reactor.cc:4033
seastar::install_oneshot_signal_handler<6, (void (*)())(&seastar::sigabrt_action)>()::{lambda(int, siginfo_t*, void*)#1}::operator()(int, siginfo_t*, void*) const at ./build/debug/seastar/./build/debug/seastar/./seastar/src/core/reactor.cc:4009
seastar::install_oneshot_signal_handler<6, (void (*)())(&seastar::sigabrt_action)>()::{lambda(int, siginfo_t*, void*)#1}::__invoke(int, siginfo_t*, void*) at ./build/debug/seastar/./build/debug/seastar/./seastar/src/core/reactor.cc:4005
/data/scylla-s3-reloc.cache/by-build-id/3698e2b372406a68eafc2c8b89a6d7294b3d7c85/extracted/scylla/libreloc/libc.so.6: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=980b86754dc53e7fd5473d38c629977f47aafd2f, for GNU/Linux 3.2.0, not stripped
__GI___sigaction at :?
__pthread_kill_implementation at ??:?
__GI_raise at :?
__GI_abort at :?
seastar::on_internal_error(seastar::logger&, std::basic_string_view<char, std::char_traits<char> >) at ./build/debug/seastar/./build/debug/seastar/./seastar/src/core/on_internal_error.cc:57
db::view::make_view_update_builder(data_dictionary::database, replica::table const&, seastar::lw_shared_ptr<schema const> const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >&&, mutation_reader&&, seastar::optimized_optional<mutation_reader>&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >) at ././db/view/view.cc:1502
db::view::view_update_generator::populate_views(replica::table const&, std::vector<db::view::view_and_base, std::allocator<db::view::view_and_base> >, dht::token, mutation_reader&&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >) at ././db/view/view_update_generator.cc:337
db::view::view_builder::consumer::flush_fragments() at ././db/view/view.cc:2506
_ZN22compact_mutation_stateIL20compact_for_sstables0EE24consume_end_of_partitionIN2db4view12view_builder8consumerE33noop_compacted_fragments_consumerQaa28CompactedFragmentsConsumerV2ITL0__E28CompactedFragmentsConsumerV2ITL0_0_EEEN7seastar10bool_classINSA_18stop_iteration_tagEEERT_RT0_ at ././db/view/view.cc:2520
seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > seastar::futurize<seastar::bool_class<seastar::stop_iteration_tag> >::invoke<mutation_reader::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >::consume(partition_end&&)::{lambda()#1}>(mutation_reader::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >::consume(partition_end&&)::{lambda()#1}&&) at ././mutation/mutation_compactor.hh:681
auto seastar::futurize_invoke<mutation_reader::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >::consume(partition_end&&)::{lambda()#1}>(mutation_reader::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >::consume(partition_end&&)::{lambda()#1}&&) at ././seastar/include/seastar/core/future.hh:2063
(inlined by) mutation_reader::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >::consume(partition_end&&) at ././readers/mutation_reader.hh:290
(inlined by) decltype(auto) mutation_fragment_v2::consume<mutation_reader::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >(mutation_reader::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >&) && requires MutationFragmentConsumerV2<mutation_reader::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >, decltype ((((std::declval<mutation_reader::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >)()).consume)((std::declval<range_tombstone_change>)()))> at ././mutation/mutation_fragment_v2.hh:287
mutation_reader::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >::operator()(mutation_fragment_v2&&) at ././readers/mutation_reader.hh:270
(inlined by) seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > std::__invoke_impl<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >, mutation_reader::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >&, mutation_fragment_v2>(std::__invoke_other, mutation_reader::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >&, mutation_fragment_v2&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/14/../../../../include/c++/14/bits/invoke.h:61
(inlined by) std::__invoke_result<mutation_reader::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >&, mutation_fragment_v2>::type std::__invoke<mutation_reader::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >&, mutation_fragment_v2>(mutation_reader::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >&, mutation_fragment_v2&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/14/../../../../include/c++/14/bits/invoke.h:96
(inlined by) std::__invoke_result<mutation_reader::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> >&, mutation_fragment_v2>::type std::reference_wrapper<mutation_reader::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >::operator()<mutation_fragment_v2>(mutation_fragment_v2&&) const at /usr/bin/../lib/gcc/x86_64-redhat-linux/14/../../../../include/c++/14/bits/refwrap.h:363
(inlined by) mutation_reader::impl::consume_pausable_in_thread<std::reference_wrapper<mutation_reader::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, mutation_reader::no_filter>(std::reference_wrapper<mutation_reader::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, mutation_reader::no_filter) requires (FlatMutationReaderConsumerV2<std::reference_wrapper<mutation_reader::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > > >)&&(FlattenedConsumerFilterV2<mutation_reader::no_filter>)::{lambda()#1}::operator()() at ././readers/mutation_reader.hh:251
(inlined by) seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > seastar::futurize<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > >::invoke<mutation_reader::impl::consume_pausable_in_thread<std::reference_wrapper<mutation_reader::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, mutation_reader::no_filter>(std::reference_wrapper<mutation_reader::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, mutation_reader::no_filter) requires (FlatMutationReaderConsumerV2<std::reference_wrapper<mutation_reader::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > > >)&&(FlattenedConsumerFilterV2<mutation_reader::no_filter>)::{lambda()#1}>(mutation_reader::impl::consume_pausable_in_thread<std::reference_wrapper<mutation_reader::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, mutation_reader::no_filter>(std::reference_wrapper<mutation_reader::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, mutation_reader::no_filter) requires (FlatMutationReaderConsumerV2<std::reference_wrapper<mutation_reader::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > > >)&&(FlattenedConsumerFilterV2<mutation_reader::no_filter>)::{lambda()#1}&&) at ././seastar/include/seastar/core/future.hh:2032
(inlined by) auto seastar::futurize_invoke<mutation_reader::impl::consume_pausable_in_thread<std::reference_wrapper<mutation_reader::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, mutation_reader::no_filter>(std::reference_wrapper<mutation_reader::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, mutation_reader::no_filter) requires (FlatMutationReaderConsumerV2<std::reference_wrapper<mutation_reader::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > > >)&&(FlattenedConsumerFilterV2<mutation_reader::no_filter>)::{lambda()#1}>(mutation_reader::impl::consume_pausable_in_thread<std::reference_wrapper<mutation_reader::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, mutation_reader::no_filter>(std::reference_wrapper<mutation_reader::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, mutation_reader::no_filter) requires (FlatMutationReaderConsumerV2<std::reference_wrapper<mutation_reader::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > > >)&&(FlattenedConsumerFilterV2<mutation_reader::no_filter>)::{lambda()#1}&&) at ././seastar/include/seastar/core/future.hh:2063
void mutation_reader::impl::consume_pausable_in_thread<std::reference_wrapper<mutation_reader::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, mutation_reader::no_filter>(std::reference_wrapper<mutation_reader::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > >, mutation_reader::no_filter) requires (FlatMutationReaderConsumerV2<std::reference_wrapper<mutation_reader::impl::consumer_adapter<compact_for_query_v2<db::view::view_builder::consumer> > > >)&&(FlattenedConsumerFilterV2<mutation_reader::no_filter>) at ././readers/mutation_reader.hh:250
auto mutation_reader::impl::consume_in_thread<compact_for_query_v2<db::view::view_builder::consumer>, mutation_reader::no_filter>(compact_for_query_v2<db::view::view_builder::consumer>, mutation_reader::no_filter) requires (FlattenedConsumerV2<compact_for_query_v2<db::view::view_builder::consumer> >)&&(FlattenedConsumerFilterV2<mutation_reader::no_filter>) at ././readers/mutation_reader.hh:337
auto mutation_reader::consume_in_thread<compact_for_query_v2<db::view::view_builder::consumer>, mutation_reader::no_filter>(compact_for_query_v2<db::view::view_builder::consumer>, mutation_reader::no_filter) requires (FlattenedConsumerV2<compact_for_query_v2<db::view::view_builder::consumer> >)&&(FlattenedConsumerFilterV2<mutation_reader::no_filter>) at ././readers/mutation_reader.hh:475
auto mutation_reader::consume_in_thread<compact_for_query_v2<db::view::view_builder::consumer> >(compact_for_query_v2<db::view::view_builder::consumer>) requires FlattenedConsumerV2<compact_for_query_v2<db::view::view_builder::consumer> > at ././readers/mutation_reader.hh:481
db::view::view_builder::execute(db::view::view_builder::build_step&, exponential_backoff_retry) at ././db/view/view.cc:2565
seastar::noncopyable_function<void ()>::direct_vtable_for<seastar::async<db::view::view_builder::do_build_step()::$_0>(seastar::thread_attributes, db::view::view_builder::do_build_step()::$_0&&)::{lambda()#1}>::call(seastar::noncopyable_function<void ()> const*) at ././db/view/view.cc:2319
seastar::noncopyable_function<void ()>::operator()() const at ./build/debug/seastar/./seastar/include/seastar/util/noncopyable_function.hh:215
seastar::thread_context::main() at ./build/debug/seastar/./build/debug/seastar/./seastar/src/core/thread.cc:311
https://github.com/scylladb/scylladb/issues/13708#issuecomment-1578242304 hints that we should really change this on_internal_error() to something more benign.
dani-tweig
commented
2 weeks ago #13708 (comment) hints that we should really change this on_internal_error() to something more benign.
@tgrabiec , how do you think we should progress with this issue? (which is celebrating its third birthday in a couple of days ;) )
kostja
commented
2 weeks ago We should not have internal errors or coredumps (#13708) in presence of concurrent DDL. @dani-tweig this will be scheduled in q4 by either @kbr-scylla or @piodul 's teams and fixed.
kostja
commented
2 weeks ago @temichus for now, if coredumps are gone, the erorr can be suppressed in sct.
tgrabiec
commented
2 weeks ago We should not have internal errors or coredumps (#13708) in presence of concurrent DDL. @dani-tweig this will be scheduled in q4 by either @kbr-scylla or @piodul 's teams and fixed.
We should not, but in this case, we should achieve that by fixing the bug which is causing on_internal_error() to be hit, not by pretending it's benign (it's not).
dani-tweig
commented
2 weeks ago We should not have internal errors or coredumps (#13708) in presence of concurrent DDL. @dani-tweig this will be scheduled in q4 by either @kbr-scylla or @piodul 's teams and fixed.
We should not, but in this case, we should achieve that by fixing the bug which is causing on_internal_error() to be hit, not by pretending it's benign (it's not).
@kbr-scylla, @piodul (Who ever owns this), please reach some kind of agreement with @tgrabiec on how to fix this issue.
kbr-scylla
commented
2 weeks ago It could be similar to https://github.com/scylladb/scylladb/issues/18709 / https://github.com/scylladb/scylladb/issues/17786 (fix: 519317dc5833127732061c3eea2e2e99577b31e6), where view update code takes schema_ptr to base or view, then yields, schema gets changed, then it takes schema_ptr to the other one, and then checks for this condition which is at this point incorrect. (For the linked issues it was about taking e_r_ms but maybe same thing can happen with taking schema_ptrs)
So it's either indeed an invariant violation -- because we don't update in-memory schema correctly (which should preserve isolation) -- or it could be that the view update code traps itself like that.
Or maybe I'm misunderstanding and schema_ptr is actually a pointer to mutable schema (which gets updated with do_merge_schema)? @tgrabiec?
tgrabiec
commented
2 weeks ago Or maybe I'm misunderstanding and
schema_ptris actually a pointer to mutable schema (which gets updated withdo_merge_schema)? @tgrabiec?
schema_ptr is supposed to be immutable.
Seen in https://jenkins.scylladb.com/view/master/job/scylla-master/job/dtest-debug-random/521/artifact/logs-all.debug.2/1626560947253_lwt_schema_modification_test.LWTSchemaModificationTester.index_drop_add_test/node1.log
Decoded: