search

scylladb / scylladb

NoSQL data store using the seastar framework, compatible with Apache Cassandra
http://scylladb.com
GNU Affero General Public License v3.0
13.65k stars 1.3k forks source link

AlternatorTest.test_slow_query_logging: Assertion `std::find_if(_tasks.begin(), _tasks.end(), [cf] (auto& task) { return task->compacting_cf == cf; }) == _tasks.end()' failed. #9634

Closed bhalevy closed 3 years ago

bhalevy commented 3 years ago

Seen in https://jenkins.scylladb.com/view/master/job/scylla-master/job/dtest-debug-random/630/testReport/junit/alternator_tests/AlternatorTest/test_slow_query_logging/ Scylla version 9d4262e264efd6d92de9c40545f59c85c840de0d

https://jenkins.scylladb.com/view/master/job/scylla-master/job/dtest-debug-random/630/artifact/logs-all.debug.2/1637031904560_alternator_tests.AlternatorTest.test_slow_query_logging/node4.log

INFO 2021-11-16 03:04:01,484 [shard 0] compaction_manager - Stopping 1 ongoing compactions due to shutdown INFO 2021-11-16 03:04:01,484 [shard 0] init - Signal received; shutting down INFO 2021-11-16 03:04:01,484 [shard 0] init - Shutting down redis service INFO 2021-11-16 03:04:01,485 [shard 0] init - Shutting down redis service was successful INFO 2021-11-16 03:04:01,485 [shard 0] init - Shutting down view builder ops INFO 2021-11-16 03:04:01,485 [shard 0] view - Draining view builder INFO 2021-11-16 03:04:01,485 [shard 1] view - Draining view builder INFO 2021-11-16 03:04:01,485 [shard 1] compaction_manager - Asked to stop INFO 2021-11-16 03:04:01,485 [shard 1] compaction_manager - Stopping 1 ongoing compactions due to shutdown WARN 2021-11-16 03:04:01,486 [shard 0] gossip - failure_detector_loop: Got error in the loop, live_nodes=[127.0.82.1, 127.0.82.2]: seastar::sleep_aborted (Sleep is aborted) INFO 2021-11-16 03:04:01,486 [shard 0] gossip - failure_detector_loop: Finished main loop INFO 2021-11-16 03:04:01,486 [shard 0] init - Shutting down view builder ops was successful INFO 2021-11-16 03:04:01,486 [shard 0] init - Shutting down local storage INFO 2021-11-16 03:04:01,490 [shard 0] storage_service - Stop transport: starts INFO 2021-11-16 03:04:01,490 [shard 0] storage_service - Shutting down native transport server INFO 2021-11-16 03:04:01,493 [shard 1] compaction_manager - off-strategy compaction: Compaction for system_traces/events was stopped due to: shutdown INFO 2021-11-16 03:04:01,497 [shard 1] compaction_manager - Stopped INFO 2021-11-16 03:04:01,506 [shard 0] compaction_manager - off-strategy compaction: Compaction for system_traces/events was stopped due to: shutdown INFO 2021-11-16 03:04:01,506 [shard 0] cql_server_controller - CQL server stopped INFO 2021-11-16 03:04:01,506 [shard 0] storage_service - Shutting down native transport server was successful INFO 2021-11-16 03:04:01,507 [shard 0] storage_service - Shutting down rpc server INFO 2021-11-16 03:04:01,507 [shard 0] storage_service - Shutting down rpc server was successful INFO 2021-11-16 03:04:01,507 [shard 0] storage_service - Shutting down alternator server INFO 2021-11-16 03:04:01,511 [shard 0] compaction_manager - Stopped INFO 2021-11-16 03:04:01,515 [shard 0] storage_service - Shutting down alternator server was successful INFO 2021-11-16 03:04:01,515 [shard 0] storage_service - Shutting down redis server INFO 2021-11-16 03:04:01,515 [shard 0] storage_service - Shutting down redis server was successful INFO 2021-11-16 03:04:01,515 [shard 0] storage_service - Stop transport: shutdown rpc and cql server done INFO 2021-11-16 03:04:01,517 [shard 0] gossip - My status = NORMAL INFO 2021-11-16 03:04:01,517 [shard 0] gossip - Announcing shutdown INFO 2021-11-16 03:04:01,528 [shard 0] storage_service - Node 127.0.82.4 state jump to normal INFO 2021-11-16 03:04:02,407 [shard 0] rpc - client 127.0.82.2:63480 msg_id 766: exception "gate closed" in no_wait handler ignored INFO 2021-11-16 03:04:02,785 [shard 0] rpc - client 127.0.82.1:54946 msg_id 756: exception "gate closed" in no_wait handler ignored INFO 2021-11-16 03:04:03,160 [shard 0] gossip - Sending a GossipShutdown to 127.0.82.1 with generation 1637031328 INFO 2021-11-16 03:04:03,160 [shard 0] gossip - Sending a GossipShutdown to 127.0.82.2 with generation 1637031328 INFO 2021-11-16 03:04:03,179 [shard 0] rpc - client 127.0.82.2:63480 msg_id 767: exception "gate closed" in no_wait handler ignored INFO 2021-11-16 03:04:03,194 [shard 0] rpc - client 127.0.82.1:54946 msg_id 757: exception "gate closed" in no_wait handler ignored INFO 2021-11-16 03:04:04,422 [shard 0] rpc - client 127.0.82.2:63480 msg_id 768: exception "gate closed" in no_wait handler ignored INFO 2021-11-16 03:04:05,160 [shard 0] gossip - Disable and wait for gossip loop started INFO 2021-11-16 03:04:05,161 [shard 0] gossip - Gossip is now stopped INFO 2021-11-16 03:04:05,161 [shard 0] storage_service - Stop transport: stop_gossiping done INFO 2021-11-16 03:04:05,161 [shard 0] messaging_service - Stopping nontls server INFO 2021-11-16 03:04:05,161 [shard 0] messaging_service - Stopping tls server INFO 2021-11-16 03:04:05,161 [shard 0] messaging_service - Stopping client for address: 127.0.82.1:0 INFO 2021-11-16 03:04:05,161 [shard 0] messaging_service - Stopping client for address: 127.0.82.2:0 INFO 2021-11-16 03:04:05,161 [shard 0] messaging_service - Stopping client for address: 127.0.82.1:0 INFO 2021-11-16 03:04:05,161 [shard 0] messaging_service - Stopping client for address: 127.0.82.2:0 INFO 2021-11-16 03:04:05,162 [shard 0] messaging_service - Stopping client for address: 127.0.82.1:0 INFO 2021-11-16 03:04:05,162 [shard 0] messaging_service - Stopping client for address: 127.0.82.2:0 INFO 2021-11-16 03:04:05,162 [shard 0] messaging_service - Stopping client for address: 127.0.82.1:0 INFO 2021-11-16 03:04:05,162 [shard 0] messaging_service - Stopping client for address: 127.0.82.2:0 INFO 2021-11-16 03:04:05,162 [shard 0] messaging_service - Stopping client for address: 127.0.82.1:0 INFO 2021-11-16 03:04:05,162 [shard 0] messaging_service - Stopping client for address: 127.0.82.2:0 INFO 2021-11-16 03:04:05,162 [shard 0] messaging_service - Stopping client for address: 127.0.82.1:0 INFO 2021-11-16 03:04:05,162 [shard 0] messaging_service - Stopping client for address: 127.0.82.2:0 INFO 2021-11-16 03:04:05,164 [shard 0] messaging_service - Stopping client for address: 127.0.82.2:0 - Done INFO 2021-11-16 03:04:05,165 [shard 1] messaging_service - Stopping nontls server INFO 2021-11-16 03:04:05,165 [shard 1] messaging_service - Stopping tls server INFO 2021-11-16 03:04:05,165 [shard 0] messaging_service - Stopping client for address: 127.0.82.2:0 - Done INFO 2021-11-16 03:04:05,165 [shard 1] messaging_service - Stopping client for address: 127.0.82.1:0 INFO 2021-11-16 03:04:05,165 [shard 1] messaging_service - Stopping client for address: 127.0.82.1:1 INFO 2021-11-16 03:04:05,165 [shard 1] messaging_service - Stopping client for address: 127.0.82.2:1 INFO 2021-11-16 03:04:05,165 [shard 1] messaging_service - Stopping client for address: 127.0.82.1:1 INFO 2021-11-16 03:04:05,165 [shard 1] messaging_service - Stopping client for address: 127.0.82.2:1 INFO 2021-11-16 03:04:05,165 [shard 0] messaging_service - Stopping client for address: 127.0.82.1:0 - Done INFO 2021-11-16 03:04:05,165 [shard 0] messaging_service - Stopping tls server - Done INFO 2021-11-16 03:04:05,166 [shard 0] messaging_service - Stopping client for address: 127.0.82.1:0 - Done INFO 2021-11-16 03:04:05,166 [shard 1] messaging_service - Stopping tls server - Done INFO 2021-11-16 03:04:05,167 [shard 1] messaging_service - Stopping client for address: 127.0.82.2:1 - Done INFO 2021-11-16 03:04:05,167 [shard 1] messaging_service - Stopping client for address: 127.0.82.1:1 - Done INFO 2021-11-16 03:04:05,167 [shard 1] messaging_service - Stopping client for address: 127.0.82.1:0 - Done INFO 2021-11-16 03:04:05,168 [shard 0] messaging_service - Stopping client for address: 127.0.82.1:0 - Done INFO 2021-11-16 03:04:05,168 [shard 0] messaging_service - Stopping client for address: 127.0.82.2:0 - Done INFO 2021-11-16 03:04:05,168 [shard 0] messaging_service - Stopping client for address: 127.0.82.1:0 - Done INFO 2021-11-16 03:04:05,168 [shard 1] messaging_service - Stopping client for address: 127.0.82.2:1 - Done INFO 2021-11-16 03:04:05,168 [shard 0] messaging_service - Stopping client for address: 127.0.82.2:0 - Done INFO 2021-11-16 03:04:05,169 [shard 0] messaging_service - Stopping client for address: 127.0.82.1:0 - Done INFO 2021-11-16 03:04:05,169 [shard 1] messaging_service - Stopping client for address: 127.0.82.1:1 - Done INFO 2021-11-16 03:04:05,169 [shard 0] messaging_service - Stopping client for address: 127.0.82.1:0 - Done INFO 2021-11-16 03:04:05,169 [shard 0] messaging_service - Stopping client for address: 127.0.82.2:0 - Done INFO 2021-11-16 03:04:05,169 [shard 0] messaging_service - Stopping client for address: 127.0.82.2:0 - Done INFO 2021-11-16 03:04:05,170 [shard 0] messaging_service - Stopping nontls server - Done INFO 2021-11-16 03:04:05,170 [shard 1] messaging_service - Stopping nontls server - Done INFO 2021-11-16 03:04:05,170 [shard 0] storage_service - messaging_service stopped INFO 2021-11-16 03:04:05,170 [shard 0] storage_service - Stop transport: shutdown messaging_service done INFO 2021-11-16 03:04:05,171 [shard 0] storage_service - stream_manager stopped INFO 2021-11-16 03:04:05,171 [shard 0] storage_service - Stop transport: shutdown stream_manager done INFO 2021-11-16 03:04:05,171 [shard 0] storage_service - Stop transport: done INFO 2021-11-16 03:04:05,171 [shard 0] tracing - Asked to shut down INFO 2021-11-16 03:04:05,171 [shard 0] tracing - Tracing is down INFO 2021-11-16 03:04:05,171 [shard 1] tracing - Asked to shut down INFO 2021-11-16 03:04:05,171 [shard 1] tracing - Tracing is down INFO 2021-11-16 03:04:05,171 [shard 0] migration_manager - stopping migration service INFO 2021-11-16 03:04:05,172 [shard 1] migration_manager - stopping migration service INFO 2021-11-16 03:04:09,842 [shard 0] init - Shutting down local storage was successful INFO 2021-11-16 03:04:09,842 [shard 0] init - Shutting down view update generator INFO 2021-11-16 03:04:09,842 [shard 0] view_update_generator - leaving 0 unstaged sstables unprocessed INFO 2021-11-16 03:04:09,842 [shard 1] view_update_generator - leaving 0 unstaged sstables unprocessed INFO 2021-11-16 03:04:09,843 [shard 0] init - Shutting down view update generator was successful INFO 2021-11-16 03:04:09,843 [shard 0] init - Shutting down service level controller update loop INFO 2021-11-16 03:04:18,266 [shard 0] service_level_controller - update_from_distributed_data: configuration polling loop aborted INFO 2021-11-16 03:04:18,266 [shard 0] init - Shutting down service level controller update loop was successful INFO 2021-11-16 03:04:18,266 [shard 0] init - Shutting down repair INFO 2021-11-16 03:04:18,266 [shard 0] init - Shutting down repair was successful INFO 2021-11-16 03:04:18,266 [shard 0] init - Shutting down rpc controller API INFO 2021-11-16 03:04:18,266 [shard 0] init - Shutting down rpc controller API was successful INFO 2021-11-16 03:04:18,267 [shard 0] init - Shutting down transport controller API INFO 2021-11-16 03:04:18,267 [shard 0] init - Shutting down transport controller API was successful INFO 2021-11-16 03:04:18,267 [shard 0] init - Shutting down native transport INFO 2021-11-16 03:04:18,267 [shard 0] init - Shutting down native transport was successful INFO 2021-11-16 03:04:18,267 [shard 0] init - Shutting down view builder API INFO 2021-11-16 03:04:18,267 [shard 0] init - Shutting down view builder API was successful INFO 2021-11-16 03:04:18,267 [shard 0] init - Shutting down view builder INFO 2021-11-16 03:04:18,267 [shard 0] view - Stopping view builder INFO 2021-11-16 03:04:18,267 [shard 1] view - Stopping view builder INFO 2021-11-16 03:04:18,268 [shard 0] init - Shutting down view builder was successful INFO 2021-11-16 03:04:18,268 [shard 0] init - Shutting down hinted handoff API INFO 2021-11-16 03:04:18,268 [shard 0] init - Shutting down hinted handoff API was successful INFO 2021-11-16 03:04:18,268 [shard 0] init - Shutting down view update backlog broker INFO 2021-11-16 03:04:18,269 [shard 0] init - Shutting down view update backlog broker was successful INFO 2021-11-16 03:04:18,269 [shard 0] init - Shutting down cf cache hit rate calculator INFO 2021-11-16 03:04:18,269 [shard 0] init - Shutting down cf cache hit rate calculator was successful INFO 2021-11-16 03:04:18,269 [shard 0] init - Shutting down load meter INFO 2021-11-16 03:04:18,270 [shard 0] init - Shutting down load meter was successful INFO 2021-11-16 03:04:18,270 [shard 0] init - Shutting down batchlog manager INFO 2021-11-16 03:04:18,270 [shard 0] init - Shutting down batchlog manager was successful INFO 2021-11-16 03:04:18,270 [shard 0] init - Shutting down snapshots API INFO 2021-11-16 03:04:18,270 [shard 0] init - Shutting down snapshots API was successful INFO 2021-11-16 03:04:18,270 [shard 0] init - Shutting down snapshots INFO 2021-11-16 03:04:18,271 [shard 0] init - Shutting down snapshots was successful INFO 2021-11-16 03:04:18,271 [shard 0] init - Shutting down auth service INFO 2021-11-16 03:04:18,271 [shard 0] init - Shutting down auth service was successful INFO 2021-11-16 03:04:18,271 [shard 0] init - Shutting down tracing INFO 2021-11-16 03:04:18,271 [shard 0] init - Shutting down tracing was successful INFO 2021-11-16 03:04:18,272 [shard 0] init - Shutting down storage service notifications INFO 2021-11-16 03:04:18,272 [shard 0] init - Shutting down storage service notifications was successful INFO 2021-11-16 03:04:18,272 [shard 0] init - Shutting down system distributed keyspace INFO 2021-11-16 03:04:18,272 [shard 0] init - Shutting down system distributed keyspace was successful INFO 2021-11-16 03:04:18,272 [shard 0] init - Shutting down storage service notifications INFO 2021-11-16 03:04:18,272 [shard 0] init - Shutting down storage service notifications was successful INFO 2021-11-16 03:04:18,272 [shard 0] init - Shutting down sstables loader API INFO 2021-11-16 03:04:18,272 [shard 0] init - Shutting down sstables loader API was successful INFO 2021-11-16 03:04:18,272 [shard 0] init - Shutting down sstables loader INFO 2021-11-16 03:04:18,273 [shard 0] init - Shutting down sstables loader was successful INFO 2021-11-16 03:04:18,273 [shard 0] init - Shutting down repair API INFO 2021-11-16 03:04:18,273 [shard 0] init - Shutting down repair API was successful INFO 2021-11-16 03:04:18,273 [shard 0] init - Shutting down messaging service API INFO 2021-11-16 03:04:18,273 [shard 0] init - Shutting down messaging service API was successful INFO 2021-11-16 03:04:18,273 [shard 0] init - Shutting down storage service messaging INFO 2021-11-16 03:04:18,273 [shard 0] init - Shutting down storage service messaging was successful INFO 2021-11-16 03:04:18,274 [shard 0] init - Shutting down cdc log service INFO 2021-11-16 03:04:18,274 [shard 0] init - Shutting down cdc log service was successful INFO 2021-11-16 03:04:18,274 [shard 0] init - Shutting down CDC Generation Management service INFO 2021-11-16 03:04:18,275 [shard 0] init - Shutting down CDC Generation Management service was successful INFO 2021-11-16 03:04:18,275 [shard 0] init - Shutting down repair service INFO 2021-11-16 03:04:18,276 [shard 0] init - Shutting down repair service was successful INFO 2021-11-16 03:04:18,276 [shard 0] init - Shutting down drain storage proxy INFO 2021-11-16 03:04:18,277 [shard 0] hints_manager - Asked to stop INFO 2021-11-16 03:04:18,277 [shard 0] hints_manager - Asked to stop INFO 2021-11-16 03:04:18,277 [shard 1] hints_manager - Asked to stop INFO 2021-11-16 03:04:18,277 [shard 1] hints_manager - Asked to stop INFO 2021-11-16 03:04:18,277 [shard 0] hints_manager - Stopped INFO 2021-11-16 03:04:18,277 [shard 0] hints_manager - Stopped INFO 2021-11-16 03:04:18,277 [shard 1] hints_manager - Stopped INFO 2021-11-16 03:04:18,278 [shard 1] hints_manager - Stopped INFO 2021-11-16 03:04:18,278 [shard 0] init - Shutting down drain storage proxy was successful INFO 2021-11-16 03:04:18,278 [shard 0] init - Shutting down streaming service INFO 2021-11-16 03:04:18,278 [shard 0] init - Shutting down streaming service was successful INFO 2021-11-16 03:04:18,278 [shard 0] init - Shutting down storage proxy RPC verbs INFO 2021-11-16 03:04:18,279 [shard 0] init - Shutting down storage proxy RPC verbs was successful INFO 2021-11-16 03:04:18,279 [shard 0] init - Shutting down sstables format selector INFO 2021-11-16 03:04:18,279 [shard 0] init - Shutting down sstables format selector was successful INFO 2021-11-16 03:04:18,279 [shard 0] init - Shutting down migration manager INFO 2021-11-16 03:04:18,279 [shard 0] init - Shutting down migration manager was successful INFO 2021-11-16 03:04:18,280 [shard 0] init - Shutting down database INFO 2021-11-16 03:04:18,284 [shard 0] compaction_manager - Stopping 0 ongoing compactions for table alternator_OOUYJ0AIEP.OOUYJ0AIEP due to column family removal INFO 2021-11-16 03:04:18,285 [shard 1] compaction_manager - Stopping 0 ongoing compactions for table system_schema.scylla_tables due to column family removal INFO 2021-11-16 03:04:18,285 [shard 0] compaction_manager - Stopping 0 ongoing compactions for table alternator_X79BIKTD05.X79BIKTD05 due to column family removal INFO 2021-11-16 03:04:18,285 [shard 1] compaction_manager - Stopping 0 ongoing compactions for table alternator_user_table.user_table due to column family removal INFO 2021-11-16 03:04:18,285 [shard 0] compaction_manager - Stopping 0 ongoing compactions for table system_traces.sessions_time_idx due to column family removal INFO 2021-11-16 03:04:18,285 [shard 1] compaction_manager - Stopping 1 ongoing compactions for table alternator_9CV3PZB15I.9CV3PZB15I due to column family removal INFO 2021-11-16 03:04:18,285 [shard 1] compaction_manager - Stopping 1 ongoing compactions for table system_auth.role_members due to column family removal INFO 2021-11-16 03:04:18,285 [shard 0] compaction_manager - Stopping 1 ongoing compactions for table system_auth.roles due to column family removal INFO 2021-11-16 03:04:18,286 [shard 0] compaction_manager - Stopping 0 ongoing compactions for table system_schema.indexes due to column family removal INFO 2021-11-16 03:04:18,286 [shard 1] compaction_manager - Stopping 0 ongoing compactions for table system_schema.view_virtual_columns due to column family removal INFO 2021-11-16 03:04:18,286 [shard 1] compaction_manager - Stopping 0 ongoing compactions for table system_schema.columns due to column family removal INFO 2021-11-16 03:04:18,286 [shard 1] compaction_manager - Stopping 0 ongoing compactions for table system_schema.computed_columns due to column family removal INFO 2021-11-16 03:04:18,287 [shard 0] compaction_manager - Stopping 0 ongoing compactions for table alternator_W1YS405NQX.W1YS405NQX due to column family removal INFO 2021-11-16 03:04:18,289 [shard 0] compaction_manager - Stopping 0 ongoing compactions for table system_schema.aggregates due to column family removal INFO 2021-11-16 03:04:18,290 [shard 1] compaction_manager - Stopping 0 ongoing compactions for table system_schema.dropped_columns due to column family removal INFO 2021-11-16 03:04:18,290 [shard 0] compaction_manager - Stopping 0 ongoing compactions for table alternator_user_table.user_table due to column family removal INFO 2021-11-16 03:04:18,290 [shard 1] compaction_manager - Stopping 0 ongoing compactions for table alternator_UF2ZKI188B.UF2ZKI188B due to column family removal scylla: compaction/compaction_manager.cc:1000: future<> compaction_manager::remove(column_family ): Assertion `std::find_if(_tasks.begin(), _tasks.end(), [cf] (auto& task) { return task->compacting_cf == cf; }) == _tasks.end()' failed. scylla: compaction/compaction_manager.cc:1000: future<> compaction_manager::remove(column_family ): Assertion `std::find_if(_tasks.begin(), _tasks.end(), [cf] (auto& task) { return task->compacting_cf == cf; }) == _tasks.end()' failed. Aborting on shard 1. Backtrace: 0x12b4f6d0 0x18f2d83d 0x18f2d43c 0x18d9b2c4 0x18dc9c5e 0x18e9a47c 0x18e9a6cd 0x18e9a4fa 0x7f4101f18a1f /jenkins/workspace/scylla-master/dtest-debug-random/scylla/.ccm/scylla-repository/9d4262e264efd6d92de9c40545f59c85c840de0d/scylla/libreloc/libc.so.6+0x3d2a1 /jenkins/workspace/scylla-master/dtest-debug-random/scylla/.ccm/scylla-repository/9d4262e264efd6d92de9c40545f59c85c840de0d/scylla/libreloc/libc.so.6+0x268a3 /jenkins/workspace/scylla-master/dtest-debug-random/scylla/.ccm/scylla-repository/9d4262e264efd6d92de9c40545f59c85c840de0d/scylla/libreloc/libc.so.6+0x26788 /jenkins/workspace/scylla-master/dtest-debug-random/scylla/.ccm/scylla-repository/9d4262e264efd6d92de9c40545f59c85c840de0d/scylla/libreloc/libc.so.6+0x35a15 0x145ba858 0x131431bd 0x13142f98 0x18dbcdae 0x18dc3419 0x18dc80c0 0x18e9c98c 0x18e9b5f2 0x18e9b4a2 0x18e9a932 0x1348531a 0x18cf4c3b /jenkins/workspace/scylla-master/dtest-debug-random/scylla/.ccm/scylla-repository/9d4262e264efd6d92de9c40545f59c85c840de0d/scylla/libreloc/libpthread.so.0+0x9298 /jenkins/workspace/scylla-master/dtest-debug-random/scylla/.ccm/scylla-repository/9d4262e264efd6d92de9c40545f59c85c840de0d/scylla/libreloc/libc.so.6+0x100352

bhalevy commented 3 years ago

Might be related to #9631 (or might not, we don't know yet)

raphaelsc commented 3 years ago

seems to me like a dup of #9631.

bhalevy commented 3 years ago

@raphaelsc I have a fix that I'm testing...

compaction_manager: submit_offstrategy: remove task in finally clause

Now, when the offstrategy task is stopped, it exits the repeat
loop if (!can_proceed(task)) without going through
_tasks.remove(task) - causing the assert in compaction_manger::remove
to trip, as stop_ongoing_compactions will be resolved
while the task is still listed in _tasks.

diff --git a/compaction/compaction_manager.cc b/compaction/compaction_manager.cc
index b3fe8ccf5..1b9e18357 100644
--- a/compaction/compaction_manager.cc
+++ b/compaction/compaction_manager.cc
@@ -747,11 +747,12 @@ void compaction_manager::submit_offstrategy(column_family* cf) {
                             return make_ready_future<stop_iteration>(stop_iteration::no);
                         });
                     }
-                    _tasks.remove(task);
                     return make_ready_future<stop_iteration>(stop_iteration::yes);
                 });
             });
         });
+    }).finally([this, task] {
+        _tasks.remove(task);
     });
 }
raphaelsc commented 3 years ago 
_tasks.remove(task);

submit_offstrategy() is assuming that it would only be stopped when off strategy actually started, which is wrong. if stopped earlier, task is never removed from list. That's why this problem has only been caught with dtest. I wonder why this problem never manifested before as it's a preexisting issue. Perhaps that series of mine made it more reproduceable.

bhalevy commented 3 years ago 
_tasks.remove(task);

submit_offstrategy() is assuming that it would only be stopped when off strategy actually started, which is wrong. if stopped earlier, task is never removed from list. That's why this problem has only been caught with dtest. I wonder why this problem never manifested before as it's a preexisting issue. Perhaps that series of mine made it more reproduceable.

Agreed. It looks like a preexisting issue. We stumbled upon in it with randomized testing in debug mode so the test is not run regularly in debug mode, with the additional assertion.

raphaelsc commented 3 years ago 
_tasks.remove(task);

submit_offstrategy() is assuming that it would only be stopped when off strategy actually started, which is wrong. if stopped earlier, task is never removed from list. That's why this problem has only been caught with dtest. I wonder why this problem never manifested before as it's a preexisting issue. Perhaps that series of mine made it more reproduceable.

Agreed. It looks like a preexisting issue. We stumbled upon in it with randomized testing in debug mode so the test is not run regularly in debug mode, with the additional assertion.

thanks a lot for the bisect and investigation.

mykaul commented 1 year ago

Probably too old to backport.