search

scylladb / scylladb

NoSQL data store using the seastar framework, compatible with Apache Cassandra
http://scylladb.com
GNU Affero General Public License v3.0
13.23k stars 1.25k forks source link

C-s failed during rolling cleanup on nodes with java.io.IOException: Operation x10 on key(s) Not enough replicas available for query at consistency QUORUM (2 required but only 1 alive) #16346

Open aleksbykov opened 9 months ago

aleksbykov commented 9 months ago

Issue description

The test job ran with instance type n2-highmem-16 on gce backend. The cluster contained 6 nodes. C-S commands used as stress tools

cassandra-stress write cl=QUORUM duration=180m -schema 'replication(strategy=NetworkTopologyStrategy,replic
ation_factor=3) compaction(strategy=SizeTieredCompactionStrategy)' -mode cql3 native -rate threads=1000 -pop seq=1..10000000 -log interval=5

When nemesis disrupt_memory_stress (it tried to fill memory with command stress-ng --vm-bytes $(awk '/MemTotal/{printf "%d\n", $2 * 0.9;}' < /proc/meminfo)k --vm-keep -m 1 -t 100) ran on node2, c-s started to report next errors:

< t:2023-11-22 08:14:42,545 f:base.py         l:228  c:RemoteLibSSH2CmdRunner p:DEBUG > com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during SIMPLE write query at consistency QUORUM (2 replica were required but only 0 acknowledged the write)
< t:2023-11-22 08:14:42,595 f:base.py         l:228  c:RemoteLibSSH2CmdRunner p:DEBUG > com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during SIMPLE write query at consistency QUORUM (2 replica were required but only 1 acknowledged the write)
< t:2023-11-22 08:14:43,019 f:base.py         l:228  c:RemoteLibSSH2CmdRunner p:DEBUG > total,      97865161,    7759,    7759,    7759,   117.8,     2.7,   541.1,  2854.2,  4311.7,  4412.4, 2585.0,  0.02343,      0,      0,       0,       0,       0,       0
< t:2023-11-22 08:14:43,019 f:base.py         l:228  c:RemoteLibSSH2CmdRunner p:DEBUG > com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during SIMPLE write query at consistency QUORUM (2 replica were required but only 0 acknowledged the write)
< t:2023-11-22 08:14:43,045 f:base.py         l:228  c:RemoteLibSSH2CmdRunner p:DEBUG > com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during SIMPLE write query at consistency QUORUM (2 replica were required but only 0 acknowledged the write)

It is not clear, why Simple write failed without, because only 1 node is under nemesis and rest nodes are not busy. The possible reason could be that at this moment node2 which under MemoryStress nemesis was coordinator

Nemesis finished without errors. Node2 reported reactor stalls and kernel stack, but they were expected, because node doesn't have enough memory.

Next nemesis was disrupt_destroy_data_then_rebuild no the same node2. The nemesis stopped scylla, removed sstables, started scylla and ran rebuild on the node. The nemesis finished successful also, but c-s command was reporting same error:

com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during SIMPLE write query at consistency QUORUM (2 replica were required but only 1 acknowledged the write)

Next nemesis was decommisssion_streaming_error on node3. The nemesis start decommission on node3 and reboot node3 when streaming is running to abort decommission.

Decommission was started on node3:

2023-11-22T08:25:32.265+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-3     !INFO | scylla[12900]:  [shard  0:strm] api - decommission
2023-11-22T08:25:32.265+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-3     !INFO | scylla[12900]:  [shard  0:strm] node_ops - decommission[72f7937d-5bb8-4dfd-ac81-518aeaf33b77]: Started decommission operation: node=28d425f4-d33a-4b84-83fd-e8681ea84a41/10.142.0.111
2023-11-22T08:25:32.265+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-3     !INFO | scylla[12900]:  [shard  0:strm] node_ops - decommission[72f7937d-5bb8-4dfd-ac81-518aeaf33b77]: sync_nodes={10.142.0.6, 10.142.0.114, 10.142.0.12, 10.142.0.112, 10.142.0.8, 10.142.0.111}, ignore_nodes={}
2023-11-22T08:25:32.265+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-3     !INFO | scylla[12900]:  [shard  0:strm] storage_service - DECOMMISSIONING: starts
2023-11-22T08:25:32.265+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-3     !INFO | scylla[12900]:  [shard  0:strm] raft_group0 - Performing a group 0 read barrier...
2023-11-22T08:25:32.265+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-3     !INFO | scylla[12900]:  [shard  0:strm] raft_group0 - Finished group 0 read barrier.
2023-11-22T08:25:32.265+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-3     !INFO | scylla[12900]:  [shard  0:strm] node_ops - decommission[72f7937d-5bb8-4dfd-ac81-518aeaf33b77]: Started heartbeat_updater (interval=10s)
2023-11-22T08:25:32.265+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-3     !INFO | scylla[12900]:  [shard  0:strm] node_ops - decommission[72f7937d-5bb8-4dfd-ac81-518aeaf33b77]: Started decommission_prepare[72f7937d-5bb8-4dfd-ac81-518aeaf33b77]: ignore_nodes={}, leaving_nodes={10.142.0.111}, replace_nodes={}, bootstrap_nodes={}, repair_tables={}
2023-11-22T08:25:32.265+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-3     !INFO | scylla[12900]:  [shard  0:strm] storage_service - decommission[72f7937d-5bb8-4dfd-ac81-518aeaf33b77]: Added node=10.142.0.111 as leaving node, coordinator=10.142.0.111
2023-11-22T08:25:32.265+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-3     !INFO | scylla[12900]:  [shard  0:strm] node_ops - decommission[72f7937d-5bb8-4dfd-ac81-518aeaf33b77]: Finished decommission_prepare[72f7937d-5bb8-4dfd-ac81-518aeaf33b77]: ignore_nodes={}, leaving_nodes={10.142.0.111}, replace_nodes={}, bootstrap_nodes={}, repair_tables={}
2023-11-22T08:25:32.265+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-3     !INFO | scylla[12900]:  [shard  0:strm] storage_service - DECOMMISSIONING: unbootstrap starts
2023-11-22T08:25:32.265+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-3     !INFO | scylla[12900]:  [shard  0:strm] storage_service - Started batchlog replay for decommission
2023-11-22T08:25:32.265+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-3     !INFO | scylla[12900]:  [shard  0:strm] storage_service - Finished batchlog replay for decommission
2023-11-22T08:25:32.265+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-3     !INFO | scylla[12900]:  [shard  0:strm] storage_service - enable_repair_based_node_ops=true, allowed_repair_based_node_ops={decommission ,bootstrap ,rebuild ,removenode ,replace}
2023-11-22T08:25:32.265+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-3     !INFO | scylla[12900]:  [shard  0:strm] repair - decommission_with_repair: started with keyspaces={ks_truncate, keyspace1, system_a

When repair_on_decommission had been running, node3 was hard rebooted:

2023-11-22T08:25:59.336+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-3     !INFO | scylla[12900]:  [shard  0:strm] repair - decommission_with_repair: keyspace=ks_truncate, range=(-2049961259577721615,-20201
51127177670419], current_replica_endpoints={10.142.0.114, 10.142.0.112, 10.142.0.8, 10.142.0.111, 10.142.0.12, 10.142.0.6}, new_replica_endpoints={10.142.0.114, 10.142.0.112, 10.142.0.8, 10.142.0.12, 10.142.0.6}
, neighbors={10.142.0.114}, mandatory_neighbor={}
2023-11-22T08:25:59.336+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-3     !INFO | scylla[12900]:  [shard  0:strm] repair - decommission_with_repair: keyspace=ks_truncate, range=(-2020151127177670419,-20125
52387951799668], current_replica_endpoints={10.142.0.114, 10.142.0.112, 10.142.0.8, 10.142.0.111, 10.142.0.12, 10.142.0.6}, new_replica_endpoints={10.142.0.114, 10.142.0.112, 10.142.0.8, 10.142.0.12, 10.142.0.6}
, neighbors={10.142.0.114}, mandatory_neighbor={}
2023-11-22T08:25:59.336+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-3   !NOTICE | kernel: Linux version 5.15.0-1046-gcp (buildd@lcy02-amd64-080) (gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, GNU ld (GNU Binu
tils for Ubuntu) 2.38) #54-Ubuntu SMP Wed Oct 11 17:49:45 UTC 2023 (Ubuntu 5.15.0-1046.54-gcp 5.15.126)
2023-11-22T08:25:59.336+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-3     !INFO | kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-5.15.0-1046-gcp root=PARTUUID=3c15a266-dbf8-468b-a117-d3f0ebfe2014 ro consol
e=ttyS0 net.ifnames=0 clocksource=tsc tsc=reliable intel_idle.max_cstate=1 processor.max_cstate=1 panic=-1
2023-11-22T08:25:59.336+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-3     !INFO | kernel: KERNEL supported cpus:
2023-11-22T08:25:59.336+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-3     !INFO | kernel:   Intel GenuineIntel
2023-11-22T08:25:59.336+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-3     !INFO | kernel:   AMD AuthenticAMD
2023-11-22T08:25:59.336+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-3     !INFO | kernel:   Hygon HygonGenuine
2023-11-22T08:25:59.336+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-3     !INFO | kernel:   Centaur CentaurHauls
2023-11-22T08:25:59.336+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-3     !INFO | kernel:   zhaoxin   Shanghai  
2023-11-22T08:25:59.336+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-3     !INFO | kernel: BIOS-provided physical RAM map:
2023-11-22T08:25:59.336+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-3     !INFO | kernel: BIOS-e820: [mem 0x0000000000000000-0x0000000000000fff] reserved
2023-11-22T08:25:59.336+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-3     !INFO | kernel: BIOS-e820: [mem 0x0000000000001000-0x0000000000054fff] usable
2023-11-22T08:25:59.336+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-3     !INFO | kernel: BIOS-e820: [mem 0x0000000000055000-0x000000000005ffff] reserved
2023-11-22T08:25:59.336+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-3     !INFO | kernel: BIOS-e820: [mem 0x0000000000060000-0x0000000000097fff] usable

Node3 started BOOTSTRAP process:

2023-11-22T08:26:58.927+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-3     !INFO | scylla[1022]:  [shard  0:strm] storage_service - entering JOINING mode
2023-11-22T08:26:58.927+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-3     !INFO | scylla[1022]:  [shard  0:strm] storage_service - waiting for ring information
2023-11-22T08:26:58.927+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-3     !INFO | scylla[1022]:  [shard  0:strm] storage_service - waiting for schema information to complete
2023-11-22T08:26:58.927+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-3     !INFO | scylla[1022]:  [shard  0:strm] migration_manager - Schema agreement check passed.
2023-11-22T08:26:58.927+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-3     !INFO | scylla[1022]:  [shard  0:comp] compaction - [Compact system.scylla_local e72381d0-8910-11ee-bfdf-6413f9d272e0] Compacted 2 
sstables to [/var/lib/scylla/data/system/scylla_local-2972ec7ffb2038ddaac1d876f2e3fcbd/me-3gba_0ngy_3863k2x1j57svinjz4-big-Data.db:level=0]. 12kB to 6kB (~51% of original) in 51ms = 241kB/s. ~256 total partition
s merged to 4.
2023-11-22T08:26:58.927+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-3     !INFO | scylla[1022]:  [shard  0:strm] storage_service - Checking bootstrapping/leaving nodes: ok
2023-11-22T08:26:58.927+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-3     !INFO | scylla[1022]:  [shard  0:strm] storage_service - getting bootstrap token
2023-11-22T08:26:58.928+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-3     !INFO | scylla[1022]:  [shard  0:strm] boot_strapper - Get random bootstrap_tokens={6796182058870355883, 7918571834954260657, 2738208566603548173, -2341549841774131149, 3848538405347755177, 4824596510557590327, 8208442805485751348, -2112115250639980470, 7535520270001286764, 6905514665629983163, 6181174230441134343, 3583727169511484059, -290
...

2023-11-22T08:26:58.928+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-3     !INFO | scylla[1022]:  [shard  0:strm] system_distributed_keyspace - system_distributed(_everywhere) keyspaces and tables are up-to-date. Not creating
2023-11-22T08:26:58.928+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-3     !INFO | scylla[1022]:  [shard  0:strm] storage_service - enable_repair_based_node_ops=true, allowed_repair_based_node_ops={decommission ,bootstrap ,rebuild ,removenode ,replace}
2023-11-22T08:26:58.928+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-3     !INFO | scylla[1022]:  [shard  0:strm] storage_service - entering BOOTSTRAP mode
2023-11-22T08:26:58.928+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-3     !INFO | scylla[1022]:  [shard  0:strm] storage_service - Wait until local node knows tokens of peer nodes
2023-11-22T08:26:58.928+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-3     !INFO | scylla[1022]:  [shard  0:strm] gossip - Waiting for pending range setup...

Is it correct behavior? it was expected that node should be just rebooted, because it was not removed from token ring yet.

And while node3 had been decommissioning and rebooted, c-s reported Connection dropping errors:

 java.io.IOException: Connection reset by peer

This could be related to c-s threads, which was closed by node3 while it rebooted

Then set of nemesis ran without errors. The latest nemesis was remove_node_then_add_node. The nemesis terminates target node, then the nemesis removes it with removenode and runs repair on each node, and bootstraps new node and runs cleanup on each nodes. After node4 was terminated and removed, node9 was bootstrapped, but once cleanup started on node9, c-s failed with critical errors:

< t:2023-11-22 09:51:33,690 f:base.py         l:228  c:RemoteLibSSH2CmdRunner p:DEBUG > com.datastax.driver.core.exceptions.OperationTimedOutException: [/10.142.0.15:9042] Timed out waiting for server response
< t:2023-11-22 09:51:33,691 f:base.py         l:228  c:RemoteLibSSH2CmdRunner p:DEBUG > java.io.IOException: Operation x10 on key(s) [394e324f4c34334d3031]: Error executing: (UnavailableException): Not enough replicas available for query at consistency QUORUM (2 required but only 1 alive)
< t:2023-11-22 09:51:33,697 f:base.py         l:228  c:RemoteLibSSH2CmdRunner p:DEBUG > com.datastax.driver.core.exceptions.OperationTimedOutException: [/10.142.0.111:9042] Timed out waiting for server response
< t:2023-11-22 09:51:33,703 f:base.py         l:228  c:RemoteLibSSH2CmdRunner p:DEBUG > 
< t:2023-11-22 09:51:33,703 f:base.py         l:228  c:RemoteLibSSH2CmdRunner p:DEBUG > com.datastax.driver.core.exceptions.OperationTimedOutException: [longevity-10gb-3h-master-db-node-ef89d7e4-0-1.c.sct-project-1.internal/10.142.0.8:9042] Timed out waiting for server response
< t:2023-11-22 09:51:33,703 f:base.py         l:228  c:RemoteLibSSH2CmdRunner p:DEBUG >     at org.apache.cassandra.stress.Operation.error(Operation.java:141)
< t:2023-11-22 09:51:33,703 f:base.py         l:228  c:RemoteLibSSH2CmdRunner p:DEBUG > com.datastax.driver.core.exceptions.OperationTimedOutException: [/10.142.0.6:9042] Timed out waiting for server response
< t:2023-11-22 09:51:33,703 f:base.py         l:228  c:RemoteLibSSH2CmdRunner p:DEBUG >     at org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:119)
< t:2023-11-22 09:51:33,703 f:base.py         l:228  c:RemoteLibSSH2CmdRunner p:DEBUG > com.datastax.driver.core.exceptions.OperationTimedOutException: [/10.142.0.111:9042] Timed out waiting for server response
< t:2023-11-22 09:51:33,703 f:base.py         l:228  c:RemoteLibSSH2CmdRunner p:DEBUG >     at org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:101)
< t:2023-11-22 09:51:33,703 f:base.py         l:228  c:RemoteLibSSH2CmdRunner p:DEBUG > com.datastax.driver.core.exceptions.OperationTimedOutException: [/10.142.0.111:9042] Timed out waiting for server response
< t:2023-11-22 09:51:33,704 f:base.py         l:228  c:RemoteLibSSH2CmdRunner p:DEBUG >     at org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:109)
< t:2023-11-22 09:51:33,710 f:base.py         l:228  c:RemoteLibSSH2CmdRunner p:DEBUG > com.datastax.driver.core.exceptions.OperationTimedOutException: [/10.142.0.111:9042] Timed out waiting for server response
< t:2023-11-22 09:51:33,710 f:base.py         l:228  c:RemoteLibSSH2CmdRunner p:DEBUG >     at org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:264)
< t:2023-11-22 09:51:33,710 f:base.py         l:228  c:RemoteLibSSH2CmdRunner p:DEBUG > com.datastax.driver.core.exceptions.OperationTimedOutException: [longevity-10gb-3h-master-db-node-ef89d7e4-0-1.c.sct-project-1.internal/10.142.0.8:9042] Timed out waiting for server response
< t:2023-11-22 09:51:33,711 f:base.py         l:228  c:RemoteLibSSH2CmdRunner p:DEBUG >     at org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:473)
< t:2023-11-22 09:51:33,711 f:base.py         l:228  c:RemoteLibSSH2CmdRunner p:DEBUG > com.datastax.driver.core.exceptions.OperationTimedOutException: [/10.142.0.6:9042] Timed out waiting for server response

Logs on all nodes at this time shows that some nodes in cluster goes down:

 grep "2023-11-22T09:51:3" db-cluster-ef89d7e4/longevity-10gb-3h-master-db-node-ef89d7e4-0-*/messages.log
db-cluster-ef89d7e4/longevity-10gb-3h-master-db-node-ef89d7e4-0-1/messages.log:2023-11-22T09:51:31.153+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-1   !NOTICE | sudo[36229]: scylla-test : PWD=/home/scylla-test ; USER=root ; COMMAND=/usr/bin/coredumpctl -q --json=short
db-cluster-ef89d7e4/longevity-10gb-3h-master-db-node-ef89d7e4-0-1/messages.log:2023-11-22T09:51:31.153+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-1     !INFO | sudo[36229]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1001)
db-cluster-ef89d7e4/longevity-10gb-3h-master-db-node-ef89d7e4-0-1/messages.log:2023-11-22T09:51:31.153+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-1     !INFO | sudo[36229]: pam_unix(sudo:session): session closed for user root
db-cluster-ef89d7e4/longevity-10gb-3h-master-db-node-ef89d7e4-0-1/messages.log:2023-11-22T09:51:33.653+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-1     !INFO | scylla[12848]:  [shard  0: gms] gossip - Got shutdown message from 10.142.0.114, received_generation=1700636950, local_generation=1700636950
db-cluster-ef89d7e4/longevity-10gb-3h-master-db-node-ef89d7e4-0-1/messages.log:2023-11-22T09:51:33.653+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-1     !INFO | scylla[12848]:  [shard  0: gms] gossip - InetAddress 10.142.0.114 is now DOWN, status = shutdown
db-cluster-ef89d7e4/longevity-10gb-3h-master-db-node-ef89d7e4-0-1/messages.log:2023-11-22T09:51:33.653+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-1     !INFO | scylla[12848]:  [shard  0: gms] gossip - Got shutdown message from 10.142.0.6, received_generation=1700639573, local_generation=1700639573
db-cluster-ef89d7e4/longevity-10gb-3h-master-db-node-ef89d7e4-0-1/messages.log:2023-11-22T09:51:33.653+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-1     !INFO | scylla[12848]:  [shard  0: gms] gossip - InetAddress 10.142.0.6 is now DOWN, status = shutdown
db-cluster-ef89d7e4/longevity-10gb-3h-master-db-node-ef89d7e4-0-1/messages.log:2023-11-22T09:51:33.653+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-1     !INFO | scylla[12848]:  [shard  0: gms] gossip - Got shutdown message from 10.142.0.15, received_generation=1700646431, local_generation=1700646431
db-cluster-ef89d7e4/longevity-10gb-3h-master-db-node-ef89d7e4-0-1/messages.log:2023-11-22T09:51:33.653+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-1     !INFO | scylla[12848]:  [shard  0: gms] gossip - InetAddress 10.142.0.15 is now DOWN, status = shutdown
db-cluster-ef89d7e4/longevity-10gb-3h-master-db-node-ef89d7e4-0-1/messages.log:2023-11-22T09:51:36.653+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-1     !INFO | scylla[12848]:  [shard  0:main] raft_group_registry - marking Raft server a91eca31-4607-4708-a42f-0ca16d7641ba as dead for raft groups
db-cluster-ef89d7e4/longevity-10gb-3h-master-db-node-ef89d7e4-0-1/messages.log:2023-11-22T09:51:36.653+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-1     !INFO | scylla[12848]:  [shard  0:main] raft_group_registry - marking Raft server c53d3f45-b020-4457-b61b-4762432f3013 as dead for raft groups
db-cluster-ef89d7e4/longevity-10gb-3h-master-db-node-ef89d7e4-0-1/messages.log:2023-11-22T09:51:36.653+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-1     !INFO | scylla[12848]:  [shard  0:main] raft_group_registry - marking Raft server 44941f77-211a-43f8-8a72-237e4ab3c600 as dead for raft groups
db-cluster-ef89d7e4/longevity-10gb-3h-master-db-node-ef89d7e4-0-3/messages.log:2023-11-22T09:51:33.679+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-3     !INFO | scylla[3065]:  [shard  0: gms] gossip - Got shutdown message from 10.142.0.114, received_generation=1700636950, local_generation=1700636950
db-cluster-ef89d7e4/longevity-10gb-3h-master-db-node-ef89d7e4-0-3/messages.log:2023-11-22T09:51:33.679+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-3     !INFO | scylla[3065]:  [shard  0: gms] gossip - InetAddress 10.142.0.114 is now DOWN, status = shutdown
db-cluster-ef89d7e4/longevity-10gb-3h-master-db-node-ef89d7e4-0-3/messages.log:2023-11-22T09:51:33.679+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-3     !INFO | scylla[3065]:  [shard  0: gms] gossip - Got shutdown message from 10.142.0.6, received_generation=1700639573, local_generation=1700639573
db-cluster-ef89d7e4/longevity-10gb-3h-master-db-node-ef89d7e4-0-3/messages.log:2023-11-22T09:51:33.679+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-3     !INFO | scylla[3065]:  [shard  0: gms] gossip - InetAddress 10.142.0.6 is now DOWN, status = shutdown
db-cluster-ef89d7e4/longevity-10gb-3h-master-db-node-ef89d7e4-0-3/messages.log:2023-11-22T09:51:33.679+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-3     !INFO | scylla[3065]:  [shard  0: gms] gossip - Got shutdown message from 10.142.0.15, received_generation=1700646431, local_generation=1700646431
db-cluster-ef89d7e4/longevity-10gb-3h-master-db-node-ef89d7e4-0-3/messages.log:2023-11-22T09:51:33.679+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-3     !INFO | scylla[3065]:  [shard  0: gms] gossip - InetAddress 10.142.0.15 is now DOWN, status = shutdown
db-cluster-ef89d7e4/longevity-10gb-3h-master-db-node-ef89d7e4-0-3/messages.log:2023-11-22T09:51:35.177+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-3   !NOTICE | sudo[14227]: scylla-test : PWD=/home/scylla-test ; USER=root ; COMMAND=/usr/bin/coredumpctl -q --json=short
db-cluster-ef89d7e4/longevity-10gb-3h-master-db-node-ef89d7e4-0-3/messages.log:2023-11-22T09:51:35.177+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-3     !INFO | sudo[14227]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1001)
db-cluster-ef89d7e4/longevity-10gb-3h-master-db-node-ef89d7e4-0-3/messages.log:2023-11-22T09:51:35.177+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-3     !INFO | sudo[14227]: pam_unix(sudo:session): session closed for user root
db-cluster-ef89d7e4/longevity-10gb-3h-master-db-node-ef89d7e4-0-3/messages.log:2023-11-22T09:51:36.677+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-3     !INFO | scylla[3065]:  [shard  0:main] raft_group_registry - marking Raft server 44941f77-211a-43f8-8a72-237e4ab3c600 as dead for raft groups
db-cluster-ef89d7e4/longevity-10gb-3h-master-db-node-ef89d7e4-0-3/messages.log:2023-11-22T09:51:36.677+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-3     !INFO | scylla[3065]:  [shard  0:main] raft_group_registry - marking Raft server a91eca31-4607-4708-a42f-0ca16d7641ba as dead for raft groups
db-cluster-ef89d7e4/longevity-10gb-3h-master-db-node-ef89d7e4-0-3/messages.log:2023-11-22T09:51:36.677+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-3     !INFO | scylla[3065]:  [shard  0:main] raft_group_registry - marking Raft server c53d3f45-b020-4457-b61b-4762432f3013 as dead for raft groups
db-cluster-ef89d7e4/longevity-10gb-3h-master-db-node-ef89d7e4-0-3/messages.log:2023-11-22T09:51:38.177+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-3     !INFO | scylla[3065]:  [shard  0:main] raft_group0 - losing leadership
db-cluster-ef89d7e4/longevity-10gb-3h-master-db-node-ef89d7e4-0-6/messages.log:2023-11-22T09:51:32.946+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-6   !NOTICE | syslog-ng[5623]: syslog-ng shutting down; version='3.35.1'
db-cluster-ef89d7e4/longevity-10gb-3h-master-db-node-ef89d7e4-0-7/messages.log:2023-11-22T09:51:32.945+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-7   !NOTICE | syslog-ng[3539]: syslog-ng shutting down; version='3.35.1'
db-cluster-ef89d7e4/longevity-10gb-3h-master-db-node-ef89d7e4-0-8/messages.log:2023-11-22T09:51:33.862+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-8     !INFO | scylla[7134]:  [shard  0: gms] gossip - Got shutdown message from 10.142.0.114, received_generation=1700636950, local_generation=1700636950
db-cluster-ef89d7e4/longevity-10gb-3h-master-db-node-ef89d7e4-0-8/messages.log:2023-11-22T09:51:33.862+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-8     !INFO | scylla[7134]:  [shard  0: gms] gossip - InetAddress 10.142.0.114 is now DOWN, status = shutdown
db-cluster-ef89d7e4/longevity-10gb-3h-master-db-node-ef89d7e4-0-8/messages.log:2023-11-22T09:51:33.862+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-8     !INFO | scylla[7134]:  [shard  0: gms] gossip - Got shutdown message from 10.142.0.6, received_generation=1700639573, local_generation=1700639573
db-cluster-ef89d7e4/longevity-10gb-3h-master-db-node-ef89d7e4-0-8/messages.log:2023-11-22T09:51:33.862+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-8     !INFO | scylla[7134]:  [shard  0: gms] gossip - InetAddress 10.142.0.6 is now DOWN, status = shutdown
db-cluster-ef89d7e4/longevity-10gb-3h-master-db-node-ef89d7e4-0-8/messages.log:2023-11-22T09:51:33.862+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-8     !INFO | scylla[7134]:  [shard  0: gms] gossip - Got shutdown message from 10.142.0.15, received_generation=1700646431, local_generation=1700646431
db-cluster-ef89d7e4/longevity-10gb-3h-master-db-node-ef89d7e4-0-8/messages.log:2023-11-22T09:51:33.862+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-8     !INFO | scylla[7134]:  [shard  0: gms] gossip - InetAddress 10.142.0.15 is now DOWN, status = shutdown
db-cluster-ef89d7e4/longevity-10gb-3h-master-db-node-ef89d7e4-0-8/messages.log:2023-11-22T09:51:36.610+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-8     !INFO | scylla[7134]:  [shard  0:main] raft_group_registry - marking Raft server c53d3f45-b020-4457-b61b-4762432f3013 as dead for raft groups
db-cluster-ef89d7e4/longevity-10gb-3h-master-db-node-ef89d7e4-0-8/messages.log:2023-11-22T09:51:36.610+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-8     !INFO | scylla[7134]:  [shard  0:main] raft_group_registry - marking Raft server 44941f77-211a-43f8-8a72-237e4ab3c600 as dead for raft groups
db-cluster-ef89d7e4/longevity-10gb-3h-master-db-node-ef89d7e4-0-8/messages.log:2023-11-22T09:51:36.610+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-8     !INFO | scylla[7134]:  [shard  0:main] raft_group_registry - marking Raft server a91eca31-4607-4708-a42f-0ca16d7641ba as dead for raft groups
db-cluster-ef89d7e4/longevity-10gb-3h-master-db-node-ef89d7e4-0-9/messages.log:2023-11-22T09:51:30.583+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-9   !NOTICE | sudo[9523]: scylla-test : PWD=/home/scylla-test ; USER=root ; COMMAND=/usr/bin/coredumpctl -q --json=short
db-cluster-ef89d7e4/longevity-10gb-3h-master-db-node-ef89d7e4-0-9/messages.log:2023-11-22T09:51:30.583+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-9     !INFO | sudo[9523]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1001)
db-cluster-ef89d7e4/longevity-10gb-3h-master-db-node-ef89d7e4-0-9/messages.log:2023-11-22T09:51:30.583+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-9     !INFO | sudo[9523]: pam_unix(sudo:session): session closed for user root
db-cluster-ef89d7e4/longevity-10gb-3h-master-db-node-ef89d7e4-0-9/messages.log:2023-11-22T09:51:32.942+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-9   !NOTICE | syslog-ng[3531]: syslog-ng shutting down; version='3.35.1'

Impact

Possible impact dropping client connections or lost data

How frequently does it reproduce?

See 1 time in this job

Monitoring: http://3.230.0.70:3000/d/advanced-master/advanced?orgId=1&from=1700645796735&to=1700646916045

Installation details

Kernel Version: 5.15.0-1046-gcp Scylla version (or git commit hash): 5.5.0~dev-20231122.65e42e4166ce with build-id 44721ca3535e729f9dd7dee34b519bfc91b8c564

Cluster size: 6 nodes (n2-highmem-16)

Scylla Nodes used in this run:

OS / Image: https://www.googleapis.com/compute/v1/projects/scylla-images/global/images/785881930585351228 (gce: undefined_region)

Test: longevity-10gb-3h-gce-test Test id: ef89d7e4-c1fe-44db-b0a5-0d0aee311d8b Test name: scylla-master/longevity/longevity-10gb-3h-gce-test Test config file(s):

Logs and commands - Restore Monitor Stack command: `$ hydra investigate show-monitor ef89d7e4-c1fe-44db-b0a5-0d0aee311d8b` - Restore monitor on AWS instance using [Jenkins job](https://jenkins.scylladb.com/view/QA/job/QA-tools/job/hydra-show-monitor/parambuild/?test_id=ef89d7e4-c1fe-44db-b0a5-0d0aee311d8b) - Show all stored logs command: `$ hydra investigate show-logs ef89d7e4-c1fe-44db-b0a5-0d0aee311d8b` ## Logs: - **db-cluster-ef89d7e4.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/ef89d7e4-c1fe-44db-b0a5-0d0aee311d8b/20231122_112227/db-cluster-ef89d7e4.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/ef89d7e4-c1fe-44db-b0a5-0d0aee311d8b/20231122_112227/db-cluster-ef89d7e4.tar.gz) - **sct-runner-events-ef89d7e4.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/ef89d7e4-c1fe-44db-b0a5-0d0aee311d8b/20231122_112227/sct-runner-events-ef89d7e4.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/ef89d7e4-c1fe-44db-b0a5-0d0aee311d8b/20231122_112227/sct-runner-events-ef89d7e4.tar.gz) - **sct-ef89d7e4.log.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/ef89d7e4-c1fe-44db-b0a5-0d0aee311d8b/20231122_112227/sct-ef89d7e4.log.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/ef89d7e4-c1fe-44db-b0a5-0d0aee311d8b/20231122_112227/sct-ef89d7e4.log.tar.gz) - **loader-set-ef89d7e4.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/ef89d7e4-c1fe-44db-b0a5-0d0aee311d8b/20231122_112227/loader-set-ef89d7e4.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/ef89d7e4-c1fe-44db-b0a5-0d0aee311d8b/20231122_112227/loader-set-ef89d7e4.tar.gz) - **monitor-set-ef89d7e4.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/ef89d7e4-c1fe-44db-b0a5-0d0aee311d8b/20231122_112227/monitor-set-ef89d7e4.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/ef89d7e4-c1fe-44db-b0a5-0d0aee311d8b/20231122_112227/monitor-set-ef89d7e4.tar.gz) - **parallel-timelines-report-ef89d7e4.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/ef89d7e4-c1fe-44db-b0a5-0d0aee311d8b/20231122_112227/parallel-timelines-report-ef89d7e4.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/ef89d7e4-c1fe-44db-b0a5-0d0aee311d8b/20231122_112227/parallel-timelines-report-ef89d7e4.tar.gz) [Jenkins job URL](https://jenkins.scylladb.com/job/scylla-master/job/longevity/job/longevity-10gb-3h-gce-test/421/) [Argus](https://argus.scylladb.com/test/6e39841d-5ed5-4f83-aa70-f51949559234/runs?additionalRuns[]=ef89d7e4-c1fe-44db-b0a5-0d0aee311d8b)
aleksbykov commented 9 months ago

Some decoded reactor stalls

2023-11-22 08:13:51.045 <2023-11-22 08:13:51.027>: (DatabaseLogEvent Severity.DEBUG) period_type=one-time event_id=6a4c4607-4ff2-4905-bdeb-06a4b474ff96: type=KERNEL_CALLSTACK regex=kernel callstack: 0x.{16} line_number=27168 node=longevity-10gb-3h-master-db-node-ef89d7e4-0-2
2023-11-22T08:13:51.027+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-2     !INFO | scylla[12848]: kernel callstack: 0xffffffffffffff80 0xffffffffb38d37f6 0xffffffffb38d3e46 0xffffffffb38d45e2 0xffffffffb38d47ae 0xffffffffb38d4f97 0xffffffffb38d722e 0xffffffffb3928efd 0xffffffffb39298d1 0xffffffffb394bb4e 0xffffffffb38b722e 0xffffffffb38bb0c2 0xffffffffb38bbe58 0xffffffffb3901fa9 0xffffffffb39035cb 0xffffffffb39036e0 0xffffffffb3903efd 0xffffffffb3905ed5 0xffffffffb3906298 0xffffffffb36a0af9 0xffffffffb43f38c7 0xffffffffb4600bb7
2023-11-22 08:13:51.030 <2023-11-22 08:13:51.002>: (DatabaseLogEvent Severity.WARNING) period_type=one-time event_id=88bd862f-b6d1-47b0-84af-1010803c4032: type=REACTOR_STALLED regex=Reactor stalled line_number=27163 node=longevity-10gb-3h-master-db-node-ef89d7e4-0-2
2023-11-22T08:13:51.002+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-2     !INFO | scylla[12848]: Reactor stalled for 40 ms on shard 3. Backtrace: 0x5b2c8ea 0x5b2bcf5 0x5b2d09f 0x3dbaf 0x1148da 0x5b7ee9a 0x5b57732 0x5b243b9 0x5b88d2c 0x5b248b7 0x5fc0063 0x286528b 0x6029d40 0x60105b1 0x6010cd9 0x601a45f 0x60122b9 0x6013943 0x157373f 0x14dc620 0x31f9757 0x3133144 0x3115ea4 0x320c524 0x3120428 0x3105a27 0x31f0380 0x31f029e 0x32034d8 0x3204625 0x3203d82 0x32034d8 0x3202c2f 0x3201bc0 0x5ae78f2 0x5b3e25f 0x5b3f537 0x5b63183 0x5b0dcba 0x8c946 0x11285f
void seastar::backtrace<seastar::backtrace_buffer::append_backtrace_oneline()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace_oneline()::{lambda(seastar::frame)#1}&&) at ./build/release/seastar/./seastar/include/seastar/util/backtrace.hh:68
 (inlined by) seastar::backtrace_buffer::append_backtrace_oneline() at ./build/release/seastar/./seastar/src/core/reactor.cc:836
 (inlined by) seastar::print_with_backtrace(seastar::backtrace_buffer&, bool) at ./build/release/seastar/./seastar/src/core/reactor.cc:855
seastar::internal::cpu_stall_detector::generate_trace() at ./build/release/seastar/./seastar/src/core/reactor.cc:1469
seastar::internal::cpu_stall_detector::maybe_report() at ./build/release/seastar/./seastar/src/core/reactor.cc:1206
 (inlined by) seastar::internal::cpu_stall_detector::on_signal() at ./build/release/seastar/./seastar/src/core/reactor.cc:1226
 (inlined by) seastar::reactor::block_notifier(int) at ./build/release/seastar/./seastar/src/core/reactor.cc:1506
?? ??:0
?? ??:0
seastar::file_desc::sendmsg(msghdr const*, int) at ./build/release/seastar/./seastar/include/seastar/core/posix.hh:274
operator() at ./build/release/seastar/./seastar/src/core/reactor.cc:395
 (inlined by) seastar::future<unsigned long> seastar::futurize<seastar::future<unsigned long> >::invoke<seastar::reactor::do_sendmsg(seastar::pollable_fd_state&, seastar::net::packet&)::$_0>(seastar::reactor::do_sendmsg(seastar::pollable_fd_state&, seastar::net::packet&)::$_0&&) at ./build/release/seastar/./seastar/include/seastar/core/future.hh:2006
 (inlined by) seastar::future<unsigned long> seastar::futurize<seastar::future<unsigned long> >::invoke<seastar::reactor::do_sendmsg(seastar::pollable_fd_state&, seastar::net::packet&)::$_0>(seastar::reactor::do_sendmsg(seastar::pollable_fd_state&, seastar::net::packet&)::$_0&&, seastar::internal::monostate) at ./build/release/seastar/./seastar/include/seastar/core/future.hh:1848
 (inlined by) seastar::future<unsigned long> seastar::future<void>::then_impl<seastar::reactor::do_sendmsg(seastar::pollable_fd_state&, seastar::net::packet&)::$_0, seastar::future<unsigned long> >(seastar::reactor::do_sendmsg(seastar::pollable_fd_state&, seastar::net::packet&)::$_0&&) at ./build/release/seastar/./seastar/include/seastar/core/future.hh:1473
seastar::future<unsigned long> seastar::future<void>::then<seastar::reactor::do_sendmsg(seastar::pollable_fd_state&, seastar::net::packet&)::$_0, seastar::future<unsigned long> >(seastar::reactor::do_sendmsg(seastar::pollable_fd_state&, seastar::net::packet&)::$_0&&) at ./build/release/seastar/./seastar/include/seastar/core/future.hh:1399
 (inlined by) seastar::reactor::do_sendmsg(seastar::pollable_fd_state&, seastar::net::packet&) at ./build/release/seastar/./seastar/src/core/reactor.cc:382
seastar::reactor_backend_aio::sendmsg(seastar::pollable_fd_state&, seastar::net::packet&) at ./build/release/seastar/./seastar/src/core/reactor_backend.cc:677
seastar::pollable_fd_state::write_some(seastar::net::packet&) at ./build/release/seastar/./seastar/src/core/reactor.cc:458
 (inlined by) seastar::pollable_fd_state::write_all(seastar::net::packet&) at ./build/release/seastar/./seastar/src/core/reactor.cc:470
seastar::pollable_fd::write_all(seastar::net::packet&) at ./build/release/seastar/./seastar/include/seastar/core/internal/pollable_fd.hh:165
 (inlined by) seastar::net::posix_data_sink_impl::put(seastar::net::packet) at ./build/release/seastar/./seastar/src/net/posix-stack.cc:666
seastar::data_sink::put(seastar::net::packet) at ././seastar/include/seastar/core/iostream.hh:174
 (inlined by) seastar::output_stream<char>::zero_copy_put(seastar::net::packet) at ././seastar/include/seastar/core/iostream-impl.hh:87
 (inlined by) seastar::output_stream<char>::write(seastar::net::packet) at ././seastar/include/seastar/core/iostream-impl.hh:129
seastar::output_stream<char>::write(seastar::temporary_buffer<char>) at ./build/release/seastar/./seastar/include/seastar/core/iostream-impl.hh:146
seastar::rpc::connection::send_buffer(seastar::rpc::snd_buf) at ./build/release/seastar/./seastar/src/rpc/rpc.cc:124
seastar::rpc::connection::send_entry(seastar::rpc::connection::outgoing_entry&) at ./build/release/seastar/./seastar/src/rpc/rpc.cc:151
operator() at ./build/release/seastar/./seastar/src/rpc/rpc.cc:301
seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<seastar::rpc::connection::send(seastar::rpc::snd_buf, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >, seastar::rpc::cancellable*)::$_1>(seastar::rpc::connection::send(seastar::rpc::snd_buf, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >, seastar::rpc::cancellable*)::$_1&&) at ./build/release/seastar/./seastar/include/seastar/core/future.hh:2006
 (inlined by) seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<seastar::rpc::connection::send(seastar::rpc::snd_buf, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >, seastar::rpc::cancellable*)::$_1>(seastar::rpc::connection::send(seastar::rpc::snd_buf, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >, seastar::rpc::cancellable*)::$_1&&, seastar::internal::monostate) at ./build/release/seastar/./seastar/include/seastar/core/future.hh:1848
 (inlined by) seastar::future<void> seastar::future<void>::then_impl<seastar::rpc::connection::send(seastar::rpc::snd_buf, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >, seastar::rpc::cancellable*)::$_1, seastar::future<void> >(seastar::rpc::connection::send(seastar::rpc::snd_buf, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >, seastar::rpc::cancellable*)::$_1&&) at ./build/release/seastar/./seastar/include/seastar/core/future.hh:1473
 (inlined by) seastar::future<void> seastar::future<void>::then<seastar::rpc::connection::send(seastar::rpc::snd_buf, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >, seastar::rpc::cancellable*)::$_1, seastar::future<void> >(seastar::rpc::connection::send(seastar::rpc::snd_buf, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >, seastar::rpc::cancellable*)::$_1&&) at ./build/release/seastar/./seastar/include/seastar/core/future.hh:1399
 (inlined by) seastar::rpc::connection::send(seastar::rpc::snd_buf, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >, seastar::rpc::cancellable*) at ./build/release/seastar/./seastar/src/rpc/rpc.cc:293
seastar::rpc::client::request(unsigned long, long, seastar::rpc::snd_buf, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >, seastar::rpc::cancellable*) at ./build/release/seastar/./seastar/src/rpc/rpc.cc:654
send at ././seastar/include/seastar/rpc/rpc_impl.hh:478
operator() at ././seastar/include/seastar/rpc/rpc_impl.hh:487
 (inlined by) auto netw::send_message_timeout<seastar::rpc::no_wait_type, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, frozen_mutation const, utils::small_vector<gms::inet_address, 3ul> const, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info> const, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>, service::fencing_token>(netw::messaging_service*, netw::messaging_verb, netw::msg_addr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, frozen_mutation const&&, utils::small_vector<gms::inet_address, 3ul> const&&, gms::inet_address&&, unsigned int&&, unsigned long&&, std::optional<tracing::trace_info> const&&, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>&&, service::fencing_token&&) at ././message/rpc_protocol_impl.hh:157
 (inlined by) auto netw::send_message_oneway_timeout<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, frozen_mutation const, utils::small_vector<gms::inet_address, 3ul> const, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info> const, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>, service::fencing_token>(netw::messaging_service*, netw::messaging_verb, netw::msg_addr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, frozen_mutation const&&, utils::small_vector<gms::inet_address, 3ul> const&&, gms::inet_address&&, unsigned int&&, unsigned long&&, std::optional<tracing::trace_info> const&&, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>&&, service::fencing_token&&) at ././message/rpc_protocol_impl.hh:217
 (inlined by) ser::storage_proxy_rpc_verbs::send_mutation(netw::messaging_service*, netw::msg_addr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, frozen_mutation const&, utils::small_vector<gms::inet_address, 3ul> const&, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info> const&, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>, service::fencing_token) at ./build/release/gen/idl/storage_proxy.dist.impl.hh:40
service::storage_proxy::remote::send_mutation(netw::msg_addr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, std::optional<tracing::trace_info> const&, frozen_mutation const&, utils::small_vector<gms::inet_address, 3ul> const&, gms::inet_address, unsigned int, unsigned long, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>, service::fencing_token) at ./service/storage_proxy.cc:234
 (inlined by) service::shared_mutation::apply_remotely(service::storage_proxy&, gms::inet_address, utils::small_vector<gms::inet_address, 3ul> const&, unsigned long, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>, service::fencing_token) at ./service/storage_proxy.cc:1095
service::abstract_write_response_handler::apply_remotely(gms::inet_address, utils::small_vector<gms::inet_address, 3ul> const&, unsigned long, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr) at ./service/storage_proxy.cc:1553
 (inlined by) operator() at ./service/storage_proxy.cc:3974
 (inlined by) seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<service::storage_proxy::send_to_live_endpoints(unsigned long, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::$_1&, gms::inet_address&, utils::small_vector<gms::inet_address, 3ul>&>(service::storage_proxy::send_to_live_endpoints(unsigned long, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::$_1&, gms::inet_address&, utils::small_vector<gms::inet_address, 3ul>&) at ././seastar/include/seastar/core/future.hh:2006
 (inlined by) auto seastar::futurize_invoke<service::storage_proxy::send_to_live_endpoints(unsigned long, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::$_1&, gms::inet_address&, utils::small_vector<gms::inet_address, 3ul>&>(service::storage_proxy::send_to_live_endpoints(unsigned long, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::$_1&, gms::inet_address&, utils::small_vector<gms::inet_address, 3ul>&) at ././seastar/include/seastar/core/future.hh:2037
 (inlined by) service::storage_proxy::send_to_live_endpoints(unsigned long, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >) at ./service/storage_proxy.cc:4004
operator() at ./service/storage_proxy.cc:3216
 (inlined by) seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > seastar::futurize<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > >::invoke<service::storage_proxy::mutate_begin(utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>, db::consistency_level, tracing::trace_state_ptr, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >)::$_0, service::storage_proxy::unique_response_handler&>(service::storage_proxy::mutate_begin(utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>, db::consistency_level, tracing::trace_state_ptr, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >)::$_0&&, service::storage_proxy::unique_response_handler&) at ././seastar/include/seastar/core/future.hh:2006
 (inlined by) auto seastar::futurize_invoke<service::storage_proxy::mutate_begin(utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>, db::consistency_level, tracing::trace_state_ptr, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >)::$_0, service::storage_proxy::unique_response_handler&>(service::storage_proxy::mutate_begin(utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>, db::consistency_level, tracing::trace_state_ptr, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >)::$_0&&, service::storage_proxy::unique_response_handler&) at ././seastar/include/seastar/core/future.hh:2037
 (inlined by) seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > utils::result_parallel_for_each<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy>, service::storage_proxy::unique_response_handler*, service::storage_proxy::mutate_begin(utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>, db::consistency_level, tracing::trace_state_ptr, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >)::$_0>(service::storage_proxy::unique_response_handler*, service::storage_proxy::unique_response_handler*, service::storage_proxy::mutate_begin(utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>, db::consistency_level, tracing::trace_state_ptr, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >)::$_0&&) at ././utils/result_loop.hh:46
 (inlined by) seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > utils::result_parallel_for_each<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy>, utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>&, service::storage_proxy::mutate_begin(utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>, db::consistency_level, tracing::trace_state_ptr, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >)::$_0>(utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>&, service::storage_proxy::mutate_begin(utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>, db::consistency_level, tracing::trace_state_ptr, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >)::$_0&&) at ././utils/result_loop.hh:102
 (inlined by) service::storage_proxy::mutate_begin(utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>, db::consistency_level, tracing::trace_state_ptr, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >) at ./service/storage_proxy.cc:3188
operator() at ./service/storage_proxy.cc:3506
 (inlined by) seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > seastar::futurize<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > >::invoke<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > service::storage_proxy::mutate_internal<boost::iterator_range<__gnu_cxx::__normal_iterator<mutation*, std::vector<mutation, std::allocator<mutation> > > > >(boost::iterator_range<__gnu_cxx::__normal_iterator<mutation*, std::vector<mutation, std::allocator<mutation> > > >, db::consistency_level, bool, tracing::trace_state_ptr, service_permit, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >, seastar::lw_shared_ptr<cdc::operation_result_tracker>, seastar::bool_class<db::allow_per_partition_rate_limit_tag>)::{lambda(utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>)#1}&, utils::small_vector<service::storage_proxy::unique_response_handler, 1ul> >(seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > service::storage_proxy::mutate_internal<boost::iterator_range<__gnu_cxx::__normal_iterator<mutation*, std::vector<mutation, std::allocator<mutation> > > > >(boost::iterator_range<__gnu_cxx::__normal_iterator<mutation*, std::vector<mutation, std::allocator<mutation> > > >, db::consistency_level, bool, tracing::trace_state_ptr, service_permit, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >, seastar::lw_shared_ptr<cdc::operation_result_tracker>, seastar::bool_class<db::allow_per_partition_rate_limit_tag>)::{lambda(utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>)#1}&, utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>&&) at ././seastar/include/seastar/core/future.hh:2006
auto seastar::futurize_invoke<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > service::storage_proxy::mutate_internal<boost::iterator_range<__gnu_cxx::__normal_iterator<mutation*, std::vector<mutation, std::allocator<mutation> > > > >(boost::iterator_range<__gnu_cxx::__normal_iterator<mutation*, std::vector<mutation, std::allocator<mutation> > > >, db::consistency_level, bool, tracing::trace_state_ptr, service_permit, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >, seastar::lw_shared_ptr<cdc::operation_result_tracker>, seastar::bool_class<db::allow_per_partition_rate_limit_tag>)::{lambda(utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>)#1}&, utils::small_vector<service::storage_proxy::unique_response_handler, 1ul> >(seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > service::storage_proxy::mutate_internal<boost::iterator_range<__gnu_cxx::__normal_iterator<mutation*, std::vector<mutation, std::allocator<mutation> > > > >(boost::iterator_range<__gnu_cxx::__normal_iterator<mutation*, std::vector<mutation, std::allocator<mutation> > > >, db::consistency_level, bool, tracing::trace_state_ptr, service_permit, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >, seastar::lw_shared_ptr<cdc::operation_result_tracker>, seastar::bool_class<db::allow_per_partition_rate_limit_tag>)::{lambda(utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>)#1}&, utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>&&) at ././seastar/include/seastar/core/future.hh:2037
 (inlined by) utils::internal::result_wrapped_call_traits<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > service::storage_proxy::mutate_internal<boost::iterator_range<__gnu_cxx::__normal_iterator<mutation*, std::vector<mutation, std::allocator<mutation> > > > >(boost::iterator_range<__gnu_cxx::__normal_iterator<mutation*, std::vector<mutation, std::allocator<mutation> > > >, db::consistency_level, bool, tracing::trace_state_ptr, service_permit, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >, seastar::lw_shared_ptr<cdc::operation_result_tracker>, seastar::bool_class<db::allow_per_partition_rate_limit_tag>)::{lambda(utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>)#1}, boost::outcome_v2::basic_result<utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy>, false>::invoke_with_value({lambda(utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>)#1}&, seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > service::storage_proxy::mutate_internal<boost::iterator_range<__gnu_cxx::__normal_iterator<mutation*, std::vector<mutation, std::allocator<mutation> > > > >(boost::iterator_range<__gnu_cxx::__normal_iterator<mutation*, std::vector<mutation, std::allocator<mutation> > > >, db::consistency_level, bool, tracing::trace_state_ptr, service_permit, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >, seastar::lw_shared_ptr<cdc::operation_result_tracker>, seastar::bool_class<db::allow_per_partition_rate_limit_tag>)::{lambda(utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>)#1}&&) at ././utils/result_combinators.hh:73
 (inlined by) auto utils::internal::result_wrapper<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > service::storage_proxy::mutate_internal<boost::iterator_range<__gnu_cxx::__normal_iterator<mutation*, std::vector<mutation, std::allocator<mutation> > > > >(boost::iterator_range<__gnu_cxx::__normal_iterator<mutation*, std::vector<mutation, std::allocator<mutation> > > >, db::consistency_level, bool, tracing::trace_state_ptr, service_permit, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >, seastar::lw_shared_ptr<cdc::operation_result_tracker>, seastar::bool_class<db::allow_per_partition_rate_limit_tag>)::{lambda(utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>)#1}, false>::operator()<boost::outcome_v2::basic_result<utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> >(boost::outcome_v2::basic_result<utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy>) at ././utils/result_combinators.hh:124
 (inlined by) seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > seastar::futurize<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > >::invoke<utils::internal::result_wrapper<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > service::storage_proxy::mutate_internal<boost::iterator_range<__gnu_cxx::__normal_iterator<mutation*, std::vector<mutation, std::allocator<mutation> > > > >(boost::iterator_range<__gnu_cxx::__normal_iterator<mutation*, std::vector<mutation, std::allocator<mutation> > > >, db::consistency_level, bool, tracing::trace_state_ptr, service_permit, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >, seastar::lw_shared_ptr<cdc::operation_result_tracker>, seastar::bool_class<db::allow_per_partition_rate_limit_tag>)::{lambda(utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>)#1}, false>, boost::outcome_v2::basic_result<utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> >(utils::internal::result_wrapper<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > service::storage_proxy::mutate_internal<boost::iterator_range<__gnu_cxx::__normal_iterator<mutation*, std::vector<mutation, std::allocator<mutation> > > > >(boost::iterator_range<__gnu_cxx::__normal_iterator<mutation*, std::vector<mutation, std::allocator<mutation> > > >, db::consistency_level, bool, tracing::trace_state_ptr, service_permit, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >, seastar::lw_shared_ptr<cdc::operation_result_tracker>, seastar::bool_class<db::allow_per_partition_rate_limit_tag>)::{lambda(utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>)#1}, false>&&, boost::outcome_v2::basic_result<utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy>&&) at ././seastar/include/seastar/core/future.hh:2006
 (inlined by) seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > seastar::future<boost::outcome_v2::basic_result<utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> >::then_impl<utils::internal::result_wrapper<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > service::storage_proxy::mutate_internal<boost::iterator_range<__gnu_cxx::__normal_iterator<mutation*, std::vector<mutation, std::allocator<mutation> > > > >(boost::iterator_range<__gnu_cxx::__normal_iterator<mutation*, std::vector<mutation, std::allocator<mutation> > > >, db::consistency_level, bool, tracing::trace_state_ptr, service_permit, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >, seastar::lw_shared_ptr<cdc::operation_result_tracker>, seastar::bool_class<db::allow_per_partition_rate_limit_tag>)::{lambda(utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>)#1}, false>, seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > >(utils::internal::result_wrapper<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > service::storage_proxy::mutate_internal<boost::iterator_range<__gnu_cxx::__normal_iterator<mutation*, std::vector<mutation, std::allocator<mutation> > > > >(boost::iterator_range<__gnu_cxx::__normal_iterator<mutation*, std::vector<mutation, std::allocator<mutation> > > >, db::consistency_level, bool, tracing::trace_state_ptr, service_permit, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >, seastar::lw_shared_ptr<cdc::operation_result_tracker>, seastar::bool_class<db::allow_per_partition_rate_limit_tag>)::{lambda(utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>)#1}, false>&&) at ././seastar/include/seastar/core/future.hh:1473
 (inlined by) seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > seastar::future<boost::outcome_v2::basic_result<utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> >::then<utils::internal::result_wrapper<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > service::storage_proxy::mutate_internal<boost::iterator_range<__gnu_cxx::__normal_iterator<mutation*, std::vector<mutation, std::allocator<mutation> > > > >(boost::iterator_range<__gnu_cxx::__normal_iterator<mutation*, std::vector<mutation, std::allocator<mutation> > > >, db::consistency_level, bool, tracing::trace_state_ptr, service_permit, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >, seastar::lw_shared_ptr<cdc::operation_result_tracker>, seastar::bool_class<db::allow_per_partition_rate_limit_tag>)::{lambda(utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>)#1}, false>, seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > >(utils::internal::result_wrapper<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > service::storage_proxy::mutate_internal<boost::iterator_range<__gnu_cxx::__normal_iterator<mutation*, std::vector<mutation, std::allocator<mutation> > > > >(boost::iterator_range<__gnu_cxx::__normal_iterator<mutation*, std::vector<mutation, std::allocator<mutation> > > >, db::consistency_level, bool, tracing::trace_state_ptr, service_permit, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >, seastar::lw_shared_ptr<cdc::operation_result_tracker>, seastar::bool_class<db::allow_per_partition_rate_limit_tag>)::{lambda(utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>)#1}, false>&&) at ././seastar/include/seastar/core/future.hh:1399
 (inlined by) seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > service::storage_proxy::mutate_internal<boost::iterator_range<__gnu_cxx::__normal_iterator<mutation*, std::vector<mutation, std::allocator<mutation> > > > >(boost::iterator_range<__gnu_cxx::__normal_iterator<mutation*, std::vector<mutation, std::allocator<mutation> > > >, db::consistency_level, bool, tracing::trace_state_ptr, service_permit, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >, seastar::lw_shared_ptr<cdc::operation_result_tracker>, seastar::bool_class<db::allow_per_partition_rate_limit_tag>) at ./service/storage_proxy.cc:3503
service::storage_proxy::do_mutate(std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker>) at ./service/storage_proxy.cc:3463
seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > std::__invoke_impl<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> >, seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > (service::storage_proxy::* const&)(std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker>), service::storage_proxy*, std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker> >(std::__invoke_memfun_deref, seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > (service::storage_proxy::* const&)(std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker>), service::storage_proxy*&&, std::vector<mutation, std::allocator<mutation> >&&, db::consistency_level&&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >&&, tracing::trace_state_ptr&&, service_permit&&, bool&&, seastar::bool_class<db::allow_per_partition_rate_limit_tag>&&, seastar::lw_shared_ptr<cdc::operation_result_tracker>&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/bits/invoke.h:74
std::__invoke_result<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > (service::storage_proxy::* const&)(std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker>), service::storage_proxy*, std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker> >::type std::__invoke<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > (service::storage_proxy::* const&)(std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker>), service::storage_proxy*, std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker> >(seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > (service::storage_proxy::* const&)(std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker>), service::storage_proxy*&&, std::vector<mutation, std::allocator<mutation> >&&, db::consistency_level&&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >&&, tracing::trace_state_ptr&&, service_permit&&, bool&&, seastar::bool_class<db::allow_per_partition_rate_limit_tag>&&, seastar::lw_shared_ptr<cdc::operation_result_tracker>&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/bits/invoke.h:96
 (inlined by) _ZNKSt12_Mem_fn_baseIMN7service13storage_proxyEFN7seastar6futureIN5boost10outcome_v212basic_resultIvN5utils19exception_containerIJN10exceptions32mutation_write_timeout_exceptionENS9_22read_timeout_exceptionENS9_22read_failure_exceptionENS9_20rate_limit_exceptionEEEENS7_32exception_container_throw_policyEEEEESt6vectorI8mutationSaISJ_EEN2db17consistency_levelENSt6chrono10time_pointINS2_12lowres_clockENSO_8durationIlSt5ratioILl1ELl1000000000EEEEEEN7tracing15trace_state_ptrE14service_permitbNS2_10bool_classINSM_34allow_per_partition_rate_limit_tagEEENS2_13lw_shared_ptrIN3cdc24operation_result_trackerEEEELb1EEclIJPS1_SL_SN_SV_SX_SY_bS11_S15_EEEDTclsr3stdE8__invokedtdefpT6_M_pmfspclsr3stdE7forwardIT_Efp_EEEDpOS1B_ at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/functional:170
 (inlined by) seastar::noncopyable_function<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > (service::storage_proxy*, std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker>)>::direct_vtable_for<std::_Mem_fn<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > (service::storage_proxy::*)(std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker>)> >::call(seastar::noncopyable_function<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > (service::storage_proxy*, std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker>)> const*, service::storage_proxy*, std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker>) at ././seastar/include/seastar/util/noncopyable_function.hh:129
seastar::noncopyable_function<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > (service::storage_proxy*, std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker>)>::operator()(service::storage_proxy*, std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker>) const at ././seastar/include/seastar/util/noncopyable_function.hh:215
operator() at ././seastar/include/seastar/core/execution_stage.hh:342
seastar::noncopyable_function<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > (service::storage_proxy*, std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker>)>::direct_vtable_for<seastar::inheriting_concrete_execution_stage<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> >, service::storage_proxy*, std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker> >::make_stage_for_group(seastar::scheduling_group)::{lambda(service::storage_proxy*, std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker>)#1}>::call(seastar::noncopyable_function<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > (service::storage_proxy*, std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker>)> const*, service::storage_proxy*, std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker>) at ././seastar/include/seastar/util/noncopyable_function.hh:129
seastar::noncopyable_function<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > (service::storage_proxy*, std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker>)>::operator()(service::storage_proxy*, std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker>) const at ././seastar/include/seastar/util/noncopyable_function.hh:215
seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > std::__invoke_impl<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> >, seastar::noncopyable_function<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > (service::storage_proxy*, std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker>)>&, service::storage_proxy*, std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker> >(std::__invoke_other, seastar::noncopyable_function<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > (service::storage_proxy*, std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker>)>&, service::storage_proxy*&&, std::vector<mutation, std::allocator<mutation> >&&, db::consistency_level&&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >&&, tracing::trace_state_ptr&&, service_permit&&, bool&&, seastar::bool_class<db::allow_per_partition_rate_limit_tag>&&, seastar::lw_shared_ptr<cdc::operation_result_tracker>&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/bits/invoke.h:61
std::__invoke_result<seastar::noncopyable_function<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > (service::storage_proxy*, std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker>)>&, service::storage_proxy*, std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker> >::type std::__invoke<seastar::noncopyable_function<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > (service::storage_proxy*, std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker>)>&, service::storage_proxy*, std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker> >(seastar::noncopyable_function<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > (service::storage_proxy*, std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker>)>&, service::storage_proxy*&&, std::vector<mutation, std::allocator<mutation> >&&, db::consistency_level&&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >&&, tracing::trace_state_ptr&&, service_permit&&, bool&&, seastar::bool_class<db::allow_per_partition_rate_limit_tag>&&, seastar::lw_shared_ptr<cdc::operation_result_tracker>&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/bits/invoke.h:96
 (inlined by) _ZSt12__apply_implIRN7seastar20noncopyable_functionIFNS0_6futureIN5boost10outcome_v212basic_resultIvN5utils19exception_containerIJN10exceptions32mutation_write_timeout_exceptionENS8_22read_timeout_exceptionENS8_22read_failure_exceptionENS8_20rate_limit_exceptionEEEENS6_32exception_container_throw_policyEEEEEPN7service13storage_proxyESt6vectorI8mutationSaISL_EEN2db17consistency_levelENSt6chrono10time_pointINS0_12lowres_clockENSQ_8durationIlSt5ratioILl1ELl1000000000EEEEEEN7tracing15trace_state_ptrE14service_permitbNS0_10bool_classINSO_34allow_per_partition_rate_limit_tagEEENS0_13lw_shared_ptrIN3cdc24operation_result_trackerEEEEEESt5tupleIJSJ_SN_SP_SX_SZ_S10_bS13_S17_EEJLm0ELm1ELm2ELm3ELm4ELm5ELm6ELm7ELm8EEEDcOT_OT0_St16integer_sequenceImJXspT1_EEE at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/tuple:2288
 (inlined by) _ZSt5applyIRN7seastar20noncopyable_functionIFNS0_6futureIN5boost10outcome_v212basic_resultIvN5utils19exception_containerIJN10exceptions32mutation_write_timeout_exceptionENS8_22read_timeout_exceptionENS8_22read_failure_exceptionENS8_20rate_limit_exceptionEEEENS6_32exception_container_throw_policyEEEEEPN7service13storage_proxyESt6vectorI8mutationSaISL_EEN2db17consistency_levelENSt6chrono10time_pointINS0_12lowres_clockENSQ_8durationIlSt5ratioILl1ELl1000000000EEEEEEN7tracing15trace_state_ptrE14service_permitbNS0_10bool_classINSO_34allow_per_partition_rate_limit_tagEEENS0_13lw_shared_ptrIN3cdc24operation_result_trackerEEEEEESt5tupleIJSJ_SN_SP_SX_SZ_S10_bS13_S17_EEEDcOT_OT0_ at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/tuple:2299
 (inlined by) seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > seastar::futurize<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > >::apply<seastar::noncopyable_function<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > (service::storage_proxy*, std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker>)>&, service::storage_proxy*, std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker> >(seastar::noncopyable_function<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > (service::storage_proxy*, std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker>)>&, std::tuple<service::storage_proxy*, std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker> >&&) at ././seastar/include/seastar/core/future.hh:1973
 (inlined by) seastar::concrete_execution_stage<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> >, service::storage_proxy*, std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker> >::do_flush() at ././seastar/include/seastar/core/execution_stage.hh:251
operator() at ./build/release/seastar/./seastar/src/core/execution_stage.cc:149
 (inlined by) seastar::future<void> seastar::futurize<void>::invoke<seastar::execution_stage::flush()::$_0&>(seastar::execution_stage::flush()::$_0&) at ./build/release/seastar/./seastar/include/seastar/core/future.hh:2003
 (inlined by) seastar::lambda_task<seastar::execution_stage::flush()::$_0>::run_and_dispose() at ./build/release/seastar/./seastar/include/seastar/core/make_task.hh:45
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at ./build/release/seastar/./seastar/src/core/reactor.cc:2651
 (inlined by) seastar::reactor::run_some_tasks() at ./build/release/seastar/./seastar/src/core/reactor.cc:3114
seastar::reactor::do_run() at ./build/release/seastar/./seastar/src/core/reactor.cc:3283
operator() at ./build/release/seastar/./seastar/src/core/reactor.cc:4506
 (inlined by) void std::__invoke_impl<void, seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_0&>(std::__invoke_other, seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_0&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/bits/invoke.h:61
 (inlined by) std::enable_if<is_invocable_r_v<void, seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_0&>, void>::type std::__invoke_r<void, seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_0&>(seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_0&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/bits/invoke.h:111
 (inlined by) std::_Function_handler<void (), seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_0>::_M_invoke(std::_Any_data const&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/bits/std_function.h:290
std::function<void ()>::operator()() const at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/bits/std_function.h:591
 (inlined by) seastar::posix_thread::start_routine(void*) at ./build/release/seastar/./seastar/src/core/posix.cc:90
?? ??:0
?? ??:0
?? ??:0

2023-11-22T08:13:51.728+00:00 longevity-10gb-3h-master-db-node-ef89d7e4-0-2     !INFO | scylla[12848]: Reactor stalled for 125 ms on shard 5. Backtrace: 0x5b2c8ea 0x5b2bcf5 0x5b2d09f 0x3dbaf 0x1148da 0x5b7ee9a 0x5b57732 0x5b243b9 0x5b88d2c 0x5b248b7 0x5fc0063 0x286528b 0x6029d40 0x60105b1 0x6010cd9 0x601a45f 0x60122b9 0x6013943 0x157373f 0x14dc620 0x31f9757 0x3133144 0x3115ea4 0x320c524 0x3120428 0x3105a27 0x31f0380 0x31f029e 0x32034d8 0x3204625 0x3203d82 0x32034d8 0x3202c2f 0x3201bc0 0x5ae78f2 0x5b3e25f 0x5b3f537 0x5b63183 0x5b0dcba 0x8c946 0x11285f
void seastar::backtrace<seastar::backtrace_buffer::append_backtrace_oneline()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace_oneline()::{lambda(seastar::frame)#1}&&) at ./build/release/seastar/./seastar/include/seastar/util/backtrace.hh:68
 (inlined by) seastar::backtrace_buffer::append_backtrace_oneline() at ./build/release/seastar/./seastar/src/core/reactor.cc:836
 (inlined by) seastar::print_with_backtrace(seastar::backtrace_buffer&, bool) at ./build/release/seastar/./seastar/src/core/reactor.cc:855
seastar::internal::cpu_stall_detector::generate_trace() at ./build/release/seastar/./seastar/src/core/reactor.cc:1469
seastar::internal::cpu_stall_detector::maybe_report() at ./build/release/seastar/./seastar/src/core/reactor.cc:1206
 (inlined by) seastar::internal::cpu_stall_detector::on_signal() at ./build/release/seastar/./seastar/src/core/reactor.cc:1226
 (inlined by) seastar::reactor::block_notifier(int) at ./build/release/seastar/./seastar/src/core/reactor.cc:1506
?? ??:0
?? ??:0
seastar::file_desc::sendmsg(msghdr const*, int) at ./build/release/seastar/./seastar/include/seastar/core/posix.hh:274
operator() at ./build/release/seastar/./seastar/src/core/reactor.cc:395
 (inlined by) seastar::future<unsigned long> seastar::futurize<seastar::future<unsigned long> >::invoke<seastar::reactor::do_sendmsg(seastar::pollable_fd_state&, seastar::net::packet&)::$_0>(seastar::reactor::do_sendmsg(seastar::pollable_fd_state&, seastar::net::packet&)::$_0&&) at ./build/release/seastar/./seastar/include/seastar/core/future.hh:2006
 (inlined by) seastar::future<unsigned long> seastar::futurize<seastar::future<unsigned long> >::invoke<seastar::reactor::do_sendmsg(seastar::pollable_fd_state&, seastar::net::packet&)::$_0>(seastar::reactor::do_sendmsg(seastar::pollable_fd_state&, seastar::net::packet&)::$_0&&, seastar::internal::monostate) at ./build/release/seastar/./seastar/include/seastar/core/future.hh:1848
 (inlined by) seastar::future<unsigned long> seastar::future<void>::then_impl<seastar::reactor::do_sendmsg(seastar::pollable_fd_state&, seastar::net::packet&)::$_0, seastar::future<unsigned long> >(seastar::reactor::do_sendmsg(seastar::pollable_fd_state&, seastar::net::packet&)::$_0&&) at ./build/release/seastar/./seastar/include/seastar/core/future.hh:1473
seastar::future<unsigned long> seastar::future<void>::then<seastar::reactor::do_sendmsg(seastar::pollable_fd_state&, seastar::net::packet&)::$_0, seastar::future<unsigned long> >(seastar::reactor::do_sendmsg(seastar::pollable_fd_state&, seastar::net::packet&)::$_0&&) at ./build/release/seastar/./seastar/include/seastar/core/future.hh:1399
 (inlined by) seastar::reactor::do_sendmsg(seastar::pollable_fd_state&, seastar::net::packet&) at ./build/release/seastar/./seastar/src/core/reactor.cc:382
seastar::reactor_backend_aio::sendmsg(seastar::pollable_fd_state&, seastar::net::packet&) at ./build/release/seastar/./seastar/src/core/reactor_backend.cc:677
seastar::pollable_fd_state::write_some(seastar::net::packet&) at ./build/release/seastar/./seastar/src/core/reactor.cc:458
 (inlined by) seastar::pollable_fd_state::write_all(seastar::net::packet&) at ./build/release/seastar/./seastar/src/core/reactor.cc:470
seastar::pollable_fd::write_all(seastar::net::packet&) at ./build/release/seastar/./seastar/include/seastar/core/internal/pollable_fd.hh:165
 (inlined by) seastar::net::posix_data_sink_impl::put(seastar::net::packet) at ./build/release/seastar/./seastar/src/net/posix-stack.cc:666
seastar::data_sink::put(seastar::net::packet) at ././seastar/include/seastar/core/iostream.hh:174
 (inlined by) seastar::output_stream<char>::zero_copy_put(seastar::net::packet) at ././seastar/include/seastar/core/iostream-impl.hh:87
 (inlined by) seastar::output_stream<char>::write(seastar::net::packet) at ././seastar/include/seastar/core/iostream-impl.hh:129
seastar::output_stream<char>::write(seastar::temporary_buffer<char>) at ./build/release/seastar/./seastar/include/seastar/core/iostream-impl.hh:146
seastar::rpc::connection::send_buffer(seastar::rpc::snd_buf) at ./build/release/seastar/./seastar/src/rpc/rpc.cc:124
seastar::rpc::connection::send_entry(seastar::rpc::connection::outgoing_entry&) at ./build/release/seastar/./seastar/src/rpc/rpc.cc:151
operator() at ./build/release/seastar/./seastar/src/rpc/rpc.cc:301
seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<seastar::rpc::connection::send(seastar::rpc::snd_buf, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >, seastar::rpc::cancellable*)::$_1>(seastar::rpc::connection::send(seastar::rpc::snd_buf, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >, seastar::rpc::cancellable*)::$_1&&) at ./build/release/seastar/./seastar/include/seastar/core/future.hh:2006
 (inlined by) seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<seastar::rpc::connection::send(seastar::rpc::snd_buf, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >, seastar::rpc::cancellable*)::$_1>(seastar::rpc::connection::send(seastar::rpc::snd_buf, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >, seastar::rpc::cancellable*)::$_1&&, seastar::internal::monostate) at ./build/release/seastar/./seastar/include/seastar/core/future.hh:1848
 (inlined by) seastar::future<void> seastar::future<void>::then_impl<seastar::rpc::connection::send(seastar::rpc::snd_buf, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >, seastar::rpc::cancellable*)::$_1, seastar::future<void> >(seastar::rpc::connection::send(seastar::rpc::snd_buf, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >, seastar::rpc::cancellable*)::$_1&&) at ./build/release/seastar/./seastar/include/seastar/core/future.hh:1473
 (inlined by) seastar::future<void> seastar::future<void>::then<seastar::rpc::connection::send(seastar::rpc::snd_buf, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >, seastar::rpc::cancellable*)::$_1, seastar::future<void> >(seastar::rpc::connection::send(seastar::rpc::snd_buf, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >, seastar::rpc::cancellable*)::$_1&&) at ./build/release/seastar/./seastar/include/seastar/core/future.hh:1399
 (inlined by) seastar::rpc::connection::send(seastar::rpc::snd_buf, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >, seastar::rpc::cancellable*) at ./build/release/seastar/./seastar/src/rpc/rpc.cc:293
seastar::rpc::client::request(unsigned long, long, seastar::rpc::snd_buf, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >, seastar::rpc::cancellable*) at ./build/release/seastar/./seastar/src/rpc/rpc.cc:654
send at ././seastar/include/seastar/rpc/rpc_impl.hh:478
operator() at ././seastar/include/seastar/rpc/rpc_impl.hh:487
 (inlined by) auto netw::send_message_timeout<seastar::rpc::no_wait_type, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, frozen_mutation const, utils::small_vector<gms::inet_address, 3ul> const, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info> const, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>, service::fencing_token>(netw::messaging_service*, netw::messaging_verb, netw::msg_addr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, frozen_mutation const&&, utils::small_vector<gms::inet_address, 3ul> const&&, gms::inet_address&&, unsigned int&&, unsigned long&&, std::optional<tracing::trace_info> const&&, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>&&, service::fencing_token&&) at ././message/rpc_protocol_impl.hh:157
 (inlined by) auto netw::send_message_oneway_timeout<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, frozen_mutation const, utils::small_vector<gms::inet_address, 3ul> const, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info> const, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>, service::fencing_token>(netw::messaging_service*, netw::messaging_verb, netw::msg_addr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, frozen_mutation const&&, utils::small_vector<gms::inet_address, 3ul> const&&, gms::inet_address&&, unsigned int&&, unsigned long&&, std::optional<tracing::trace_info> const&&, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>&&, service::fencing_token&&) at ././message/rpc_protocol_impl.hh:217
 (inlined by) ser::storage_proxy_rpc_verbs::send_mutation(netw::messaging_service*, netw::msg_addr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, frozen_mutation const&, utils::small_vector<gms::inet_address, 3ul> const&, gms::inet_address, unsigned int, unsigned long, std::optional<tracing::trace_info> const&, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>, service::fencing_token) at ./build/release/gen/idl/storage_proxy.dist.impl.hh:40
service::storage_proxy::remote::send_mutation(netw::msg_addr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, std::optional<tracing::trace_info> const&, frozen_mutation const&, utils::small_vector<gms::inet_address, 3ul> const&, gms::inet_address, unsigned int, unsigned long, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>, service::fencing_token) at ./service/storage_proxy.cc:234
 (inlined by) service::shared_mutation::apply_remotely(service::storage_proxy&, gms::inet_address, utils::small_vector<gms::inet_address, 3ul> const&, unsigned long, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>, service::fencing_token) at ./service/storage_proxy.cc:1095
service::abstract_write_response_handler::apply_remotely(gms::inet_address, utils::small_vector<gms::inet_address, 3ul> const&, unsigned long, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr) at ./service/storage_proxy.cc:1553
 (inlined by) operator() at ./service/storage_proxy.cc:3974
 (inlined by) seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<service::storage_proxy::send_to_live_endpoints(unsigned long, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::$_1&, gms::inet_address&, utils::small_vector<gms::inet_address, 3ul>&>(service::storage_proxy::send_to_live_endpoints(unsigned long, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::$_1&, gms::inet_address&, utils::small_vector<gms::inet_address, 3ul>&) at ././seastar/include/seastar/core/future.hh:2006
 (inlined by) auto seastar::futurize_invoke<service::storage_proxy::send_to_live_endpoints(unsigned long, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::$_1&, gms::inet_address&, utils::small_vector<gms::inet_address, 3ul>&>(service::storage_proxy::send_to_live_endpoints(unsigned long, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::$_1&, gms::inet_address&, utils::small_vector<gms::inet_address, 3ul>&) at ././seastar/include/seastar/core/future.hh:2037
 (inlined by) service::storage_proxy::send_to_live_endpoints(unsigned long, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >) at ./service/storage_proxy.cc:4004
operator() at ./service/storage_proxy.cc:3216
 (inlined by) seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > seastar::futurize<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > >::invoke<service::storage_proxy::mutate_begin(utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>, db::consistency_level, tracing::trace_state_ptr, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >)::$_0, service::storage_proxy::unique_response_handler&>(service::storage_proxy::mutate_begin(utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>, db::consistency_level, tracing::trace_state_ptr, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >)::$_0&&, service::storage_proxy::unique_response_handler&) at ././seastar/include/seastar/core/future.hh:2006
 (inlined by) auto seastar::futurize_invoke<service::storage_proxy::mutate_begin(utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>, db::consistency_level, tracing::trace_state_ptr, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >)::$_0, service::storage_proxy::unique_response_handler&>(service::storage_proxy::mutate_begin(utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>, db::consistency_level, tracing::trace_state_ptr, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >)::$_0&&, service::storage_proxy::unique_response_handler&) at ././seastar/include/seastar/core/future.hh:2037
 (inlined by) seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > utils::result_parallel_for_each<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy>, service::storage_proxy::unique_response_handler*, service::storage_proxy::mutate_begin(utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>, db::consistency_level, tracing::trace_state_ptr, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >)::$_0>(service::storage_proxy::unique_response_handler*, service::storage_proxy::unique_response_handler*, service::storage_proxy::mutate_begin(utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>, db::consistency_level, tracing::trace_state_ptr, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >)::$_0&&) at ././utils/result_loop.hh:46
 (inlined by) seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > utils::result_parallel_for_each<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy>, utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>&, service::storage_proxy::mutate_begin(utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>, db::consistency_level, tracing::trace_state_ptr, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >)::$_0>(utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>&, service::storage_proxy::mutate_begin(utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>, db::consistency_level, tracing::trace_state_ptr, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >)::$_0&&) at ././utils/result_loop.hh:102
 (inlined by) service::storage_proxy::mutate_begin(utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>, db::consistency_level, tracing::trace_state_ptr, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >) at ./service/storage_proxy.cc:3188
operator() at ./service/storage_proxy.cc:3506
 (inlined by) seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > seastar::futurize<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > >::invoke<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > service::storage_proxy::mutate_internal<boost::iterator_range<__gnu_cxx::__normal_iterator<mutation*, std::vector<mutation, std::allocator<mutation> > > > >(boost::iterator_range<__gnu_cxx::__normal_iterator<mutation*, std::vector<mutation, std::allocator<mutation> > > >, db::consistency_level, bool, tracing::trace_state_ptr, service_permit, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >, seastar::lw_shared_ptr<cdc::operation_result_tracker>, seastar::bool_class<db::allow_per_partition_rate_limit_tag>)::{lambda(utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>)#1}&, utils::small_vector<service::storage_proxy::unique_response_handler, 1ul> >(seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > service::storage_proxy::mutate_internal<boost::iterator_range<__gnu_cxx::__normal_iterator<mutation*, std::vector<mutation, std::allocator<mutation> > > > >(boost::iterator_range<__gnu_cxx::__normal_iterator<mutation*, std::vector<mutation, std::allocator<mutation> > > >, db::consistency_level, bool, tracing::trace_state_ptr, service_permit, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >, seastar::lw_shared_ptr<cdc::operation_result_tracker>, seastar::bool_class<db::allow_per_partition_rate_limit_tag>)::{lambda(utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>)#1}&, utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>&&) at ././seastar/include/seastar/core/future.hh:2006
auto seastar::futurize_invoke<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > service::storage_proxy::mutate_internal<boost::iterator_range<__gnu_cxx::__normal_iterator<mutation*, std::vector<mutation, std::allocator<mutation> > > > >(boost::iterator_range<__gnu_cxx::__normal_iterator<mutation*, std::vector<mutation, std::allocator<mutation> > > >, db::consistency_level, bool, tracing::trace_state_ptr, service_permit, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >, seastar::lw_shared_ptr<cdc::operation_result_tracker>, seastar::bool_class<db::allow_per_partition_rate_limit_tag>)::{lambda(utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>)#1}&, utils::small_vector<service::storage_proxy::unique_response_handler, 1ul> >(seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > service::storage_proxy::mutate_internal<boost::iterator_range<__gnu_cxx::__normal_iterator<mutation*, std::vector<mutation, std::allocator<mutation> > > > >(boost::iterator_range<__gnu_cxx::__normal_iterator<mutation*, std::vector<mutation, std::allocator<mutation> > > >, db::consistency_level, bool, tracing::trace_state_ptr, service_permit, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >, seastar::lw_shared_ptr<cdc::operation_result_tracker>, seastar::bool_class<db::allow_per_partition_rate_limit_tag>)::{lambda(utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>)#1}&, utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>&&) at ././seastar/include/seastar/core/future.hh:2037
 (inlined by) utils::internal::result_wrapped_call_traits<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > service::storage_proxy::mutate_internal<boost::iterator_range<__gnu_cxx::__normal_iterator<mutation*, std::vector<mutation, std::allocator<mutation> > > > >(boost::iterator_range<__gnu_cxx::__normal_iterator<mutation*, std::vector<mutation, std::allocator<mutation> > > >, db::consistency_level, bool, tracing::trace_state_ptr, service_permit, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >, seastar::lw_shared_ptr<cdc::operation_result_tracker>, seastar::bool_class<db::allow_per_partition_rate_limit_tag>)::{lambda(utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>)#1}, boost::outcome_v2::basic_result<utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy>, false>::invoke_with_value({lambda(utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>)#1}&, seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > service::storage_proxy::mutate_internal<boost::iterator_range<__gnu_cxx::__normal_iterator<mutation*, std::vector<mutation, std::allocator<mutation> > > > >(boost::iterator_range<__gnu_cxx::__normal_iterator<mutation*, std::vector<mutation, std::allocator<mutation> > > >, db::consistency_level, bool, tracing::trace_state_ptr, service_permit, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >, seastar::lw_shared_ptr<cdc::operation_result_tracker>, seastar::bool_class<db::allow_per_partition_rate_limit_tag>)::{lambda(utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>)#1}&&) at ././utils/result_combinators.hh:73
 (inlined by) auto utils::internal::result_wrapper<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > service::storage_proxy::mutate_internal<boost::iterator_range<__gnu_cxx::__normal_iterator<mutation*, std::vector<mutation, std::allocator<mutation> > > > >(boost::iterator_range<__gnu_cxx::__normal_iterator<mutation*, std::vector<mutation, std::allocator<mutation> > > >, db::consistency_level, bool, tracing::trace_state_ptr, service_permit, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >, seastar::lw_shared_ptr<cdc::operation_result_tracker>, seastar::bool_class<db::allow_per_partition_rate_limit_tag>)::{lambda(utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>)#1}, false>::operator()<boost::outcome_v2::basic_result<utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> >(boost::outcome_v2::basic_result<utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy>) at ././utils/result_combinators.hh:124
 (inlined by) seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > seastar::futurize<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > >::invoke<utils::internal::result_wrapper<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > service::storage_proxy::mutate_internal<boost::iterator_range<__gnu_cxx::__normal_iterator<mutation*, std::vector<mutation, std::allocator<mutation> > > > >(boost::iterator_range<__gnu_cxx::__normal_iterator<mutation*, std::vector<mutation, std::allocator<mutation> > > >, db::consistency_level, bool, tracing::trace_state_ptr, service_permit, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >, seastar::lw_shared_ptr<cdc::operation_result_tracker>, seastar::bool_class<db::allow_per_partition_rate_limit_tag>)::{lambda(utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>)#1}, false>, boost::outcome_v2::basic_result<utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> >(utils::internal::result_wrapper<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > service::storage_proxy::mutate_internal<boost::iterator_range<__gnu_cxx::__normal_iterator<mutation*, std::vector<mutation, std::allocator<mutation> > > > >(boost::iterator_range<__gnu_cxx::__normal_iterator<mutation*, std::vector<mutation, std::allocator<mutation> > > >, db::consistency_level, bool, tracing::trace_state_ptr, service_permit, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >, seastar::lw_shared_ptr<cdc::operation_result_tracker>, seastar::bool_class<db::allow_per_partition_rate_limit_tag>)::{lambda(utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>)#1}, false>&&, boost::outcome_v2::basic_result<utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy>&&) at ././seastar/include/seastar/core/future.hh:2006
 (inlined by) seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > seastar::future<boost::outcome_v2::basic_result<utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> >::then_impl<utils::internal::result_wrapper<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > service::storage_proxy::mutate_internal<boost::iterator_range<__gnu_cxx::__normal_iterator<mutation*, std::vector<mutation, std::allocator<mutation> > > > >(boost::iterator_range<__gnu_cxx::__normal_iterator<mutation*, std::vector<mutation, std::allocator<mutation> > > >, db::consistency_level, bool, tracing::trace_state_ptr, service_permit, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >, seastar::lw_shared_ptr<cdc::operation_result_tracker>, seastar::bool_class<db::allow_per_partition_rate_limit_tag>)::{lambda(utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>)#1}, false>, seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > >(utils::internal::result_wrapper<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > service::storage_proxy::mutate_internal<boost::iterator_range<__gnu_cxx::__normal_iterator<mutation*, std::vector<mutation, std::allocator<mutation> > > > >(boost::iterator_range<__gnu_cxx::__normal_iterator<mutation*, std::vector<mutation, std::allocator<mutation> > > >, db::consistency_level, bool, tracing::trace_state_ptr, service_permit, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >, seastar::lw_shared_ptr<cdc::operation_result_tracker>, seastar::bool_class<db::allow_per_partition_rate_limit_tag>)::{lambda(utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>)#1}, false>&&) at ././seastar/include/seastar/core/future.hh:1473
 (inlined by) seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > seastar::future<boost::outcome_v2::basic_result<utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> >::then<utils::internal::result_wrapper<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > service::storage_proxy::mutate_internal<boost::iterator_range<__gnu_cxx::__normal_iterator<mutation*, std::vector<mutation, std::allocator<mutation> > > > >(boost::iterator_range<__gnu_cxx::__normal_iterator<mutation*, std::vector<mutation, std::allocator<mutation> > > >, db::consistency_level, bool, tracing::trace_state_ptr, service_permit, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >, seastar::lw_shared_ptr<cdc::operation_result_tracker>, seastar::bool_class<db::allow_per_partition_rate_limit_tag>)::{lambda(utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>)#1}, false>, seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > >(utils::internal::result_wrapper<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > service::storage_proxy::mutate_internal<boost::iterator_range<__gnu_cxx::__normal_iterator<mutation*, std::vector<mutation, std::allocator<mutation> > > > >(boost::iterator_range<__gnu_cxx::__normal_iterator<mutation*, std::vector<mutation, std::allocator<mutation> > > >, db::consistency_level, bool, tracing::trace_state_ptr, service_permit, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >, seastar::lw_shared_ptr<cdc::operation_result_tracker>, seastar::bool_class<db::allow_per_partition_rate_limit_tag>)::{lambda(utils::small_vector<service::storage_proxy::unique_response_handler, 1ul>)#1}, false>&&) at ././seastar/include/seastar/core/future.hh:1399
 (inlined by) seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > service::storage_proxy::mutate_internal<boost::iterator_range<__gnu_cxx::__normal_iterator<mutation*, std::vector<mutation, std::allocator<mutation> > > > >(boost::iterator_range<__gnu_cxx::__normal_iterator<mutation*, std::vector<mutation, std::allocator<mutation> > > >, db::consistency_level, bool, tracing::trace_state_ptr, service_permit, std::optional<std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > >, seastar::lw_shared_ptr<cdc::operation_result_tracker>, seastar::bool_class<db::allow_per_partition_rate_limit_tag>) at ./service/storage_proxy.cc:3503
service::storage_proxy::do_mutate(std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker>) at ./service/storage_proxy.cc:3463
seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > std::__invoke_impl<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> >, seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > (service::storage_proxy::* const&)(std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker>), service::storage_proxy*, std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker> >(std::__invoke_memfun_deref, seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > (service::storage_proxy::* const&)(std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker>), service::storage_proxy*&&, std::vector<mutation, std::allocator<mutation> >&&, db::consistency_level&&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >&&, tracing::trace_state_ptr&&, service_permit&&, bool&&, seastar::bool_class<db::allow_per_partition_rate_limit_tag>&&, seastar::lw_shared_ptr<cdc::operation_result_tracker>&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/bits/invoke.h:74
std::__invoke_result<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > (service::storage_proxy::* const&)(std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker>), service::storage_proxy*, std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker> >::type std::__invoke<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > (service::storage_proxy::* const&)(std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker>), service::storage_proxy*, std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker> >(seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > (service::storage_proxy::* const&)(std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker>), service::storage_proxy*&&, std::vector<mutation, std::allocator<mutation> >&&, db::consistency_level&&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >&&, tracing::trace_state_ptr&&, service_permit&&, bool&&, seastar::bool_class<db::allow_per_partition_rate_limit_tag>&&, seastar::lw_shared_ptr<cdc::operation_result_tracker>&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/bits/invoke.h:96
 (inlined by) _ZNKSt12_Mem_fn_baseIMN7service13storage_proxyEFN7seastar6futureIN5boost10outcome_v212basic_resultIvN5utils19exception_containerIJN10exceptions32mutation_write_timeout_exceptionENS9_22read_timeout_exceptionENS9_22read_failure_exceptionENS9_20rate_limit_exceptionEEEENS7_32exception_container_throw_policyEEEEESt6vectorI8mutationSaISJ_EEN2db17consistency_levelENSt6chrono10time_pointINS2_12lowres_clockENSO_8durationIlSt5ratioILl1ELl1000000000EEEEEEN7tracing15trace_state_ptrE14service_permitbNS2_10bool_classINSM_34allow_per_partition_rate_limit_tagEEENS2_13lw_shared_ptrIN3cdc24operation_result_trackerEEEELb1EEclIJPS1_SL_SN_SV_SX_SY_bS11_S15_EEEDTclsr3stdE8__invokedtdefpT6_M_pmfspclsr3stdE7forwardIT_Efp_EEEDpOS1B_ at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/functional:170
 (inlined by) seastar::noncopyable_function<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > (service::storage_proxy*, std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker>)>::direct_vtable_for<std::_Mem_fn<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > (service::storage_proxy::*)(std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker>)> >::call(seastar::noncopyable_function<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > (service::storage_proxy*, std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker>)> const*, service::storage_proxy*, std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker>) at ././seastar/include/seastar/util/noncopyable_function.hh:129
seastar::noncopyable_function<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > (service::storage_proxy*, std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker>)>::operator()(service::storage_proxy*, std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker>) const at ././seastar/include/seastar/util/noncopyable_function.hh:215
operator() at ././seastar/include/seastar/core/execution_stage.hh:342
seastar::noncopyable_function<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > (service::storage_proxy*, std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker>)>::direct_vtable_for<seastar::inheriting_concrete_execution_stage<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> >, service::storage_proxy*, std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker> >::make_stage_for_group(seastar::scheduling_group)::{lambda(service::storage_proxy*, std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker>)#1}>::call(seastar::noncopyable_function<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > (service::storage_proxy*, std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker>)> const*, service::storage_proxy*, std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker>) at ././seastar/include/seastar/util/noncopyable_function.hh:129
seastar::noncopyable_function<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > (service::storage_proxy*, std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker>)>::operator()(service::storage_proxy*, std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker>) const at ././seastar/include/seastar/util/noncopyable_function.hh:215
seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > std::__invoke_impl<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> >, seastar::noncopyable_function<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > (service::storage_proxy*, std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker>)>&, service::storage_proxy*, std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker> >(std::__invoke_other, seastar::noncopyable_function<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > (service::storage_proxy*, std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker>)>&, service::storage_proxy*&&, std::vector<mutation, std::allocator<mutation> >&&, db::consistency_level&&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >&&, tracing::trace_state_ptr&&, service_permit&&, bool&&, seastar::bool_class<db::allow_per_partition_rate_limit_tag>&&, seastar::lw_shared_ptr<cdc::operation_result_tracker>&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/bits/invoke.h:61
std::__invoke_result<seastar::noncopyable_function<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > (service::storage_proxy*, std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker>)>&, service::storage_proxy*, std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker> >::type std::__invoke<seastar::noncopyable_function<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > (service::storage_proxy*, std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker>)>&, service::storage_proxy*, std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker> >(seastar::noncopyable_function<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > (service::storage_proxy*, std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker>)>&, service::storage_proxy*&&, std::vector<mutation, std::allocator<mutation> >&&, db::consistency_level&&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >&&, tracing::trace_state_ptr&&, service_permit&&, bool&&, seastar::bool_class<db::allow_per_partition_rate_limit_tag>&&, seastar::lw_shared_ptr<cdc::operation_result_tracker>&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/bits/invoke.h:96
 (inlined by) _ZSt12__apply_implIRN7seastar20noncopyable_functionIFNS0_6futureIN5boost10outcome_v212basic_resultIvN5utils19exception_containerIJN10exceptions32mutation_write_timeout_exceptionENS8_22read_timeout_exceptionENS8_22read_failure_exceptionENS8_20rate_limit_exceptionEEEENS6_32exception_container_throw_policyEEEEEPN7service13storage_proxyESt6vectorI8mutationSaISL_EEN2db17consistency_levelENSt6chrono10time_pointINS0_12lowres_clockENSQ_8durationIlSt5ratioILl1ELl1000000000EEEEEEN7tracing15trace_state_ptrE14service_permitbNS0_10bool_classINSO_34allow_per_partition_rate_limit_tagEEENS0_13lw_shared_ptrIN3cdc24operation_result_trackerEEEEEESt5tupleIJSJ_SN_SP_SX_SZ_S10_bS13_S17_EEJLm0ELm1ELm2ELm3ELm4ELm5ELm6ELm7ELm8EEEDcOT_OT0_St16integer_sequenceImJXspT1_EEE at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/tuple:2288
 (inlined by) _ZSt5applyIRN7seastar20noncopyable_functionIFNS0_6futureIN5boost10outcome_v212basic_resultIvN5utils19exception_containerIJN10exceptions32mutation_write_timeout_exceptionENS8_22read_timeout_exceptionENS8_22read_failure_exceptionENS8_20rate_limit_exceptionEEEENS6_32exception_container_throw_policyEEEEEPN7service13storage_proxyESt6vectorI8mutationSaISL_EEN2db17consistency_levelENSt6chrono10time_pointINS0_12lowres_clockENSQ_8durationIlSt5ratioILl1ELl1000000000EEEEEEN7tracing15trace_state_ptrE14service_permitbNS0_10bool_classINSO_34allow_per_partition_rate_limit_tagEEENS0_13lw_shared_ptrIN3cdc24operation_result_trackerEEEEEESt5tupleIJSJ_SN_SP_SX_SZ_S10_bS13_S17_EEEDcOT_OT0_ at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/tuple:2299
 (inlined by) seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > seastar::futurize<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > >::apply<seastar::noncopyable_function<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > (service::storage_proxy*, std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker>)>&, service::storage_proxy*, std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker> >(seastar::noncopyable_function<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> > (service::storage_proxy*, std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker>)>&, std::tuple<service::storage_proxy*, std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker> >&&) at ././seastar/include/seastar/core/future.hh:1973
 (inlined by) seastar::concrete_execution_stage<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception, exceptions::rate_limit_exception>, utils::exception_container_throw_policy> >, service::storage_proxy*, std::vector<mutation, std::allocator<mutation> >, db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tracing::trace_state_ptr, service_permit, bool, seastar::bool_class<db::allow_per_partition_rate_limit_tag>, seastar::lw_shared_ptr<cdc::operation_result_tracker> >::do_flush() at ././seastar/include/seastar/core/execution_stage.hh:251
operator() at ./build/release/seastar/./seastar/src/core/execution_stage.cc:149
 (inlined by) seastar::future<void> seastar::futurize<void>::invoke<seastar::execution_stage::flush()::$_0&>(seastar::execution_stage::flush()::$_0&) at ./build/release/seastar/./seastar/include/seastar/core/future.hh:2003
 (inlined by) seastar::lambda_task<seastar::execution_stage::flush()::$_0>::run_and_dispose() at ./build/release/seastar/./seastar/include/seastar/core/make_task.hh:45
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at ./build/release/seastar/./seastar/src/core/reactor.cc:2651
 (inlined by) seastar::reactor::run_some_tasks() at ./build/release/seastar/./seastar/src/core/reactor.cc:3114
seastar::reactor::do_run() at ./build/release/seastar/./seastar/src/core/reactor.cc:3283
operator() at ./build/release/seastar/./seastar/src/core/reactor.cc:4506
 (inlined by) void std::__invoke_impl<void, seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_0&>(std::__invoke_other, seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_0&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/bits/invoke.h:61
 (inlined by) std::enable_if<is_invocable_r_v<void, seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_0&>, void>::type std::__invoke_r<void, seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_0&>(seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_0&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/bits/invoke.h:111
 (inlined by) std::_Function_handler<void (), seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_0>::_M_invoke(std::_Any_data const&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/bits/std_function.h:290
std::function<void ()>::operator()() const at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/bits/std_function.h:591
 (inlined by) seastar::posix_thread::start_routine(void*) at ./build/release/seastar/./seastar/src/core/posix.cc:90
?? ??:0
?? ??:0
?? ??:0
aleksbykov commented 9 months ago

Could be similar to: https://github.com/scylladb/scylladb/issues/9621 https://github.com/scylladb/scylladb/issues/11805

mykaul commented 9 months ago

I think we are wasting time with disrupt_memory_stress nemesis. It does not find real bugs, IMHO.