search

scylladb / scylladb

NoSQL data store using the seastar framework, compatible with Apache Cassandra
http://scylladb.com
GNU Affero General Public License v3.0
13.58k stars 1.29k forks source link

A lot of warning, errors and oversized allocation during repair based operation - rebuild #7205

Open aleksbykov opened 4 years ago

aleksbykov commented 4 years ago

Installation details Scylla version (or git commit hash): 4.2.rc4-0.20200907.bf0c493c28 Cluster size: 6 OS (RHEL/CentOS/Ubuntu/AWS AMI): ami-0787db7b551e1d03c(eu-north-1) instance type: i3.4xlarge

During job https://jenkins.scylladb.com/job/scylla-4.2/job/longevity/job/longevity-cdc-100gb-4h-test/45/ latest nemesis was RebuildStreamingErr. This nemesis stops the scylla, then remove several sstables:

2020-09-08T15:06:32+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !NOTICE  | sudo:  centos : TTY=unknown ; PWD=/home/centos ; USER=root ; COMMAND=/bin/rm -f /var/lib/scylla/data/cdc_test/test_table_preimage_postimage_scylla_cdc_log-2fe2af00f1d311eaa3bf00000000000a/mc-1568-big-CompressionInfo.db /var/lib/scylla/data/cdc_test/test_table_preimage_postimage_scylla_cdc_log-2fe2af00f1d311eaa3bf00000000000a/mc-1568-big-Data.db /var/lib/scylla/data/cdc_test/test_table_preimage_postimage_scylla_cdc_log-2fe2af00f1d311eaa3bf00000000000a/mc-1568-big-Digest.crc32 /var/lib/scylla/data/cdc_test/test_table_preimage_postimage_scylla_cdc_log-2fe2af00f1d311eaa3bf00000000000a/mc-1568-big-Filter.db /var/lib/scylla/data/cdc_test/test_table_preimage_postimage_scylla_cdc_log-2fe2af00f1d311eaa3bf00000000000a/mc-1568-big-Index.db /var/lib/scylla/data/cdc_test/test_table_preimage_postimage_scylla_cdc_log-2fe2af00f1d311eaa3bf00000000000a/mc-1568-big-Scylla.db
2020-09-08T15:06:32+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !NOTICE  | sudo:  centos : (command continued) /var/lib/scylla/data/cdc_test/test_table_preimage_postimage_scylla_cdc_log-2fe2af00f1d311eaa3bf00000000000a/mc-1568-big-Statistics.db /var/lib/scylla/data/cdc_test/test_table_preimage_postimage_scylla_cdc_log-2fe2af00f1d311eaa3bf00000000000a/mc-1568-big-Summary.db /var/lib/scylla/data/cdc_test/test_table_preimage_postimage_scylla_cdc_log-2fe2af00f1d311eaa3bf00000000000a/mc-1568-big-TOC.txt
2020-09-08T15:06:32+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | sudo: pam_unix(sudo:session): session opened for user root by (uid=0)
2020-09-08T15:06:32+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | sudo: pam_unix(sudo:session): session closed for user root
2020-09-08T15:06:33+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | sshd[36501]: Received disconnect from 173.94.201.78 port 40474:11: Bye Bye [preauth]
2020-09-08T15:06:33+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | sshd[36501]: Disconnected from 173.94.201.78 port 40474 [preauth]
2020-09-08T15:06:33+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !NOTICE  | sudo:  centos : TTY=unknown ; PWD=/home/centos ; USER=root ; COMMAND=/bin/sh -c find /var/lib/scylla/data/cdc_test/test_table_preimage_scylla_cdc_log-* -maxdepth 1 -type f
2020-09-08T15:06:33+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | sudo: pam_unix(sudo:session): session opened for user root by (uid=0)
2020-09-08T15:06:33+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | sudo: pam_unix(sudo:session): session closed for user root
2020-09-08T15:06:33+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !NOTICE  | sudo:  centos : TTY=unknown ; PWD=/home/centos ; USER=root ; COMMAND=/bin/rm -f /var/lib/scylla/data/cdc_test/test_table_preimage_scylla_cdc_log-df616672f1d211ea983a000000000003/mc-1338-big-CompressionInfo.db /var/lib/scylla/data/cdc_test/test_table_preimage_scylla_cdc_log-df616672f1d211ea983a000000000003/mc-1338-big-Data.db /var/lib/scylla/data/cdc_test/test_table_preimage_scylla_cdc_log-df616672f1d211ea983a000000000003/mc-1338-big-Digest.crc32 /var/lib/scylla/data/cdc_test/test_table_preimage_scylla_cdc_log-df616672f1d211ea983a000000000003/mc-1338-big-Filter.db /var/lib/scylla/data/cdc_test/test_table_preimage_scylla_cdc_log-df616672f1d211ea983a000000000003/mc-1338-big-Index.db /var/lib/scylla/data/cdc_test/test_table_preimage_scylla_cdc_log-df616672f1d211ea983a000000000003/mc-1338-big-Scylla.db /var/lib/scylla/data/cdc_test/test_table_preimage_scylla_cdc_log-df616672f1d211ea983a000000000003/mc-1338-big-Statistics.db
2020-09-08T15:06:33+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !NOTICE  | sudo:  centos : (command continued) /var/lib/scylla/data/cdc_test/test_table_preimage_scylla_cdc_log-df616672f1d211ea983a000000000003/mc-1338-big-Summary.db /var/lib/scylla/data/cdc_test/test_table_preimage_scylla_cdc_log-df616672f1d211ea983a000000000003/mc-1338-big-TOC.txt
2020-09-08T15:06:33+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | sudo: pam_unix(sudo:session): session opened for user root by (uid=0)
2020-09-08T15:06:33+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | sudo: pam_unix(sudo:session): session closed for user root
2020-09-08T15:06:33+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !NOTICE  | sudo:  centos : TTY=unknown ; PWD=/home/centos ; USER=root ; COMMAND=/bin/sh -c find /var/lib/scylla/data/cdc_test/test_table-* -maxdepth 1 -type f
2020-09-08T15:06:33+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | sudo: pam_unix(sudo:session): session opened for user root by (uid=0)
2020-09-08T15:06:33+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | sudo: pam_unix(sudo:session): session closed for user root
2020-09-08T15:06:33+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !NOTICE  | sudo:  centos : TTY=unknown ; PWD=/home/centos ; USER=root ; COMMAND=/bin/rm -f /var/lib/scylla/data/cdc_test/test_table-de936810f1d211ea8b07000000000008/mc-518-big-CompressionInfo.db /var/lib/scylla/data/cdc_test/test_table-de936810f1d211ea8b07000000000008/mc-518-big-Data.db /var/lib/scylla/data/cdc_test/test_table-de936810f1d211ea8b07000000000008/mc-518-big-Digest.crc32 /var/lib/scylla/data/cdc_test/test_table-de936810f1d211ea8b07000000000008/mc-518-big-Filter.db /var/lib/scylla/data/cdc_test/test_table-de936810f1d211ea8b07000000000008/mc-518-big-Index.db /var/lib/scylla/data/cdc_test/test_table-de936810f1d211ea8b07000000000008/mc-518-big-Scylla.db /var/lib/scylla/data/cdc_test/test_table-de936810f1d211ea8b07000000000008/mc-518-big-Statistics.db /var/lib/scylla/data/cdc_test/test_table-de936810f1d211ea8b07000000000008/mc-518-big-Summary.db /var/lib/scylla/data/cdc_test/test_table-de936810f1d211ea8b07000000000008/mc-518-big-TOC.txt
2020-09-08T15:06:33+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | sudo: pam_unix(sudo:session): session opened for user root by (uid=0)
2020-09-08T15:06:33+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | sudo: pam_unix(sudo:session): session closed for user root
2020-09-08T15:06:33+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !NOTICE  | sudo:  centos : TTY=unknown ; PWD=/home/centos ; USER=root ; COMMAND=/bin/sh -c find /var/lib/scylla/data/cdc_test/test_table_preimage_postimage-* -maxdepth 1 -type f
2020-09-08T15:06:33+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | sudo: pam_unix(sudo:session): session opened for user root by (uid=0)
2020-09-08T15:06:33+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | sudo: pam_unix(sudo:session): session closed for user root
2020-09-08T15:06:33+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !NOTICE  | sudo:  centos : TTY=unknown ; PWD=/home/centos ; USER=root ; COMMAND=/bin/rm -f /var/lib/scylla/data/cdc_test/test_table_preimage_postimage-2fe287f0f1d311eaa3bf00000000000a/mc-467-big-CompressionInfo.db /var/lib/scylla/data/cdc_test/test_table_preimage_postimage-2fe287f0f1d311eaa3bf00000000000a/mc-467-big-Data.db /var/lib/scylla/data/cdc_test/test_table_preimage_postimage-2fe287f0f1d311eaa3bf00000000000a/mc-467-big-Digest.crc32 /var/lib/scylla/data/cdc_test/test_table_preimage_postimage-2fe287f0f1d311eaa3bf00000000000a/mc-467-big-Filter.db /var/lib/scylla/data/cdc_test/test_table_preimage_postimage-2fe287f0f1d311eaa3bf00000000000a/mc-467-big-Index.db /var/lib/scylla/data/cdc_test/test_table_preimage_postimage-2fe287f0f1d311eaa3bf00000000000a/mc-467-big-Scylla.db /var/lib/scylla/data/cdc_test/test_table_preimage_postimage-2fe287f0f1d311eaa3bf00000000000a/mc-467-big-Statistics.db
2020-09-08T15:06:33+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !NOTICE  | sudo:  centos : (command continued) /var/lib/scylla/data/cdc_test/test_table_preimage_postimage-2fe287f0f1d311eaa3bf00000000000a/mc-467-big-Summary.db /var/lib/scylla/data/cdc_test/test_table_preimage_postimage-2fe287f0f1d311eaa3bf00000000000a/mc-467-big-TOC.txt
2020-09-08T15:06:33+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | sudo: pam_unix(sudo:session): session opened for user root by (uid=0)
2020-09-08T15:06:33+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | sudo: pam_unix(sudo:session): session closed for user root
2020-09-08T15:06:33+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !NOTICE  | sudo:  centos : TTY=unknown ; PWD=/home/centos ; USER=root ; COMMAND=/bin/sh -c find /var/lib/scylla/data/cdc_test/test_table_preimage-* -maxdepth 1 -type f
2020-09-08T15:06:33+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | sudo: pam_unix(sudo:session): session opened for user root by (uid=0)
2020-09-08T15:06:33+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | sudo: pam_unix(sudo:session): session closed for user root
2020-09-08T15:06:33+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !NOTICE  | sudo:  centos : TTY=unknown ; PWD=/home/centos ; USER=root ; COMMAND=/bin/rm -f /var/lib/scylla/data/cdc_test/test_table_preimage-df616670f1d211ea983a000000000003/mc-498-big-CompressionInfo.db /var/lib/scylla/data/cdc_test/test_table_preimage-df616670f1d211ea983a000000000003/mc-498-big-Data.db /var/lib/scylla/data/cdc_test/test_table_preimage-df616670f1d211ea983a000000000003/mc-498-big-Digest.crc32 /var/lib/scylla/data/cdc_test/test_table_preimage-df616670f1d211ea983a000000000003/mc-498-big-Filter.db /var/lib/scylla/data/cdc_test/test_table_preimage-df616670f1d211ea983a000000000003/mc-498-big-Index.db /var/lib/scylla/data/cdc_test/test_table_preimage-df616670f1d211ea983a000000000003/mc-498-big-Scylla.db /var/lib/scylla/data/cdc_test/test_table_preimage-df616670f1d211ea983a000000000003/mc-498-big-Statistics.db /var/lib/scylla/data/cdc_test/test_table_preimage-df616670f1d211ea983a000000000003/mc-498-big-Summary.db
2020-09-08T15:06:33+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !NOTICE  | sudo:  centos : (command continued) /var/lib/scylla/data/cdc_test/test_table_preimage-df616670f1d211ea983a000000000003/mc-498-big-TOC.txt

Then start scylla and wait it is up and run nodetool rebuild. While rebuilding is running node is hard rebooted.

2020-09-08T15:10:15+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | scylla: [shard 0] repair - Repair 132 out of 768 ranges, id=2, shard=0, keyspace=cdc_test, table={test_table_preimage_postimage_scylla_cdc_log, test_table_postimage_scylla_cdc_log, test_table_postimage, test_table_preimage, test_table_scylla_cdc_log, test_table_preimage_scylla_cdc_log, test_table, test_table_preimage_postimage}, range=(-5699084353407894596, -5696150695988892296]
2020-09-08T15:10:15+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | sshd[37969]: pam_unix(sshd:session): session closed for user centos
2020-09-08T15:10:15+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | systemd-logind: Removed session 818.
2020-09-08T15:10:15+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | scylla: [shard 0] repair - Repair 133 out of 768 ranges, id=2, shard=0, keyspace=cdc_test, table={test_table_preimage_postimage_scylla_cdc_log, test_table_postimage_scylla_cdc_log, test_table_postimage, test_table_preimage, test_table_scylla_cdc_log, test_table_preimage_scylla_cdc_log, test_table, test_table_preimage_postimage}, range=(-5696150695988892296, -5667135860316636920]
2020-09-08T15:10:15+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | scylla: [shard 7] repair - Repair 131 out of 768 ranges, id=2, shard=7, keyspace=cdc_test, table={test_table_preimage_postimage_scylla_cdc_log, test_table_postimage_scylla_cdc_log, test_table_postimage, test_table_preimage, test_table_scylla_cdc_log, test_table_preimage_scylla_cdc_log, test_table, test_table_preimage_postimage}, range=(-5879053870079713325, -5857675979813888023]
2020-09-08T15:10:15+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | systemd: Received SIGINT.
2020-09-08T15:10:15+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | systemd: Stopping Session 2 of user centos.
2020-09-08T15:10:15+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | systemd: Stopped target rpc_pipefs.target.
2020-09-08T15:10:15+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | systemd: Unmounting RPC Pipe File System...
2020-09-08T15:10:15+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | sshd[1584]: pam_unix(sshd:session): session closed for user centos
2020-09-08T15:10:15+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | systemd: Stopping Scylla JMX...
2020-09-08T15:10:15+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | systemd: Stopping Authorization Manager...
2020-09-08T15:10:15+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | systemd: Stopped target Cloud-init target.
2020-09-08T15:10:15+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | systemd: Stopped Execute cloud user/final scripts.
2020-09-08T15:10:15+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | systemd: Removed slice system-selinux\x2dpolicy\x2dmigrate\x2dlocal\x2dchanges.slice.
2020-09-08T15:10:15+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | systemd: Stopped target Multi-User System.
2020-09-08T15:14:49+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !NOTICE  | kernel: Linux version 5.7.6-1.el7.elrepo.x86_64 (mockbuild@Build64R7) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-39) (GCC), GNU ld version 2.27-43.base.el7_8.1) #1 SMP Fri Jun 26 02:56:56 EDT 2020
2020-09-08T15:14:49+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-5.7.6-1.el7.elrepo.x86_64 root=UUID=0a84de8e-5bfe-43e7-992b-5bfff8cdce43 ro console=tty0 crashkernel=auto console=ttyS0,115200 clocksource=tsc tsc=reliable net.ifnames=0
2020-09-08T15:14:49+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | kernel: x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
2020-09-08T15:14:49+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | kernel: x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
2020-09-08T15:14:49+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | kernel: x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
2020-09-08T15:14:49+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | kernel: x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256
2020-09-08T15:14:49+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | kernel: x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format.

After that it waits while scylla is up and run again nodetool rebuild

2020-09-08T15:17:13+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | scylla: [shard 0] thrift_controller - Thrift server listening on 10.0.3.65:9160 ...
2020-09-08T15:17:13+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | scylla: [shard 0] init - serving
2020-09-08T15:17:13+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | scylla: [shard 0] init - Scylla version 4.2.rc4-0.20200907.bf0c493c28 initialization completed.
2020-09-08T15:17:28+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | scylla: [shard 0] storage_service - rebuild from dc: (any dc)
2020-09-08T15:17:28+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | scylla: [shard 0] repair - rebuild_with_repair: started with keyspaces={system_distributed, cdc_test, system_auth, keyspace1, system_traces}, source_dc=eu-north
2020-09-08T15:17:28+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | scylla: [shard 0] repair - rebuild_with_repair: started with keyspace=system_distributed, source_dc=eu-north, nr_ranges=768
2020-09-08T15:17:28+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | scylla: [shard 0] repair - repair id 1 to sync data for keyspace=system_distributed, status=started
2020-09-08T15:17:28+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | scylla: [shard 0] repair - Repair 1 out of 768 ranges, id=1, shard=0, keyspace=system_distributed, table={cdc_generations, view_build_status, cdc_streams}, range=(-9170120285580919132, -9163989809594868941]

This rebuild had been running for ~2 hours there are a lot of warning, errors messages:

2020-09-08T15:48:21+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !ERR     | scylla: [shard 13] storage_proxy - exception during mutation write to 10.0.3.65: seastar::named_semaphore_timed_out (Semaphore timed out: smp_service_group#2 2->13 semaphore)
2020-09-08T15:48:21+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !ERR     | scylla: [shard 13] storage_proxy - exception during mutation write to 10.0.3.65: seastar::named_semaphore_timed_out (Semaphore timed out: smp_service_group#2 4->13 semaphore)
2020-09-08T15:48:21+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !ERR     | scylla: [shard 13] storage_proxy - exception during mutation write to 10.0.3.65: seastar::named_semaphore_timed_out (Semaphore timed out: smp_service_group#2 4->13 semaphore)
2020-09-08T15:48:21+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | scylla: [shard 3] repair - Repair 601 out of 768 ranges, id=2, shard=3, keyspace=cdc_test, table={test_table_scylla_cdc_log, test_table, test_table_postimage_scylla_cdc_log, test_table_preimage, test_table_postimage, test_table_preimage_postimage_scylla_cdc_log, test_table_preimage_postimage, test_table_preimage_scylla_cdc_log}, range=(5019186291919357794, 5025148025909796251]
2020-09-08T15:48:22+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !WARNING | scylla: [shard 13] repair - repair id 2 on shard 13, keyspace=cdc_test, cf=test_table_preimage, range=(9115195453211822725, 9145549985566761651], got error in row level repair: std::runtime_error (get_repair_meta: repair_meta_id 97741 for node 10.0.3.65 does not exist)
2020-09-08T15:48:22+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !WARNING | scylla: [shard 13] repair - repair id 2 on shard 13, keyspace=cdc_test, cf=test_table_preimage_postimage, range=(8889616484391223530, 8919026584701873986], got error in row level repair: std::runtime_error (get_full_row_hashes: Peer failed to process)
2020-09-08T15:48:22+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !WARNING | scylla: [shard 13] repair - repair id 2 on shard 13, keyspace=cdc_test, cf=test_table_postimage_scylla_cdc_log, range=(9163423337630610980, 9198735911656716164], got error in row level repair: std::runtime_error (get_repair_meta: repair_meta_id 97439 for node 10.0.3.65 does not exist)
2020-09-08T15:48:22+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !WARNING | scylla: [shard 13] repair - repair id 2 on shard 13, keyspace=cdc_test, cf=test_table_preimage_scylla_cdc_log, range=(9073831044666273016, 9089748399629900978], got error in row level repair: std::runtime_error (get_full_row_hashes: Peer failed to process)
2020-09-08T15:48:22+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !WARNING | scylla: [shard 13] repair - repair id 2 on shard 13, keyspace=cdc_test, cf=test_table_preimage_postimage_scylla_cdc_log, range=(8766137089546557658, 8815284578756880444], got error in row level repair: std::runtime_error (get_full_row_hashes: Peer failed to process)
2020-09-08T15:48:23+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !ERR     | scylla: [shard 13] storage_proxy - exception during mutation write to 10.0.3.65: seastar::named_semaphore_timed_out (Semaphore timed out: smp_service_group#2 7->13 semaphore)
2020-09-08T15:48:23+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !ERR     | scylla: [shard 13] storage_proxy - exception during mutation write to 10.0.3.65: seastar::named_semaphore_timed_out (Semaphore timed out: smp_service_group#2 7->13 semaphore)
2020-09-08T15:48:23+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !ERR     | scylla: [shard 13] storage_proxy - exception during mutation write to 10.0.3.65: seastar::named_semaphore_timed_out (Semaphore timed out: smp_service_group#2 6->13 semaphore)
2020-09-08T15:48:23+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !ERR     | scylla: [shard 13] storage_proxy - exception during mutation write to 10.0.3.65: seastar::named_semaphore_timed_out (Semaphore timed out: smp_service_group#2 6->13 semaphore)
2020-09-08T15:48:23+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !ERR     | scylla: [shard 13] storage_proxy - exception during mutation write to 10.0.3.65: seastar::named_semaphore_timed_out (Semaphore timed out: smp_service_group#2 4->13 semaphore)
2020-09-08T15:48:23+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !ERR     | scylla: [shard 13] storage_proxy - exception during mutation write to 10.0.3.65: seastar::named_semaphore_timed_out (Semaphore timed out: smp_service_group#2 4->13 semaphore)
addr2line -Cpife /usr/lib/debug/opt/scylladb/libexec/scylla-4.2.rc4-0.20200907.bf0c493c28.x86_64.debug 0x333167d#012   0x3331990#012   0x3331e19#012   0x2e289c3#012   0x2e2d4fb#012   0x2eee2a4#012   0x2eedce4#012   0x3148d14#012   0x3149032#012   0x2e75b0d#012   0x2deca04#012   0x2df71a3#012   0xef8811#012   0xef8b08#012   0x26ed300#012   0x2df4eee#012   0x2e1e8c4#012   0x2e1f881#012   0x12acfed#012   0x12bd71a#012   0x12bfffb#012   0x12ef684#012   0x12f0bfb#012   0x1392f1f#012   0x1393b9e#012   0x13b239a#012   0x1167e5c#012   0x116912a#012   0x116bfa5#012   0x116e51d#012   0x116ea9a#012   0x117aa61#012   0x117b00f#012   0x2e77ea7#012   0x2e7821e#012   0x2eaf2fd#012   0x2ebecfa#012   0x2e42f7d#012   /opt/scylladb/libreloc/libpthread.so.0+0x9431#012   /opt/scylladb/libreloc/libc.so.6+0x101912#012 
seastar::current_backtrace_tasklocal() at backtrace.cc:?
 (inlined by) seastar::current_backtrace_tasklocal() at /jenkins/workspace/scylla-4.2/build/scylla/seastar/src/util/backtrace.cc:86
seastar::current_tasktrace() at backtrace.cc:?
seastar::current_backtrace() at backtrace.cc:?
seastar::memory::cpu_pages::warn_large_allocation(unsigned long) at /usr/include/c++/10/bits/basic_string.h:323
seastar::memory::cpu_pages::check_large_allocation(unsigned long) at /usr/include/c++/10/bits/basic_string.h:323
 (inlined by) seastar::memory::cpu_pages::allocate_large(unsigned int) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/src/core/memory.cc:677
 (inlined by) seastar::memory::allocate_large(unsigned long) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/src/core/memory.cc:1203
 (inlined by) seastar::memory::allocate(unsigned long) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/src/core/memory.cc:1255
 (inlined by) operator new(unsigned long) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/src/core/memory.cc:1685
__gnu_cxx::new_allocator<seastar::priority_class::request>::allocate(unsigned long, void const*) at /usr/include/c++/10/ext/new_allocator.h:115
 (inlined by) std::allocator<seastar::priority_class::request>::allocate(unsigned long) at /usr/include/c++/10/bits/allocator.h:173
 (inlined by) seastar::circular_buffer<seastar::priority_class::request, std::allocator<seastar::priority_class::request> >::expand(unsigned long) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/circular_buffer.hh:301
seastar::circular_buffer<seastar::priority_class::request, std::allocator<seastar::priority_class::request> >::expand() at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/circular_buffer.hh:295
 (inlined by) seastar::circular_buffer<seastar::priority_class::request, std::allocator<seastar::priority_class::request> >::maybe_expand(unsigned long) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/circular_buffer.hh:331
 (inlined by) seastar::circular_buffer<seastar::priority_class::request, std::allocator<seastar::priority_class::request> >::push_back(seastar::priority_class::request&&) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/circular_buffer.hh:380
 (inlined by) seastar::fair_queue::queue(seastar::lw_shared_ptr<seastar::priority_class>, seastar::fair_queue_ticket, seastar::noncopyable_function<void ()>) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/src/core/fair_queue.cc:152
seastar::io_queue::queue_request(seastar::io_priority_class const&, unsigned long, seastar::internal::io_request)::{lambda()#1}::operator()() at io_queue.cc:?
seastar::io_queue::queue_request(seastar::io_priority_class const&, unsigned long, seastar::internal::io_request) at io_queue.cc:?
 (inlined by) submit_to<seastar::io_queue::queue_request(const seastar::io_priority_class&, size_t, seastar::internal::io_request)::<lambda()> > at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/smp.hh:333
 (inlined by) submit_to<seastar::io_queue::queue_request(const seastar::io_priority_class&, size_t, seastar::internal::io_request)::<lambda()> > at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/smp.hh:360
 (inlined by) seastar::io_queue::queue_request(seastar::io_priority_class const&, unsigned long, seastar::internal::io_request) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/src/core/io_queue.cc:305
seastar::reactor::submit_io_read(seastar::io_queue*, seastar::io_priority_class const&, unsigned long, seastar::internal::io_request) at /usr/include/fmt/format.h:2188
seastar::posix_file_impl::read_dma(unsigned long, void*, unsigned long, seastar::io_priority_class const&) at /usr/include/c++/10/bits/basic_string.h:323
seastar::posix_file_impl::dma_read_bulk(unsigned long, unsigned long, seastar::io_priority_class const&) at /usr/include/c++/10/bits/basic_string.h:323
checked_file_impl::dma_read_bulk(unsigned long, unsigned long, seastar::io_priority_class const&)::{lambda()#1}::operator()() const at /usr/include/fmt/format.h:2188
 (inlined by) auto do_io_check<checked_file_impl::dma_read_bulk(unsigned long, unsigned long, seastar::io_priority_class const&)::{lambda()#1}, , void>(std::function<void (std::__exception_ptr::exception_ptr)> const&, checked_file_impl::dma_read_bulk(unsigned long, unsigned long, seastar::io_priority_class const&)::{lambda()#1}&&) at /jenkins/workspace/scylla-4.2/build/scylla/utils/disk-error-handler.hh:73
checked_file_impl::dma_read_bulk(unsigned long, unsigned long, seastar::io_priority_class const&) at /usr/include/fmt/format.h:2188
tracking_file_impl::dma_read_bulk(unsigned long, unsigned long, seastar::io_priority_class const&) at reader_concurrency_semaphore.cc:?
seastar::file::dma_read_bulk_impl(unsigned long, unsigned long, seastar::io_priority_class const&) at /usr/include/c++/10/bits/basic_string.h:323
seastar::file_data_source_impl::issue_read_aheads(unsigned int) at fstream.cc:?
 (inlined by) seastar::file_data_source_impl::issue_read_aheads(unsigned int)::{lambda()#1}::operator()() const at /jenkins/workspace/scylla-4.2/build/scylla/seastar/src/core/fstream.cc:273
 (inlined by) seastar::future<seastar::temporary_buffer<char> > seastar::futurize<seastar::future<seastar::temporary_buffer<char> > >::invoke<seastar::file_data_source_impl::issue_read_aheads(unsigned int)::{lambda()#1}>(seastar::file_data_source_impl::issue_read_aheads(unsigned int)::{lambda()#1}&&) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future.hh:2026
 (inlined by) auto seastar::futurize_invoke<seastar::file_data_source_impl::issue_read_aheads(unsigned int)::{lambda()#1}>(seastar::file_data_source_impl::issue_read_aheads(unsigned int)::{lambda()#1}&&) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future.hh:2110
 (inlined by) seastar::file_data_source_impl::issue_read_aheads(unsigned int) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/src/core/fstream.cc:274
seastar::file_data_source_impl::get() at fstream.cc:?
seastar::data_source::get() at /usr/include/fmt/format.h:2188
 (inlined by) seastar::future<> seastar::input_stream<char>::consume<std::reference_wrapper<sstables::index_consume_entry_context<sstables::index_consumer> > >(std::reference_wrapper<sstables::index_consume_entry_context<sstables::index_consumer> >&&)::{lambda()#1}::operator()() at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/iostream-impl.hh:202
 (inlined by) seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > seastar::futurize<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > >::invoke<seastar::future<> seastar::input_stream<char>::consume<std::reference_wrapper<sstables::index_consume_entry_context<sstables::index_consumer> > >(std::reference_wrapper<sstables::index_consume_entry_context<sstables::index_consumer> >&&)::{lambda()#1}&>(std::reference_wrapper<sstables::index_consume_entry_context<sstables::index_consumer> >&&) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future.hh:2026
seastar::future<> seastar::repeat<seastar::future<> seastar::input_stream<char>::consume<std::reference_wrapper<sstables::index_consume_entry_context<sstables::index_consumer> > >(std::reference_wrapper<sstables::index_consume_entry_context<sstables::index_consumer> >&&)::{lambda()#1}>(seastar::future<> seastar::input_stream<char>::consume<std::reference_wrapper<sstables::index_consume_entry_context<sstables::index_consumer> > >(std::reference_wrapper<sstables::index_consume_entry_context<sstables::index_consumer> >&&)::{lambda()#1}) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future-util.hh:299
 (inlined by) seastar::future<> seastar::input_stream<char>::consume<std::reference_wrapper<sstables::index_consume_entry_context<sstables::index_consumer> > >(std::reference_wrapper<sstables::index_consume_entry_context<sstables::index_consumer> >&&) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/iostream-impl.hh:228
 (inlined by) seastar::future<> seastar::input_stream<char>::consume<sstables::index_consume_entry_context<sstables::index_consumer> >(sstables::index_consume_entry_context<sstables::index_consumer>&) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/iostream-impl.hh:236
 (inlined by) data_consumer::continuous_data_consumer<sstables::index_consume_entry_context<sstables::index_consumer> >::consume_input() at /jenkins/workspace/scylla-4.2/build/scylla/sstables/consumer.hh:411
 (inlined by) _ZZZN8sstables12index_reader15advance_to_pageERNS0_11index_boundEmENKUlmE_clEmENKUlRT_E_clISt10unique_ptrINS0_6readerESt14default_deleteIS9_EEEEDaS5_ at /jenkins/workspace/scylla-4.2/build/scylla/sstables/index_reader.hh:447
 (inlined by) _ZSt13__invoke_implIN7seastar6futureIJN5utils14chunked_vectorIN8sstables11index_entryELm131072EEEEEERZZNS4_12index_reader15advance_to_pageERNS8_11index_boundEmENKUlmE_clEmEUlRT_E_JRSt10unique_ptrINS8_6readerESt14default_deleteISH_EEEESC_St14__invoke_otherOT0_DpOT1_ at /usr/include/c++/10/bits/invoke.h:60
 (inlined by) _ZSt8__invokeIRZZN8sstables12index_reader15advance_to_pageERNS1_11index_boundEmENKUlmE_clEmEUlRT_E_JRSt10unique_ptrINS1_6readerESt14default_deleteISA_EEEENSt15__invoke_resultIS5_JDpT0_EE4typeEOS5_DpOSG_ at /usr/include/c++/10/bits/invoke.h:96
 (inlined by) _ZSt12__apply_implIRZZN8sstables12index_reader15advance_to_pageERNS1_11index_boundEmENKUlmE_clEmEUlRT_E_RSt5tupleIJSt10unique_ptrINS1_6readerESt14default_deleteISB_EEEEJLm0EEEDcOS5_OT0_St16integer_sequenceImJXspT1_EEE at /usr/include/c++/10/tuple:1724
 (inlined by) _ZSt5applyIRZZN8sstables12index_reader15advance_to_pageERNS1_11index_boundEmENKUlmE_clEmEUlRT_E_RSt5tupleIJSt10unique_ptrINS1_6readerESt14default_deleteISB_EEEEEDcOS5_OT0_ at /usr/include/c++/10/tuple:1736
 (inlined by) _ZN7seastar8internal12do_with_implISt10unique_ptrIN8sstables12index_reader6readerESt14default_deleteIS5_EEZZNS4_15advance_to_pageERNS4_11index_boundEmENKUlmE_clEmEUlRT_E_JEEEDaOSC_OT0_DpOT1_ at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/do_with.hh:106
 (inlined by) _ZN7seastar8futurizeINS_6futureIJN5utils14chunked_vectorIN8sstables11index_entryELm131072EEEEEEE6invokeIRPFS7_OSt10unique_ptrINS4_12index_reader6readerESt14default_deleteISC_EEOZZNSB_15advance_to_pageERNSB_11index_boundEmENKUlmE_clEmEUlRT_E_EJSF_SM_EEES7_OSK_DpOT0_ at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future.hh:2026
 (inlined by) _ZN7seastar15futurize_invokeIRPFNS_6futureIJN5utils14chunked_vectorIN8sstables11index_entryELm131072EEEEEEOSt10unique_ptrINS4_12index_reader6readerESt14default_deleteISA_EEOZZNS9_15advance_to_pageERNS9_11index_boundEmENKUlmE_clEmEUlRT_E_EJSD_SK_EEEDaOSI_DpOT0_ at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future.hh:2110
 (inlined by) _ZN7seastar7do_withISt10unique_ptrIN8sstables12index_reader6readerESt14default_deleteIS4_EEZZNS3_15advance_to_pageERNS3_11index_boundEmENKUlmE_clEmEUlRT_E_JEEEDaOSB_OT0_DpOT1_ at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/do_with.hh:141
 (inlined by) sstables::index_reader::advance_to_page(sstables::index_reader::index_bound&, unsigned long)::{lambda(unsigned long)#1}::operator()(unsigned long) const at /jenkins/workspace/scylla-4.2/build/scylla/sstables/index_reader.hh:469
seastar::future<utils::loading_shared_values<unsigned long, utils::chunked_vector<sstables::index_entry, 131072ul>, std::hash<unsigned long>, std::equal_to<unsigned long>, sstables::shared_index_lists::stats_updater, 16ul>::entry_ptr> utils::loading_shared_values<unsigned long, utils::chunked_vector<sstables::index_entry, 131072ul>, std::hash<unsigned long>, std::equal_to<unsigned long>, sstables::shared_index_lists::stats_updater, 16ul>::get_or_load<sstables::index_reader::advance_to_page(sstables::index_reader::index_bound&, unsigned long)::{lambda(unsigned long)#1}&>(unsigned long const&, sstables::index_reader::advance_to_page(sstables::index_reader::index_bound&, unsigned long)::{lambda(unsigned long)#1}&)::{lambda()#1}::operator()() const at /jenkins/workspace/scylla-4.2/build/scylla/./utils/loading_shared_values.hh:238
 (inlined by) seastar::future<utils::chunked_vector<sstables::index_entry, 131072ul> > seastar::futurize<seastar::future<utils::chunked_vector<sstables::index_entry, 131072ul> > >::invoke<seastar::future<utils::loading_shared_values<unsigned long, utils::chunked_vector<sstables::index_entry, 131072ul>, std::hash<unsigned long>, std::equal_to<unsigned long>, sstables::shared_index_lists::stats_updater, 16ul>::entry_ptr> utils::loading_shared_values<unsigned long, utils::chunked_vector<sstables::index_entry, 131072ul>, std::hash<unsigned long>, std::equal_to<unsigned long>, sstables::shared_index_lists::stats_updater, 16ul>::get_or_load<sstables::index_reader::advance_to_page(sstables::index_reader::index_bound&, unsigned long)::{lambda(unsigned long)#1}&>(unsigned long const&, sstables::index_reader::advance_to_page(sstables::index_reader::index_bound&, unsigned long)::{lambda(unsigned long)#1}&)::{lambda()#1}>(seastar::future<utils::loading_shared_values<unsigned long, utils::chunked_vector<sstables::index_entry, 131072ul>, std::hash<unsigned long>, std::equal_to<unsigned long>, sstables::shared_index_lists::stats_updater, 16ul>::entry_ptr> utils::loading_shared_values<unsigned long, utils::chunked_vector<sstables::index_entry, 131072ul>, std::hash<unsigned long>, std::equal_to<unsigned long>, sstables::shared_index_lists::stats_updater, 16ul>::get_or_load<sstables::index_reader::advance_to_page(sstables::index_reader::index_bound&, unsigned long)::{lambda(unsigned long)#1}&>(unsigned long const&, sstables::index_reader::advance_to_page(sstables::index_reader::index_bound&, unsigned long)::{lambda(unsigned long)#1}&)::{lambda()#1}) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future.hh:2026
 (inlined by) auto seastar::futurize_invoke<seastar::future<utils::loading_shared_values<unsigned long, utils::chunked_vector<sstables::index_entry, 131072ul>, std::hash<unsigned long>, std::equal_to<unsigned long>, sstables::shared_index_lists::stats_updater, 16ul>::entry_ptr> utils::loading_shared_values<unsigned long, utils::chunked_vector<sstables::index_entry, 131072ul>, std::hash<unsigned long>, std::equal_to<unsigned long>, sstables::shared_index_lists::stats_updater, 16ul>::get_or_load<sstables::index_reader::advance_to_page(sstables::index_reader::index_bound&, unsigned long)::{lambda(unsigned long)#1}&>(unsigned long const&, sstables::index_reader::advance_to_page(sstables::index_reader::index_bound&, unsigned long)::{lambda(unsigned long)#1}&)::{lambda()#1}>(seastar::future<utils::loading_shared_values<unsigned long, utils::chunked_vector<sstables::index_entry, 131072ul>, std::hash<unsigned long>, std::equal_to<unsigned long>, sstables::shared_index_lists::stats_updater, 16ul>::entry_ptr> utils::loading_shared_values<unsigned long, utils::chunked_vector<sstables::index_entry, 131072ul>, std::hash<unsigned long>, std::equal_to<unsigned long>, sstables::shared_index_lists::stats_updater, 16ul>::get_or_load<sstables::index_reader::advance_to_page(sstables::index_reader::index_bound&, unsigned long)::{lambda(unsigned long)#1}&>(unsigned long const&, sstables::index_reader::advance_to_page(sstables::index_reader::index_bound&, unsigned long)::{lambda(unsigned long)#1}&)::{lambda()#1}) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future.hh:2110
 (inlined by) seastar::future<utils::loading_shared_values<unsigned long, utils::chunked_vector<sstables::index_entry, 131072ul>, std::hash<unsigned long>, std::equal_to<unsigned long>, sstables::shared_index_lists::stats_updater, 16ul>::entry_ptr> utils::loading_shared_values<unsigned long, utils::chunked_vector<sstables::index_entry, 131072ul>, std::hash<unsigned long>, std::equal_to<unsigned long>, sstables::shared_index_lists::stats_updater, 16ul>::get_or_load<sstables::index_reader::advance_to_page(sstables::index_reader::index_bound&, unsigned long)::{lambda(unsigned long)#1}&>(unsigned long const&, sstables::index_reader::advance_to_page(sstables::index_reader::index_bound&, unsigned long)::{lambda(unsigned long)#1}&) at /jenkins/workspace/scylla-4.2/build/scylla/./utils/loading_shared_values.hh:238
seastar::future<utils::loading_shared_values<unsigned long, seastar::future::chunked_vector<sstables::index_entry, 131072ul>, std::hash<unsigned long>, std::equal_to<unsigned long>, sstables::shared_index_lists::stats_updater, 16ul>::entry_ptr> sstables::shared_index_lists::get_or_load<sstables::index_reader::advance_to_page(sstables::index_reader::index_bound&, unsigned long)::{lambda(unsigned long)#1}&>(unsigned long const&, sstables::index_reader::advance_to_page(sstables::index_reader::index_bound&, unsigned long)::{lambda(unsigned long)#1}&) at /jenkins/workspace/scylla-4.2/build/scylla/./sstables/shared_index_lists.hh:74
 (inlined by) sstables::index_reader::advance_to_page(sstables::index_reader::index_bound&, unsigned long) at /jenkins/workspace/scylla-4.2/build/scylla/sstables/index_reader.hh:472
sstables::index_reader::advance_to(sstables::index_reader::index_bound&, dht::ring_position_view) at /jenkins/workspace/scylla-4.2/build/scylla/sstables/index_reader.hh:603
sstables::index_reader::advance_upper_to_end(nonwrapping_interval<dht::ring_position> const&) at /jenkins/workspace/scylla-4.2/build/scylla/sstables/index_reader.hh:511
 (inlined by) sstables::index_reader::advance_to(nonwrapping_interval<dht::ring_position> const&) at /jenkins/workspace/scylla-4.2/build/scylla/sstables/index_reader.hh:696
sstables::sstable_mutation_reader<sstables::data_consume_rows_context_m, sstables::mp_row_consumer_m>::sstable_mutation_reader(seastar::lw_shared_ptr<sstables::sstable>, seastar::lw_shared_ptr<schema const>, reader_permit, nonwrapping_interval<dht::ring_position> const&, query::partition_slice const&, seastar::io_priority_class const&, tracing::trace_state_ptr, seastar::bool_class<streamed_mutation::forwarding_tag>, seastar::bool_class<flat_mutation_reader::partition_range_forwarding_tag>, sstables::read_monitor&)::{lambda()#1}::operator()() at /jenkins/workspace/scylla-4.2/build/scylla/sstables/partition.cc:167
 (inlined by) seastar::future<> std::__invoke_impl<seastar::future<>, sstables::sstable_mutation_reader<sstables::data_consume_rows_context_m, sstables::mp_row_consumer_m>::sstable_mutation_reader(seastar::lw_shared_ptr<sstables::sstable>, seastar::lw_shared_ptr<schema const>, reader_permit, nonwrapping_interval<dht::ring_position> const&, query::partition_slice const&, seastar::io_priority_class const&, tracing::trace_state_ptr, seastar::bool_class<streamed_mutation::forwarding_tag>, seastar::bool_class<flat_mutation_reader::partition_range_forwarding_tag>, sstables::read_monitor&)::{lambda()#1}&>(std::__invoke_other, sstables::sstable_mutation_reader<sstables::data_consume_rows_context_m, sstables::mp_row_consumer_m>::sstable_mutation_reader(seastar::lw_shared_ptr<sstables::sstable>, seastar::lw_shared_ptr<schema const>, reader_permit, nonwrapping_interval<dht::ring_position> const&, query::partition_slice const&, seastar::io_priority_class const&, tracing::trace_state_ptr, seastar::bool_class<streamed_mutation::forwarding_tag>, seastar::bool_class<flat_mutation_reader::partition_range_forwarding_tag>, sstables::read_monitor&)::{lambda()#1}&) at /usr/include/c++/10/bits/invoke.h:60
 (inlined by) std::enable_if<is_invocable_r_v<seastar::future<>, sstables::sstable_mutation_reader<sstables::data_consume_rows_context_m, sstables::mp_row_consumer_m>::sstable_mutation_reader(seastar::lw_shared_ptr<sstables::sstable>, seastar::lw_shared_ptr<schema const>, reader_permit, nonwrapping_interval<dht::ring_position> const&, query::partition_slice const&, seastar::io_priority_class const&, tracing::trace_state_ptr, seastar::bool_class<streamed_mutation::forwarding_tag>, seastar::bool_class<flat_mutation_reader::partition_range_forwarding_tag>, sstables::read_monitor&)::{lambda()#1}&>, std::enable_if>::type std::__invoke_r<seastar::future<>, sstables::sstable_mutation_reader<sstables::data_consume_rows_context_m, sstables::mp_row_consumer_m>::sstable_mutation_reader(seastar::lw_shared_ptr<sstables::sstable>, seastar::lw_shared_ptr<schema const>, reader_permit, nonwrapping_interval<dht::ring_position> const&, query::partition_slice const&, seastar::io_priority_class const&, tracing::trace_state_ptr, seastar::bool_class<streamed_mutation::forwarding_tag>, seastar::bool_class<flat_mutation_reader::partition_range_forwarding_tag>, sstables::read_monitor&)::{lambda()#1}&>(seastar::future<>&&, (sstables::sstable_mutation_reader<sstables::data_consume_rows_context_m, sstables::mp_row_consumer_m>::sstable_mutation_reader(seastar::lw_shared_ptr<sstables::sstable>, seastar::lw_shared_ptr<schema const>, reader_permit, nonwrapping_interval<dht::ring_position> const&, query::partition_slice const&, seastar::io_priority_class const&, tracing::trace_state_ptr, seastar::bool_class<streamed_mutation::forwarding_tag>, seastar::bool_class<flat_mutation_reader::partition_range_forwarding_tag>, sstables::read_monitor&)::{lambda()#1}&)...) at /usr/include/c++/10/bits/invoke.h:115
 (inlined by) std::_Function_handler<seastar::future<> (), sstables::sstable_mutation_reader<sstables::data_consume_rows_context_m, sstables::mp_row_consumer_m>::sstable_mutation_reader(seastar::lw_shared_ptr<sstables::sstable>, seastar::lw_shared_ptr<schema const>, reader_permit, nonwrapping_interval<dht::ring_position> const&, query::partition_slice const&, seastar::io_priority_class const&, tracing::trace_state_ptr, seastar::bool_class<streamed_mutation::forwarding_tag>, seastar::bool_class<flat_mutation_reader::partition_range_forwarding_tag>, sstables::read_monitor&)::{lambda()#1}>::_M_invoke(std::_Any_data const&) at /usr/include/c++/10/bits/std_function.h:292
std::function<seastar::future<> ()>::operator()() const at /usr/include/c++/10/bits/std_function.h:622
 (inlined by) sstables::sstable_mutation_reader<sstables::data_consume_rows_context_m, sstables::mp_row_consumer_m>::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at /jenkins/workspace/scylla-4.2/build/scylla/sstables/partition.cc:414
mutation_reader_merger::prepare_one(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, mutation_reader_merger::reader_and_last_fragment_kind, seastar::bool_class<mutation_reader_merger::reader_galloping_tag>) at mutation_reader.cc:?
 (inlined by) flat_mutation_reader::operator()(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at /jenkins/workspace/scylla-4.2/build/scylla/flat_mutation_reader.hh:319
 (inlined by) mutation_reader_merger::prepare_one(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, mutation_reader_merger::reader_and_last_fragment_kind, seastar::bool_class<mutation_reader_merger::reader_galloping_tag>) at /jenkins/workspace/scylla-4.2/build/scylla/mutation_reader.cc:361
mutation_reader_merger::prepare_next(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
 (inlined by) invoke<mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)>, mutation_reader_merger::reader_and_last_fragment_kind&> at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future.hh:2026
 (inlined by) futurize_invoke<mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)>, mutation_reader_merger::reader_and_last_fragment_kind&> at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future.hh:2110
 (inlined by) parallel_for_each<mutation_reader_merger::reader_and_last_fragment_kind*, mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)> > at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future-util.hh:159
 (inlined by) parallel_for_each_impl<utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4>&, mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)> > at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future-util.hh:204
 (inlined by) invoke<seastar::future<> (*&)(utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4>&, mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)>&&), utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4>&, mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)> > at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future.hh:2026
 (inlined by) futurize_invoke<seastar::future<> (*&)(utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4>&, mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)>&&), utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4>&, mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)> > at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future.hh:2110
 (inlined by) parallel_for_each<utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4>&, mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)> > at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future-util.hh:216
 (inlined by) mutation_reader_merger::prepare_next(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at /jenkins/workspace/scylla-4.2/build/scylla/mutation_reader.cc:353
mutation_reader_merger::operator()(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > seastar::futurize<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > >::invoke<combined_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}&>(combined_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}&) at mutation_reader.cc:?
 (inlined by) mutation_fragment_merger<mutation_reader_merger>::operator()(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at /jenkins/workspace/scylla-4.2/build/scylla/mutation_reader.cc:108
 (inlined by) operator() at /jenkins/workspace/scylla-4.2/build/scylla/mutation_reader.cc:578
 (inlined by) invoke<combined_mutation_reader::fill_buffer(seastar::lowres_clock::time_point)::<lambda()>&> at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future.hh:2026
combined_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
 (inlined by) combined_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at /jenkins/workspace/scylla-4.2/build/scylla/mutation_reader.cc:589
_ZZN27restricting_mutation_reader11with_readerIZNS_11fill_bufferENSt6chrono10time_pointIN7seastar12lowres_clockENS1_8durationIlSt5ratioILl1ELl1000EEEEEEEUlR20flat_mutation_readerE_EEDcT_S9_ENUlN13reader_permit14resource_unitsEE_clESF_ at mutation_reader.cc:?
 (inlined by) restricting_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda(flat_mutation_reader&)#1}::operator()(flat_mutation_reader&) const at /jenkins/workspace/scylla-4.2/build/scylla/mutation_reader.cc:725
 (inlined by) _ZZN27restricting_mutation_reader11with_readerIZNS_11fill_bufferENSt6chrono10time_pointIN7seastar12lowres_clockENS1_8durationIlSt5ratioILl1ELl1000EEEEEEEUlR20flat_mutation_readerE_EEDcT_S9_ENUlN13reader_permit14resource_unitsEE_clESF_ at /jenkins/workspace/scylla-4.2/build/scylla/mutation_reader.cc:704
_ZN7seastar12continuationINS_8internal22promise_base_with_typeIJEEEZN27restricting_mutation_reader11with_readerIZNS4_11fill_bufferENSt6chrono10time_pointINS_12lowres_clockENS6_8durationIlSt5ratioILl1ELl1000EEEEEEEUlR20flat_mutation_readerE_EEDcT_SD_EUlN13reader_permit14resource_unitsEE_ZZNS_6futureIJSJ_EE14then_impl_nrvoISK_NSL_IJEEEEET0_OSH_ENKUlvE_clEvEUlRS3_RSK_ONS_12future_stateIJSJ_EEEE_JSJ_EE15run_and_disposeEv at mutation_reader.cc:?
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /usr/include/fmt/format.h:2188
seastar::reactor::run_some_tasks() at /usr/include/fmt/format.h:2188
seastar::reactor::run_some_tasks() at /usr/include/fmt/format.h:2188
 (inlined by) seastar::reactor::run() at /jenkins/workspace/scylla-4.2/build/scylla/seastar/src/core/reactor.cc:2715
seastar::smp::configure(boost::program_options::variables_map, seastar::reactor_config)::{lambda()#3}::operator()() const at /usr/include/fmt/format.h:2188
std::function<void ()>::operator()() const at /usr/include/c++/10/bits/basic_string.h:323
 (inlined by) seastar::posix_thread::start_routine(void*) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/src/core/posix.cc:60
?? ??:0
?? ??:0

2020-09-08T15:52:26+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !ERR     | scylla: [shard 0] storage_proxy - Exception when communicating with 10.0.3.65, to read from cdc_test.test_table_postimage: seastar::named_semaphore_timed_out (Semaphore timed out: _read_concurrency_sem)
2020-09-08T15:52:26+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !ERR     | scylla: [shard 0] storage_proxy - Exception when communicating with 10.0.3.65, to read from cdc_test.test_table_postimage: seastar::named_semaphore_timed_out (Semaphore timed out: _read_concurrency_sem)
2020-09-08T15:52:28+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !WARNING | scylla: [shard 12] seastar_memory - oversized allocation: 3145728 bytes. This is non-fatal, but could lead to latency and/or fragmentation issues. Please report: at    0x333167d#012   0x3331990#012   0x3331e19#012   0x2e289c3#012   0x2e2d4fb#012   0x2eee2a4#012   0x2eedce4#012   0x3148d14#012   0x3149032#012   0x2e75b0d#012   0x2deca04#012   0x2df71a3#012   0xef8811#012   0xef8b08#012   0x26ed300#012   0x2df4eee#012   0x2e1e8c4#012   0x2e1f881#012   0x12acfed#012   0x12bd71a#012   0x12bfffb#012   0x12ef684#012   0x12f0bfb#012   0x1392f1f#012   0x1393b9e#012   0x13b239a#012   0x1167e5c#012   0x116912a#012   0x116bfa5#012   0x116e51d#012   0x116ea9a#012   0x117aa61#012   0x117b5c5#012   0x1167e5c#012   0x116912a#012   0x116bfa5#012   0x116e51d#012   0x116ea9a#012   0x11370b4#012   0x116a523#012   0x116aaf7#012   0x116b17d#012   0xfdb0e3#012   0x25c4791#012   0x25c5160#012   0x25c5d83#012   0x25c7208#012   0x25c92d1#012   0x25cae17#012   0x25d4c58#012   0x25d65a3#012   0x313f25c#012   --------#012   N7seastar12continuationINS_8internal22promise_base_with_typeIJEEEZNS_5asyncIZN16row_level_repair3runEvEUlvE_JEEENS_8futurizeINSt9result_ofIFNSt5decayIT_E4typeEDpNS9_IT0_E4typeEEE4typeEE4typeENS_17thread_attributesEOSA_DpOSD_EUlvE0_ZZNS_6futureIJEE14then_impl_nrvoISQ_SS_EET0_SN_ENKUlvE_clEvEUlRS3_RSQ_ONS_12future_stateIJEEEE_JEEE#012   --------#012   seastar::continuation<seastar::internal::promise_base_with_type<>, seastar::future<>::finally_body<seastar::async<row_level_repair::run()::{lambda()#1}>(seastar::thread_attributes, std::decay&&, (std::decay<row_level_repair::run()::{lambda()#1}>::type&&)...)::{lambda()#3}, false>, seastar::future<>::then_wrapped_nrvo<seastar::future<>, {lambda()#3}>({lambda()#3}&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<>&, {lambda()#3}&, seastar::future_state<>&&)#1}>#012   --------#012   seastar::internal::do_with_state<std::tuple<row_level_repair>, seastar::future<> >#012   --------#012   seastar::continuation<seastar::internal::promise_base_with_type<>, seastar::future<>::handle_exception_type<repair_range(repair_info&, nonwrapping_interval<dht::token> const&)::{lambda(auto:1&, auto:2&)#1}::operator()<std::vector<gms::inet_address, std::allocator<gms> >, std::allocator<gms> >(std::vector<gms::inet_address, std::allocator<gms> >&, std::allocator<gms>&) const::{lambda()#2}::operator()() const::{lambda(utils::UUID)#1}::operator()({lambda()#2}) const::{lambda(no_such_column_family&)#1}>(std::vector<gms::inet_address, std::allocator<gms> >&&)::{lambda(no_such_column_family)#1}, seastar::future<>::then_wrapped_nrvo<seastar::future<>, no_such_column_family&>(std::allocator<gms>&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<>&, no_such_column_family&, seastar::future_state<>&&)#1}>#012   --------#012   seastar::continuation<seastar::internal::promise_base_with_type<>, seastar::future<>::handle_exception<repair_range(repair_info&, nonwrapping_interval<dht::token> const&)::{lambda(auto:1&, auto:2&)#1}::operator()<std::vector<gms::inet_address, std::allocator<gms> >, std::allocator<gms> >(std::vector<gms::inet_address, std::allocator<gms> >&, std::allocator<gms>&) const::{lambda()#2}::operator()() const::{lambda(utils::UUID)#1}::operator()({lambda()#2}) const::{lambda(std::__exception_ptr::exception_ptr)#2}>(std::vector<gms::inet_address, std::allocator<gms> >&&)::{lambda(std::__exception_ptr)#1}, seastar::future<>::then_wrapped_nrvo<seastar::future<>, std::__exception_ptr::exception_ptr>(std::allocator<gms>&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<>&, std::__exception_ptr::exception_ptr&, seastar::future_state<>&&)#1}>#012   --------#012   seastar::internal::do_for_each_state<__gnu_cxx::__normal_iterator<utils::UUID*, std::vector<utils::UUID, std::allocator<utils::UUID> > >, repair_range(repair_info&, nonwrapping_interval<dht::token> const&)::{lambda(auto:1&, auto:2&)#1}::operator()<std::vector<gms::inet_address, std::allocator<gms> >, std::allocator<gms> >(std::vector<gms::inet_address, std::allocator<gms> >&, std::allocator<gms>&) const::{lambda()#2}::operator()() const::{lambda(utils::UUID)#1}>#012   --------#012   seastar::internal::do_with_state<std::tuple<std::vector<gms::inet_address, std::allocator<gms::inet_address> >, std::vector<gms::inet_address, std::allocator<gms::inet_address> > >, seastar::future<> >#012   --------#012   N7seastar12continuationINS_8internal22promise_base_with_typeIJEEENS_6futureIJEE12finally_bodyIZZNS_14with_semaphoreINS_33named_semaphore_exception_factoryEZZL16do_repair_rangesNS_13lw_shared_ptrI11repair_infoEEENKUlOT_E_clIR20nonwrapping_intervalIN3dht5tokenEEEEDaSD_EUlvE_NSt6chrono3_V212steady_clockEEENS_8futurizeINSt9result_ofIFT0_vEE4typeEE4typeERNS_15basic_semaphoreISC_T1_EEmOSR_ENUlSC_E_clINS_15semaphore_unitsIS8_SO_EEEEDaSC_EUlvE_Lb0EEEZZNS5_17then_wrapped_nrvoIS5_S17_EENSP_ISC_E4typeES11_ENKUlvE_clEvEUlRS3_RS17_ONS_12future_stateIJEEEE_JEEE
2020-09-08T15:52:28+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | sshd[9498]: Accepted publickey for centos from 10.0.2.227 port 38060 ssh2: RSA SHA256:g7OmDbWC8qLUZHxjMohQe3LsQHfcOJbG1ZHqodB6smc

[centos@longevity-100gb-4h-4-2-db-node-74ebd0d8-3 ~]$ addr2line -Cpife /usr/lib/debug/opt/scylladb/libexec/scylla-4.2.rc4-0.20200907.bf0c493c28.x86_64.debug  0x333167d#012   0x3331990#012   0x3331e19#012   0x2e289c3#012   0x2e2d4fb#012   0x2eee2a4#012   0x2eedce4#012   0x3148d14#012   0x3149032#012   0x2e75b0d#012   0x2deca04#012   0x2df71a3#012   0xef8811#012   0xef8b08#012   0x26ed300#012   0x2df4eee#012   0x2e1e8c4#012   0x2e1f881#012   0x12acfed#012   0x12bd71a#012   0x12bfffb#012   0x12ef684#012   0x12f0bfb#012   0x1392f1f#012   0x1393b9e#012   0x13b239a#012   0x1167e5c#012   0x116912a#012   0x116bfa5#012   0x116e51d#012   0x116ea9a#012   0x117aa61#012   0x117b5c5#012   0x1167e5c#012   0x116912a#012   0x116bfa5#012   0x116e51d#012   0x116ea9a#012   0x11370b4#012   0x116a523#012   0x116aaf7#012   0x116b17d#012   0xfdb0e3#012   0x25c4791#012   0x25c5160#012   0x25c5d83#012   0x25c7208#012   0x25c92d1#012   0x25cae17#012   0x25d4c58#012   0x25d65a3#012   0x313f25c#012
seastar::current_backtrace_tasklocal() at backtrace.cc:?
 (inlined by) seastar::current_backtrace_tasklocal() at /jenkins/workspace/scylla-4.2/build/scylla/seastar/src/util/backtrace.cc:86
seastar::current_tasktrace() at backtrace.cc:?
seastar::current_backtrace() at backtrace.cc:?
seastar::memory::cpu_pages::warn_large_allocation(unsigned long) at /usr/include/c++/10/bits/basic_string.h:323
seastar::memory::cpu_pages::check_large_allocation(unsigned long) at /usr/include/c++/10/bits/basic_string.h:323
 (inlined by) seastar::memory::cpu_pages::allocate_large(unsigned int) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/src/core/memory.cc:677
 (inlined by) seastar::memory::allocate_large(unsigned long) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/src/core/memory.cc:1203
 (inlined by) seastar::memory::allocate(unsigned long) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/src/core/memory.cc:1255
 (inlined by) operator new(unsigned long) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/src/core/memory.cc:1685
__gnu_cxx::new_allocator<seastar::priority_class::request>::allocate(unsigned long, void const*) at /usr/include/c++/10/ext/new_allocator.h:115
 (inlined by) std::allocator<seastar::priority_class::request>::allocate(unsigned long) at /usr/include/c++/10/bits/allocator.h:173
 (inlined by) seastar::circular_buffer<seastar::priority_class::request, std::allocator<seastar::priority_class::request> >::expand(unsigned long) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/circular_buffer.hh:301
seastar::circular_buffer<seastar::priority_class::request, std::allocator<seastar::priority_class::request> >::expand() at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/circular_buffer.hh:295
 (inlined by) seastar::circular_buffer<seastar::priority_class::request, std::allocator<seastar::priority_class::request> >::maybe_expand(unsigned long) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/circular_buffer.hh:331
 (inlined by) seastar::circular_buffer<seastar::priority_class::request, std::allocator<seastar::priority_class::request> >::push_back(seastar::priority_class::request&&) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/circular_buffer.hh:380
 (inlined by) seastar::fair_queue::queue(seastar::lw_shared_ptr<seastar::priority_class>, seastar::fair_queue_ticket, seastar::noncopyable_function<void ()>) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/src/core/fair_queue.cc:152
seastar::io_queue::queue_request(seastar::io_priority_class const&, unsigned long, seastar::internal::io_request)::{lambda()#1}::operator()() at io_queue.cc:?
seastar::io_queue::queue_request(seastar::io_priority_class const&, unsigned long, seastar::internal::io_request) at io_queue.cc:?
 (inlined by) submit_to<seastar::io_queue::queue_request(const seastar::io_priority_class&, size_t, seastar::internal::io_request)::<lambda()> > at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/smp.hh:333
 (inlined by) submit_to<seastar::io_queue::queue_request(const seastar::io_priority_class&, size_t, seastar::internal::io_request)::<lambda()> > at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/smp.hh:360
 (inlined by) seastar::io_queue::queue_request(seastar::io_priority_class const&, unsigned long, seastar::internal::io_request) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/src/core/io_queue.cc:305
seastar::reactor::submit_io_read(seastar::io_queue*, seastar::io_priority_class const&, unsigned long, seastar::internal::io_request) at /usr/include/fmt/format.h:2188
seastar::posix_file_impl::read_dma(unsigned long, void*, unsigned long, seastar::io_priority_class const&) at /usr/include/c++/10/bits/basic_string.h:323
seastar::posix_file_impl::dma_read_bulk(unsigned long, unsigned long, seastar::io_priority_class const&) at /usr/include/c++/10/bits/basic_string.h:323
checked_file_impl::dma_read_bulk(unsigned long, unsigned long, seastar::io_priority_class const&)::{lambda()#1}::operator()() const at /usr/include/fmt/format.h:2188
 (inlined by) auto do_io_check<checked_file_impl::dma_read_bulk(unsigned long, unsigned long, seastar::io_priority_class const&)::{lambda()#1}, , void>(std::function<void (std::__exception_ptr::exception_ptr)> const&, checked_file_impl::dma_read_bulk(unsigned long, unsigned long, seastar::io_priority_class const&)::{lambda()#1}&&) at /jenkins/workspace/scylla-4.2/build/scylla/utils/disk-error-handler.hh:73
checked_file_impl::dma_read_bulk(unsigned long, unsigned long, seastar::io_priority_class const&) at /usr/include/fmt/format.h:2188
tracking_file_impl::dma_read_bulk(unsigned long, unsigned long, seastar::io_priority_class const&) at reader_concurrency_semaphore.cc:?
seastar::file::dma_read_bulk_impl(unsigned long, unsigned long, seastar::io_priority_class const&) at /usr/include/c++/10/bits/basic_string.h:323
seastar::file_data_source_impl::issue_read_aheads(unsigned int) at fstream.cc:?
 (inlined by) seastar::file_data_source_impl::issue_read_aheads(unsigned int)::{lambda()#1}::operator()() const at /jenkins/workspace/scylla-4.2/build/scylla/seastar/src/core/fstream.cc:273
 (inlined by) seastar::future<seastar::temporary_buffer<char> > seastar::futurize<seastar::future<seastar::temporary_buffer<char> > >::invoke<seastar::file_data_source_impl::issue_read_aheads(unsigned int)::{lambda()#1}>(seastar::file_data_source_impl::issue_read_aheads(unsigned int)::{lambda()#1}&&) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future.hh:2026
 (inlined by) auto seastar::futurize_invoke<seastar::file_data_source_impl::issue_read_aheads(unsigned int)::{lambda()#1}>(seastar::file_data_source_impl::issue_read_aheads(unsigned int)::{lambda()#1}&&) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future.hh:2110
 (inlined by) seastar::file_data_source_impl::issue_read_aheads(unsigned int) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/src/core/fstream.cc:274
seastar::file_data_source_impl::get() at fstream.cc:?
seastar::data_source::get() at /usr/include/fmt/format.h:2188
 (inlined by) seastar::future<> seastar::input_stream<char>::consume<std::reference_wrapper<sstables::index_consume_entry_context<sstables::index_consumer> > >(std::reference_wrapper<sstables::index_consume_entry_context<sstables::index_consumer> >&&)::{lambda()#1}::operator()() at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/iostream-impl.hh:202
 (inlined by) seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > seastar::futurize<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > >::invoke<seastar::future<> seastar::input_stream<char>::consume<std::reference_wrapper<sstables::index_consume_entry_context<sstables::index_consumer> > >(std::reference_wrapper<sstables::index_consume_entry_context<sstables::index_consumer> >&&)::{lambda()#1}&>(std::reference_wrapper<sstables::index_consume_entry_context<sstables::index_consumer> >&&) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future.hh:2026
seastar::future<> seastar::repeat<seastar::future<> seastar::input_stream<char>::consume<std::reference_wrapper<sstables::index_consume_entry_context<sstables::index_consumer> > >(std::reference_wrapper<sstables::index_consume_entry_context<sstables::index_consumer> >&&)::{lambda()#1}>(seastar::future<> seastar::input_stream<char>::consume<std::reference_wrapper<sstables::index_consume_entry_context<sstables::index_consumer> > >(std::reference_wrapper<sstables::index_consume_entry_context<sstables::index_consumer> >&&)::{lambda()#1}) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future-util.hh:299
 (inlined by) seastar::future<> seastar::input_stream<char>::consume<std::reference_wrapper<sstables::index_consume_entry_context<sstables::index_consumer> > >(std::reference_wrapper<sstables::index_consume_entry_context<sstables::index_consumer> >&&) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/iostream-impl.hh:228
 (inlined by) seastar::future<> seastar::input_stream<char>::consume<sstables::index_consume_entry_context<sstables::index_consumer> >(sstables::index_consume_entry_context<sstables::index_consumer>&) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/iostream-impl.hh:236
 (inlined by) data_consumer::continuous_data_consumer<sstables::index_consume_entry_context<sstables::index_consumer> >::consume_input() at /jenkins/workspace/scylla-4.2/build/scylla/sstables/consumer.hh:411
 (inlined by) _ZZZN8sstables12index_reader15advance_to_pageERNS0_11index_boundEmENKUlmE_clEmENKUlRT_E_clISt10unique_ptrINS0_6readerESt14default_deleteIS9_EEEEDaS5_ at /jenkins/workspace/scylla-4.2/build/scylla/sstables/index_reader.hh:447
 (inlined by) _ZSt13__invoke_implIN7seastar6futureIJN5utils14chunked_vectorIN8sstables11index_entryELm131072EEEEEERZZNS4_12index_reader15advance_to_pageERNS8_11index_boundEmENKUlmE_clEmEUlRT_E_JRSt10unique_ptrINS8_6readerESt14default_deleteISH_EEEESC_St14__invoke_otherOT0_DpOT1_ at /usr/include/c++/10/bits/invoke.h:60
 (inlined by) _ZSt8__invokeIRZZN8sstables12index_reader15advance_to_pageERNS1_11index_boundEmENKUlmE_clEmEUlRT_E_JRSt10unique_ptrINS1_6readerESt14default_deleteISA_EEEENSt15__invoke_resultIS5_JDpT0_EE4typeEOS5_DpOSG_ at /usr/include/c++/10/bits/invoke.h:96
 (inlined by) _ZSt12__apply_implIRZZN8sstables12index_reader15advance_to_pageERNS1_11index_boundEmENKUlmE_clEmEUlRT_E_RSt5tupleIJSt10unique_ptrINS1_6readerESt14default_deleteISB_EEEEJLm0EEEDcOS5_OT0_St16integer_sequenceImJXspT1_EEE at /usr/include/c++/10/tuple:1724
 (inlined by) _ZSt5applyIRZZN8sstables12index_reader15advance_to_pageERNS1_11index_boundEmENKUlmE_clEmEUlRT_E_RSt5tupleIJSt10unique_ptrINS1_6readerESt14default_deleteISB_EEEEEDcOS5_OT0_ at /usr/include/c++/10/tuple:1736
 (inlined by) _ZN7seastar8internal12do_with_implISt10unique_ptrIN8sstables12index_reader6readerESt14default_deleteIS5_EEZZNS4_15advance_to_pageERNS4_11index_boundEmENKUlmE_clEmEUlRT_E_JEEEDaOSC_OT0_DpOT1_ at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/do_with.hh:106
 (inlined by) _ZN7seastar8futurizeINS_6futureIJN5utils14chunked_vectorIN8sstables11index_entryELm131072EEEEEEE6invokeIRPFS7_OSt10unique_ptrINS4_12index_reader6readerESt14default_deleteISC_EEOZZNSB_15advance_to_pageERNSB_11index_boundEmENKUlmE_clEmEUlRT_E_EJSF_SM_EEES7_OSK_DpOT0_ at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future.hh:2026
 (inlined by) _ZN7seastar15futurize_invokeIRPFNS_6futureIJN5utils14chunked_vectorIN8sstables11index_entryELm131072EEEEEEOSt10unique_ptrINS4_12index_reader6readerESt14default_deleteISA_EEOZZNS9_15advance_to_pageERNS9_11index_boundEmENKUlmE_clEmEUlRT_E_EJSD_SK_EEEDaOSI_DpOT0_ at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future.hh:2110
 (inlined by) _ZN7seastar7do_withISt10unique_ptrIN8sstables12index_reader6readerESt14default_deleteIS4_EEZZNS3_15advance_to_pageERNS3_11index_boundEmENKUlmE_clEmEUlRT_E_JEEEDaOSB_OT0_DpOT1_ at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/do_with.hh:141
 (inlined by) sstables::index_reader::advance_to_page(sstables::index_reader::index_bound&, unsigned long)::{lambda(unsigned long)#1}::operator()(unsigned long) const at /jenkins/workspace/scylla-4.2/build/scylla/sstables/index_reader.hh:469
seastar::future<utils::loading_shared_values<unsigned long, utils::chunked_vector<sstables::index_entry, 131072ul>, std::hash<unsigned long>, std::equal_to<unsigned long>, sstables::shared_index_lists::stats_updater, 16ul>::entry_ptr> utils::loading_shared_values<unsigned long, utils::chunked_vector<sstables::index_entry, 131072ul>, std::hash<unsigned long>, std::equal_to<unsigned long>, sstables::shared_index_lists::stats_updater, 16ul>::get_or_load<sstables::index_reader::advance_to_page(sstables::index_reader::index_bound&, unsigned long)::{lambda(unsigned long)#1}&>(unsigned long const&, sstables::index_reader::advance_to_page(sstables::index_reader::index_bound&, unsigned long)::{lambda(unsigned long)#1}&)::{lambda()#1}::operator()() const at /jenkins/workspace/scylla-4.2/build/scylla/./utils/loading_shared_values.hh:238
 (inlined by) seastar::future<utils::chunked_vector<sstables::index_entry, 131072ul> > seastar::futurize<seastar::future<utils::chunked_vector<sstables::index_entry, 131072ul> > >::invoke<seastar::future<utils::loading_shared_values<unsigned long, utils::chunked_vector<sstables::index_entry, 131072ul>, std::hash<unsigned long>, std::equal_to<unsigned long>, sstables::shared_index_lists::stats_updater, 16ul>::entry_ptr> utils::loading_shared_values<unsigned long, utils::chunked_vector<sstables::index_entry, 131072ul>, std::hash<unsigned long>, std::equal_to<unsigned long>, sstables::shared_index_lists::stats_updater, 16ul>::get_or_load<sstables::index_reader::advance_to_page(sstables::index_reader::index_bound&, unsigned long)::{lambda(unsigned long)#1}&>(unsigned long const&, sstables::index_reader::advance_to_page(sstables::index_reader::index_bound&, unsigned long)::{lambda(unsigned long)#1}&)::{lambda()#1}>(seastar::future<utils::loading_shared_values<unsigned long, utils::chunked_vector<sstables::index_entry, 131072ul>, std::hash<unsigned long>, std::equal_to<unsigned long>, sstables::shared_index_lists::stats_updater, 16ul>::entry_ptr> utils::loading_shared_values<unsigned long, utils::chunked_vector<sstables::index_entry, 131072ul>, std::hash<unsigned long>, std::equal_to<unsigned long>, sstables::shared_index_lists::stats_updater, 16ul>::get_or_load<sstables::index_reader::advance_to_page(sstables::index_reader::index_bound&, unsigned long)::{lambda(unsigned long)#1}&>(unsigned long const&, sstables::index_reader::advance_to_page(sstables::index_reader::index_bound&, unsigned long)::{lambda(unsigned long)#1}&)::{lambda()#1}) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future.hh:2026
 (inlined by) auto seastar::futurize_invoke<seastar::future<utils::loading_shared_values<unsigned long, utils::chunked_vector<sstables::index_entry, 131072ul>, std::hash<unsigned long>, std::equal_to<unsigned long>, sstables::shared_index_lists::stats_updater, 16ul>::entry_ptr> utils::loading_shared_values<unsigned long, utils::chunked_vector<sstables::index_entry, 131072ul>, std::hash<unsigned long>, std::equal_to<unsigned long>, sstables::shared_index_lists::stats_updater, 16ul>::get_or_load<sstables::index_reader::advance_to_page(sstables::index_reader::index_bound&, unsigned long)::{lambda(unsigned long)#1}&>(unsigned long const&, sstables::index_reader::advance_to_page(sstables::index_reader::index_bound&, unsigned long)::{lambda(unsigned long)#1}&)::{lambda()#1}>(seastar::future<utils::loading_shared_values<unsigned long, utils::chunked_vector<sstables::index_entry, 131072ul>, std::hash<unsigned long>, std::equal_to<unsigned long>, sstables::shared_index_lists::stats_updater, 16ul>::entry_ptr> utils::loading_shared_values<unsigned long, utils::chunked_vector<sstables::index_entry, 131072ul>, std::hash<unsigned long>, std::equal_to<unsigned long>, sstables::shared_index_lists::stats_updater, 16ul>::get_or_load<sstables::index_reader::advance_to_page(sstables::index_reader::index_bound&, unsigned long)::{lambda(unsigned long)#1}&>(unsigned long const&, sstables::index_reader::advance_to_page(sstables::index_reader::index_bound&, unsigned long)::{lambda(unsigned long)#1}&)::{lambda()#1}) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future.hh:2110
 (inlined by) seastar::future<utils::loading_shared_values<unsigned long, utils::chunked_vector<sstables::index_entry, 131072ul>, std::hash<unsigned long>, std::equal_to<unsigned long>, sstables::shared_index_lists::stats_updater, 16ul>::entry_ptr> utils::loading_shared_values<unsigned long, utils::chunked_vector<sstables::index_entry, 131072ul>, std::hash<unsigned long>, std::equal_to<unsigned long>, sstables::shared_index_lists::stats_updater, 16ul>::get_or_load<sstables::index_reader::advance_to_page(sstables::index_reader::index_bound&, unsigned long)::{lambda(unsigned long)#1}&>(unsigned long const&, sstables::index_reader::advance_to_page(sstables::index_reader::index_bound&, unsigned long)::{lambda(unsigned long)#1}&) at /jenkins/workspace/scylla-4.2/build/scylla/./utils/loading_shared_values.hh:238
seastar::future<utils::loading_shared_values<unsigned long, seastar::future::chunked_vector<sstables::index_entry, 131072ul>, std::hash<unsigned long>, std::equal_to<unsigned long>, sstables::shared_index_lists::stats_updater, 16ul>::entry_ptr> sstables::shared_index_lists::get_or_load<sstables::index_reader::advance_to_page(sstables::index_reader::index_bound&, unsigned long)::{lambda(unsigned long)#1}&>(unsigned long const&, sstables::index_reader::advance_to_page(sstables::index_reader::index_bound&, unsigned long)::{lambda(unsigned long)#1}&) at /jenkins/workspace/scylla-4.2/build/scylla/./sstables/shared_index_lists.hh:74
 (inlined by) sstables::index_reader::advance_to_page(sstables::index_reader::index_bound&, unsigned long) at /jenkins/workspace/scylla-4.2/build/scylla/sstables/index_reader.hh:472
sstables::index_reader::advance_to(sstables::index_reader::index_bound&, dht::ring_position_view) at /jenkins/workspace/scylla-4.2/build/scylla/sstables/index_reader.hh:603
sstables::index_reader::advance_upper_to_end(nonwrapping_interval<dht::ring_position> const&) at /jenkins/workspace/scylla-4.2/build/scylla/sstables/index_reader.hh:511
 (inlined by) sstables::index_reader::advance_to(nonwrapping_interval<dht::ring_position> const&) at /jenkins/workspace/scylla-4.2/build/scylla/sstables/index_reader.hh:696
sstables::sstable_mutation_reader<sstables::data_consume_rows_context_m, sstables::mp_row_consumer_m>::sstable_mutation_reader(seastar::lw_shared_ptr<sstables::sstable>, seastar::lw_shared_ptr<schema const>, reader_permit, nonwrapping_interval<dht::ring_position> const&, query::partition_slice const&, seastar::io_priority_class const&, tracing::trace_state_ptr, seastar::bool_class<streamed_mutation::forwarding_tag>, seastar::bool_class<flat_mutation_reader::partition_range_forwarding_tag>, sstables::read_monitor&)::{lambda()#1}::operator()() at /jenkins/workspace/scylla-4.2/build/scylla/sstables/partition.cc:167
 (inlined by) seastar::future<> std::__invoke_impl<seastar::future<>, sstables::sstable_mutation_reader<sstables::data_consume_rows_context_m, sstables::mp_row_consumer_m>::sstable_mutation_reader(seastar::lw_shared_ptr<sstables::sstable>, seastar::lw_shared_ptr<schema const>, reader_permit, nonwrapping_interval<dht::ring_position> const&, query::partition_slice const&, seastar::io_priority_class const&, tracing::trace_state_ptr, seastar::bool_class<streamed_mutation::forwarding_tag>, seastar::bool_class<flat_mutation_reader::partition_range_forwarding_tag>, sstables::read_monitor&)::{lambda()#1}&>(std::__invoke_other, sstables::sstable_mutation_reader<sstables::data_consume_rows_context_m, sstables::mp_row_consumer_m>::sstable_mutation_reader(seastar::lw_shared_ptr<sstables::sstable>, seastar::lw_shared_ptr<schema const>, reader_permit, nonwrapping_interval<dht::ring_position> const&, query::partition_slice const&, seastar::io_priority_class const&, tracing::trace_state_ptr, seastar::bool_class<streamed_mutation::forwarding_tag>, seastar::bool_class<flat_mutation_reader::partition_range_forwarding_tag>, sstables::read_monitor&)::{lambda()#1}&) at /usr/include/c++/10/bits/invoke.h:60
 (inlined by) std::enable_if<is_invocable_r_v<seastar::future<>, sstables::sstable_mutation_reader<sstables::data_consume_rows_context_m, sstables::mp_row_consumer_m>::sstable_mutation_reader(seastar::lw_shared_ptr<sstables::sstable>, seastar::lw_shared_ptr<schema const>, reader_permit, nonwrapping_interval<dht::ring_position> const&, query::partition_slice const&, seastar::io_priority_class const&, tracing::trace_state_ptr, seastar::bool_class<streamed_mutation::forwarding_tag>, seastar::bool_class<flat_mutation_reader::partition_range_forwarding_tag>, sstables::read_monitor&)::{lambda()#1}&>, std::enable_if>::type std::__invoke_r<seastar::future<>, sstables::sstable_mutation_reader<sstables::data_consume_rows_context_m, sstables::mp_row_consumer_m>::sstable_mutation_reader(seastar::lw_shared_ptr<sstables::sstable>, seastar::lw_shared_ptr<schema const>, reader_permit, nonwrapping_interval<dht::ring_position> const&, query::partition_slice const&, seastar::io_priority_class const&, tracing::trace_state_ptr, seastar::bool_class<streamed_mutation::forwarding_tag>, seastar::bool_class<flat_mutation_reader::partition_range_forwarding_tag>, sstables::read_monitor&)::{lambda()#1}&>(seastar::future<>&&, (sstables::sstable_mutation_reader<sstables::data_consume_rows_context_m, sstables::mp_row_consumer_m>::sstable_mutation_reader(seastar::lw_shared_ptr<sstables::sstable>, seastar::lw_shared_ptr<schema const>, reader_permit, nonwrapping_interval<dht::ring_position> const&, query::partition_slice const&, seastar::io_priority_class const&, tracing::trace_state_ptr, seastar::bool_class<streamed_mutation::forwarding_tag>, seastar::bool_class<flat_mutation_reader::partition_range_forwarding_tag>, sstables::read_monitor&)::{lambda()#1}&)...) at /usr/include/c++/10/bits/invoke.h:115
 (inlined by) std::_Function_handler<seastar::future<> (), sstables::sstable_mutation_reader<sstables::data_consume_rows_context_m, sstables::mp_row_consumer_m>::sstable_mutation_reader(seastar::lw_shared_ptr<sstables::sstable>, seastar::lw_shared_ptr<schema const>, reader_permit, nonwrapping_interval<dht::ring_position> const&, query::partition_slice const&, seastar::io_priority_class const&, tracing::trace_state_ptr, seastar::bool_class<streamed_mutation::forwarding_tag>, seastar::bool_class<flat_mutation_reader::partition_range_forwarding_tag>, sstables::read_monitor&)::{lambda()#1}>::_M_invoke(std::_Any_data const&) at /usr/include/c++/10/bits/std_function.h:292
std::function<seastar::future<> ()>::operator()() const at /usr/include/c++/10/bits/std_function.h:622
 (inlined by) sstables::sstable_mutation_reader<sstables::data_consume_rows_context_m, sstables::mp_row_consumer_m>::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at /jenkins/workspace/scylla-4.2/build/scylla/sstables/partition.cc:414
mutation_reader_merger::prepare_one(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, mutation_reader_merger::reader_and_last_fragment_kind, seastar::bool_class<mutation_reader_merger::reader_galloping_tag>) at mutation_reader.cc:?
 (inlined by) flat_mutation_reader::operator()(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at /jenkins/workspace/scylla-4.2/build/scylla/flat_mutation_reader.hh:319
 (inlined by) mutation_reader_merger::prepare_one(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, mutation_reader_merger::reader_and_last_fragment_kind, seastar::bool_class<mutation_reader_merger::reader_galloping_tag>) at /jenkins/workspace/scylla-4.2/build/scylla/mutation_reader.cc:361
mutation_reader_merger::prepare_next(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
 (inlined by) invoke<mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)>, mutation_reader_merger::reader_and_last_fragment_kind&> at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future.hh:2026
 (inlined by) futurize_invoke<mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)>, mutation_reader_merger::reader_and_last_fragment_kind&> at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future.hh:2110
 (inlined by) parallel_for_each<mutation_reader_merger::reader_and_last_fragment_kind*, mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)> > at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future-util.hh:159
 (inlined by) parallel_for_each_impl<utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4>&, mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)> > at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future-util.hh:204
 (inlined by) invoke<seastar::future<> (*&)(utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4>&, mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)>&&), utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4>&, mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)> > at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future.hh:2026
 (inlined by) futurize_invoke<seastar::future<> (*&)(utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4>&, mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)>&&), utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4>&, mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)> > at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future.hh:2110
 (inlined by) parallel_for_each<utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4>&, mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)> > at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future-util.hh:216
 (inlined by) mutation_reader_merger::prepare_next(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at /jenkins/workspace/scylla-4.2/build/scylla/mutation_reader.cc:353
mutation_reader_merger::operator()(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > seastar::futurize<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > >::invoke<combined_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}&>(combined_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}&) at mutation_reader.cc:?
 (inlined by) mutation_fragment_merger<mutation_reader_merger>::operator()(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at /jenkins/workspace/scylla-4.2/build/scylla/mutation_reader.cc:108
 (inlined by) operator() at /jenkins/workspace/scylla-4.2/build/scylla/mutation_reader.cc:578
 (inlined by) invoke<combined_mutation_reader::fill_buffer(seastar::lowres_clock::time_point)::<lambda()>&> at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future.hh:2026
combined_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
 (inlined by) combined_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at /jenkins/workspace/scylla-4.2/build/scylla/mutation_reader.cc:589
_ZZN27restricting_mutation_reader11with_readerIZNS_11fill_bufferENSt6chrono10time_pointIN7seastar12lowres_clockENS1_8durationIlSt5ratioILl1ELl1000EEEEEEEUlR20flat_mutation_readerE_EEDcT_S9_ENUlN13reader_permit14resource_unitsEE_clESF_ at mutation_reader.cc:?
 (inlined by) restricting_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda(flat_mutation_reader&)#1}::operator()(flat_mutation_reader&) const at /jenkins/workspace/scylla-4.2/build/scylla/mutation_reader.cc:725
 (inlined by) _ZZN27restricting_mutation_reader11with_readerIZNS_11fill_bufferENSt6chrono10time_pointIN7seastar12lowres_clockENS1_8durationIlSt5ratioILl1ELl1000EEEEEEEUlR20flat_mutation_readerE_EEDcT_S9_ENUlN13reader_permit14resource_unitsEE_clESF_ at /jenkins/workspace/scylla-4.2/build/scylla/mutation_reader.cc:704
restricting_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
 (inlined by) _ZSt8__invokeIZN27restricting_mutation_reader11with_readerIZNS0_11fill_bufferENSt6chrono10time_pointIN7seastar12lowres_clockENS2_8durationIlSt5ratioILl1ELl1000EEEEEEEUlR20flat_mutation_readerE_EEDcT_SA_EUlN13reader_permit14resource_unitsEE_JSG_EENSt15__invoke_resultISE_JDpT0_EE4typeEOSE_DpOSJ_ at /usr/include/c++/10/bits/invoke.h:96
 (inlined by) _ZSt12__apply_implIZN27restricting_mutation_reader11with_readerIZNS0_11fill_bufferENSt6chrono10time_pointIN7seastar12lowres_clockENS2_8durationIlSt5ratioILl1ELl1000EEEEEEEUlR20flat_mutation_readerE_EEDcT_SA_EUlN13reader_permit14resource_unitsEE_St5tupleIJSG_EEJLm0EEEDcOSE_OT0_St16integer_sequenceImJXspT1_EEE at /usr/include/c++/10/tuple:1724
 (inlined by) _ZSt5applyIZN27restricting_mutation_reader11with_readerIZNS0_11fill_bufferENSt6chrono10time_pointIN7seastar12lowres_clockENS2_8durationIlSt5ratioILl1ELl1000EEEEEEEUlR20flat_mutation_readerE_EEDcT_SA_EUlN13reader_permit14resource_unitsEE_St5tupleIJSG_EEEDcOSE_OT0_ at /usr/include/c++/10/tuple:1736
 (inlined by) _ZN7seastar8futurizeINS_6futureIJEEEE5applyIZN27restricting_mutation_reader11with_readerIZNS5_11fill_bufferENSt6chrono10time_pointINS_12lowres_clockENS7_8durationIlSt5ratioILl1ELl1000EEEEEEEUlR20flat_mutation_readerE_EEDcT_SE_EUlN13reader_permit14resource_unitsEE_JSK_EEES2_OSI_OSt5tupleIJDpT0_EE at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future.hh:2009
 (inlined by) _ZN7seastar6futureIJN13reader_permit14resource_unitsEEE9then_implIZN27restricting_mutation_reader11with_readerIZNS5_11fill_bufferENSt6chrono10time_pointINS_12lowres_clockENS7_8durationIlSt5ratioILl1ELl1000EEEEEEEUlR20flat_mutation_readerE_EEDcT_SE_EUlS2_E_NS0_IJEEEEET0_OSI_ at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future.hh:1544
 (inlined by) _ZN7seastar8internal14call_then_implINS_6futureIJN13reader_permit14resource_unitsEEEEE3runIZN27restricting_mutation_reader11with_readerIZNS8_11fill_bufferENSt6chrono10time_pointINS_12lowres_clockENSA_8durationIlSt5ratioILl1ELl1000EEEEEEEUlR20flat_mutation_readerE_EEDcT_SH_EUlS4_E_EEDaRS5_OSL_ at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future.hh:1198
 (inlined by) _ZN7seastar6futureIJN13reader_permit14resource_unitsEEE4thenIZN27restricting_mutation_reader11with_readerIZNS5_11fill_bufferENSt6chrono10time_pointINS_12lowres_clockENS7_8durationIlSt5ratioILl1ELl1000EEEEEEEUlR20flat_mutation_readerE_EEDcT_SE_EUlS2_E_NS0_IJEEEEET0_OSI_ at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future.hh:1468
 (inlined by) _ZN27restricting_mutation_reader11with_readerIZNS_11fill_bufferENSt6chrono10time_pointIN7seastar12lowres_clockENS1_8durationIlSt5ratioILl1ELl1000EEEEEEEUlR20flat_mutation_readerE_EEDcT_S9_ at /jenkins/workspace/scylla-4.2/build/scylla/mutation_reader.cc:705
 (inlined by) restricting_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at /jenkins/workspace/scylla-4.2/build/scylla/mutation_reader.cc:729
mutation_reader_merger::prepare_one(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, mutation_reader_merger::reader_and_last_fragment_kind, seastar::bool_class<mutation_reader_merger::reader_galloping_tag>) at mutation_reader.cc:?
 (inlined by) flat_mutation_reader::operator()(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at /jenkins/workspace/scylla-4.2/build/scylla/flat_mutation_reader.hh:319
 (inlined by) mutation_reader_merger::prepare_one(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, mutation_reader_merger::reader_and_last_fragment_kind, seastar::bool_class<mutation_reader_merger::reader_galloping_tag>) at /jenkins/workspace/scylla-4.2/build/scylla/mutation_reader.cc:361
mutation_reader_merger::prepare_next(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
 (inlined by) invoke<mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)>, mutation_reader_merger::reader_and_last_fragment_kind&> at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future.hh:2026
 (inlined by) futurize_invoke<mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)>, mutation_reader_merger::reader_and_last_fragment_kind&> at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future.hh:2110
 (inlined by) parallel_for_each<mutation_reader_merger::reader_and_last_fragment_kind*, mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)> > at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future-util.hh:159
 (inlined by) parallel_for_each_impl<utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4>&, mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)> > at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future-util.hh:204
 (inlined by) invoke<seastar::future<> (*&)(utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4>&, mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)>&&), utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4>&, mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)> > at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future.hh:2026
 (inlined by) futurize_invoke<seastar::future<> (*&)(utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4>&, mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)>&&), utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4>&, mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)> > at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future.hh:2110
 (inlined by) parallel_for_each<utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4>&, mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)> > at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future-util.hh:216
 (inlined by) mutation_reader_merger::prepare_next(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at /jenkins/workspace/scylla-4.2/build/scylla/mutation_reader.cc:353
mutation_reader_merger::operator()(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > seastar::futurize<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > >::invoke<combined_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}&>(combined_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}&) at mutation_reader.cc:?
 (inlined by) mutation_fragment_merger<mutation_reader_merger>::operator()(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at /jenkins/workspace/scylla-4.2/build/scylla/mutation_reader.cc:108
 (inlined by) operator() at /jenkins/workspace/scylla-4.2/build/scylla/mutation_reader.cc:578
 (inlined by) invoke<combined_mutation_reader::fill_buffer(seastar::lowres_clock::time_point)::<lambda()>&> at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future.hh:2026
combined_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
 (inlined by) combined_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at /jenkins/workspace/scylla-4.2/build/scylla/mutation_reader.cc:589
evictable_reader::do_fill_buffer(flat_mutation_reader&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
 (inlined by) evictable_reader::do_fill_buffer(flat_mutation_reader&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at /jenkins/workspace/scylla-4.2/build/scylla/mutation_reader.cc:1233
evictable_reader::fill_buffer(flat_mutation_reader&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
auto seastar::internal::do_with_impl<flat_mutation_reader, evictable_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda(flat_mutation_reader&)#1}>(flat_mutation_reader&&, evictable_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda(flat_mutation_reader&)#1}&&) at mutation_reader.cc:?
 (inlined by) __invoke_impl<seastar::future<>, evictable_reader::fill_buffer(seastar::lowres_clock::time_point)::<lambda(flat_mutation_reader&)>&, flat_mutation_reader&> at /usr/include/c++/10/bits/invoke.h:60
 (inlined by) __invoke<evictable_reader::fill_buffer(seastar::lowres_clock::time_point)::<lambda(flat_mutation_reader&)>&, flat_mutation_reader&> at /usr/include/c++/10/bits/invoke.h:96
 (inlined by) __apply_impl<evictable_reader::fill_buffer(seastar::lowres_clock::time_point)::<lambda(flat_mutation_reader&)>&, std::tuple<flat_mutation_reader>&, 0> at /usr/include/c++/10/tuple:1724
 (inlined by) apply<evictable_reader::fill_buffer(seastar::lowres_clock::time_point)::<lambda(flat_mutation_reader&)>&, std::tuple<flat_mutation_reader>&> at /usr/include/c++/10/tuple:1736
 (inlined by) do_with_impl<flat_mutation_reader, evictable_reader::fill_buffer(seastar::lowres_clock::time_point)::<lambda(flat_mutation_reader&)> > at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/do_with.hh:106
evictable_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
 (inlined by) futurize_invoke<seastar::future<> (*&)(flat_mutation_reader&&, evictable_reader::fill_buffer(seastar::lowres_clock::time_point)::<lambda(flat_mutation_reader&)>&&), flat_mutation_reader, evictable_reader::fill_buffer(seastar::lowres_clock::time_point)::<lambda(flat_mutation_reader&)> > at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future.hh:2110
 (inlined by) do_with<flat_mutation_reader, evictable_reader::fill_buffer(seastar::lowres_clock::time_point)::<lambda(flat_mutation_reader&)> > at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/do_with.hh:141
 (inlined by) evictable_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at /jenkins/workspace/scylla-4.2/build/scylla/mutation_reader.cc:1342
flat_mutation_reader::impl::operator()(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at row_cache.cc:?
flat_mutation_reader::operator()(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at /jenkins/workspace/scylla-4.2/build/scylla/./flat_mutation_reader.hh:319
 (inlined by) repair_reader::read_mutation_fragment() at /jenkins/workspace/scylla-4.2/build/scylla/repair/row_level.cc:429
 (inlined by) repair_meta::read_rows_from_disk[abi:cxx11](unsigned long)::{lambda(unsigned long&, unsigned long&, std::__cxx11::list<repair_row, std::allocator<repair_row> >&)#1}::operator()(unsigned long&, unsigned long&, std::__cxx11::list<repair_row, std::allocator<repair_row> >&) const::{lambda()#1}::operator()() at /jenkins/workspace/scylla-4.2/build/scylla/repair/row_level.cc:1076
 (inlined by) seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > seastar::futurize<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > >::invoke<repair_meta::read_rows_from_disk(unsigned long)::{lambda(unsigned long&, unsigned long&, std::__cxx11::list<repair_row, std::allocator<repair_row> >&)#1}::operator()(unsigned long&, unsigned long&, std::__cxx11::list<repair_row, std::allocator<repair_row> >&) const::{lambda()#1}&>(repair_meta::read_rows_from_disk(unsigned long)::{lambda(unsigned long&, unsigned long&, std::__cxx11::list<repair_row, std::allocator<repair_row> >&)#1}::operator()(unsigned long&, unsigned long&, std::__cxx11::list<repair_row, std::allocator<repair_row> >&) const::{lambda()#1}&) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future.hh:2026
 (inlined by) seastar::future<> seastar::repeat<repair_meta::read_rows_from_disk(unsigned long)::{lambda(unsigned long&, unsigned long&, std::__cxx11::list<repair_row, std::allocator<repair_row> >&)#1}::operator()(unsigned long&, unsigned long&, std::__cxx11::list<repair_row, std::allocator<repair_row> >&) const::{lambda()#1}>(repair_meta::read_rows_from_disk(unsigned long)::{lambda(unsigned long&, unsigned long&, std::__cxx11::list<repair_row, std::allocator<repair_row> >&)#1}::operator()(unsigned long&, unsigned long&, std::__cxx11::list<repair_row, std::allocator<repair_row> >&) const::{lambda()#1}) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future-util.hh:299
repair_meta::read_rows_from_disk[abi:cxx11](unsigned long)::{lambda(unsigned long&, unsigned long&, std::__cxx11::list<repair_row, std::allocator<repair_row> >&)#1}::operator()(unsigned long&, unsigned long&, std::__cxx11::list<repair_row, std::allocator<repair_row> >&) const at /jenkins/workspace/scylla-4.2/build/scylla/repair/row_level.cc:1083
 (inlined by) seastar::future<std::__cxx11::list<repair_row, std::allocator<repair_row> >, unsigned long> std::__invoke_impl<seastar::future<std::__cxx11::list<repair_row, std::allocator<repair_row> >, unsigned long>, repair_meta::read_rows_from_disk(unsigned long)::{lambda(unsigned long&, unsigned long&, std::__cxx11::list<repair_row, std::allocator<repair_row> >&)#1}&, unsigned long&, unsigned long&, std::__cxx11::list<repair_row, std::allocator<repair_row> >&>(std::__invoke_other, repair_meta::read_rows_from_disk(unsigned long)::{lambda(unsigned long&, unsigned long&, std::__cxx11::list<repair_row, std::allocator<repair_row> >&)#1}&, unsigned long&, unsigned long&, std::__cxx11::list<repair_row, std::allocator<repair_row> >&) at /usr/include/c++/10/bits/invoke.h:60
 (inlined by) std::__invoke_result<repair_meta::read_rows_from_disk(unsigned long)::{lambda(unsigned long&, unsigned long&, std::__cxx11::list<repair_row, std::allocator<repair_row> >&)#1}&, unsigned long&, unsigned long&, std::__cxx11::list<repair_row, std::allocator<repair_row> >&>::type std::__invoke<repair_meta::read_rows_from_disk(unsigned long)::{lambda(unsigned long&, unsigned long&, std::__cxx11::list<repair_row, std::allocator<repair_row> >&)#1}&, unsigned long&, unsigned long&, std::__cxx11::list<repair_row, std::allocator<repair_row> >&>(std::__invoke_result&&, (repair_meta::read_rows_from_disk(unsigned long)::{lambda(unsigned long&, unsigned long&, std::__cxx11::list<repair_row, std::allocator<repair_row> >&)#1}&)...) at /usr/include/c++/10/bits/invoke.h:96
 (inlined by) _ZSt12__apply_implIRZN11repair_meta19read_rows_from_diskB5cxx11EmEUlRmS1_RNSt7__cxx114listI10repair_rowSaIS4_EEEE_RSt5tupleIJmmS6_EEJLm0ELm1ELm2EEEDcOT_OT0_St16integer_sequenceImJXspT1_EEE at /usr/include/c++/10/tuple:1724
 (inlined by) _ZSt5applyIRZN11repair_meta19read_rows_from_diskB5cxx11EmEUlRmS1_RNSt7__cxx114listI10repair_rowSaIS4_EEEE_RSt5tupleIJmmS6_EEEDcOT_OT0_ at /usr/include/c++/10/tuple:1736
 (inlined by) auto seastar::internal::do_with_impl<unsigned long&, unsigned long, std::__cxx11::list<repair_row, std::allocator<repair_row> >, repair_meta::read_rows_from_disk[abi:cxx11](unsigned long)::{lambda(unsigned long&, unsigned long&, std::__cxx11::list<repair_row, std::allocator<repair_row> >&)#1}>(unsigned long&, unsigned long&&, std::__cxx11::list<repair_row, std::allocator<repair_row> >&&, repair_meta::read_rows_from_disk[abi:cxx11](unsigned long)::{lambda(unsigned long&, unsigned long&, std::__cxx11::list<repair_row, std::allocator<repair_row> >&)#1}&&) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/do_with.hh:106
seastar::future<std::__cxx11::list<repair_row, std::allocator<repair_row> >, unsigned long> seastar::futurize<seastar::future<std::__cxx11::list<repair_row, std::allocator<repair_row> >, unsigned long> >::invoke<seastar::future<std::__cxx11::list<repair_row, std::allocator<repair_row> >, unsigned long> (*&)(unsigned long&, unsigned long&&, std::__cxx11::list<repair_row, std::allocator<repair_row> >&&, repair_meta::read_rows_from_disk(unsigned long)::{lambda(unsigned long&, unsigned long&, std::__cxx11::list<repair_row, std::allocator<repair_row> >&)#1}&&), unsigned long&, unsigned long, std::__cxx11::list<repair_row, std::allocator<repair_row> >, {lambda(unsigned long&, unsigned long&, std::__cxx11::list<repair_row, std::allocator<repair_row> >&)#1}>(seastar::future<std::__cxx11::list<repair_row, std::allocator<repair_row> >, unsigned long> (*&)(unsigned long&, unsigned long&&, std::__cxx11::list<repair_row, std::allocator<repair_row> >&&, repair_meta::read_rows_from_disk(unsigned long)::{lambda(unsigned long&, unsigned long&, std::__cxx11::list<repair_row, std::allocator<repair_row> >&)#1}&&), unsigned long&, unsigned long&&, std::__cxx11::list<repair_row, std::allocator<repair_row> >&&, {lambda(unsigned long&, unsigned long&, std::__cxx11::list<repair_row, std::allocator<repair_row> >&)#1}&&) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future.hh:2026
 (inlined by) auto seastar::futurize_invoke<seastar::future<std::__cxx11::list<repair_row, std::allocator<repair_row> >, unsigned long> (*&)(unsigned long&, unsigned long&&, std::__cxx11::list<repair_row, std::allocator<repair_row> >&&, repair_meta::read_rows_from_disk[abi:cxx11](unsigned long)::{lambda(unsigned long&, unsigned long&, std::__cxx11::list<repair_row, std::allocator<repair_row> >&)#1}&&), unsigned long&, unsigned long, std::__cxx11::list<repair_row, std::allocator<repair_row> >, {lambda(unsigned long&, unsigned long&, std::__cxx11::list<repair_row, std::allocator<repair_row> >&)#1}>(seastar::future<std::__cxx11::list<repair_row, std::allocator<repair_row> >, unsigned long> (*&)(unsigned long&, unsigned long&&, std::__cxx11::list<repair_row, std::allocator<repair_row> >&&, repair_meta::read_rows_from_disk[abi:cxx11](unsigned long)::{lambda(unsigned long&, unsigned long&, std::__cxx11::list<repair_row, std::allocator<repair_row> >&)#1}&&), unsigned long&, unsigned long&&, std::__cxx11::list<repair_row, std::allocator<repair_row> >&&, {lambda(unsigned long&, unsigned long&, std::__cxx11::list<repair_row, std::allocator<repair_row> >&)#1}&&) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future.hh:2110
 (inlined by) auto seastar::do_with<unsigned long&, unsigned long, std::__cxx11::list<repair_row, std::allocator<repair_row> >, repair_meta::read_rows_from_disk[abi:cxx11](unsigned long)::{lambda(unsigned long&, unsigned long&, std::__cxx11::list<repair_row, std::allocator<repair_row> >&)#1}>(unsigned long&, unsigned long&&, std::__cxx11::list<repair_row, std::allocator<repair_row> >&&, repair_meta::read_rows_from_disk[abi:cxx11](unsigned long)::{lambda(unsigned long&, unsigned long&, std::__cxx11::list<repair_row, std::allocator<repair_row> >&)#1}&&) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/do_with.hh:141
 (inlined by) repair_meta::read_rows_from_disk[abi:cxx11](unsigned long) at /jenkins/workspace/scylla-4.2/build/scylla/repair/row_level.cc:1091
 (inlined by) repair_meta::get_sync_boundary(std::optional<repair_sync_boundary>)::{lambda(unsigned long)#1}::operator()(unsigned long) const at /jenkins/workspace/scylla-4.2/build/scylla/repair/row_level.cc:1112
seastar::future<get_sync_boundary_response> std::__invoke_impl<seastar::future<get_sync_boundary_response>, repair_meta::get_sync_boundary(std::optional<repair_sync_boundary>)::{lambda(unsigned long)#1}, unsigned long>(std::__invoke_other, repair_meta::get_sync_boundary(std::optional<repair_sync_boundary>)::{lambda(unsigned long)#1}&&, unsigned long&&) at /usr/include/c++/10/bits/invoke.h:60
 (inlined by) std::__invoke_result<repair_meta::get_sync_boundary(std::optional<repair_sync_boundary>)::{lambda(unsigned long)#1}, unsigned long>::type std::__invoke<repair_meta::get_sync_boundary(std::optional<repair_sync_boundary>)::{lambda(unsigned long)#1}, unsigned long>(std::__invoke_result&&, (repair_meta::get_sync_boundary(std::optional<repair_sync_boundary>)::{lambda(unsigned long)#1}&&)...) at /usr/include/c++/10/bits/invoke.h:96
 (inlined by) _ZSt12__apply_implIZN11repair_meta17get_sync_boundaryESt8optionalI20repair_sync_boundaryEEUlmE_St5tupleIJmEEJLm0EEEDcOT_OT0_St16integer_sequenceImJXspT1_EEE at /usr/include/c++/10/tuple:1724
 (inlined by) _ZSt5applyIZN11repair_meta17get_sync_boundaryESt8optionalI20repair_sync_boundaryEEUlmE_St5tupleIJmEEEDcOT_OT0_ at /usr/include/c++/10/tuple:1736
 (inlined by) seastar::future<get_sync_boundary_response> seastar::futurize<seastar::future<get_sync_boundary_response> >::apply<repair_meta::get_sync_boundary(std::optional<repair_sync_boundary>)::{lambda(unsigned long)#1}, unsigned long>(repair_meta::get_sync_boundary(std::optional<repair_sync_boundary>)::{lambda(unsigned long)#1}&&, std::tuple<unsigned long>&&) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future.hh:2009
 (inlined by) seastar::future<get_sync_boundary_response> seastar::future<unsigned long>::then_impl<repair_meta::get_sync_boundary(std::optional<repair_sync_boundary>)::{lambda(unsigned long)#1}, seastar::future<get_sync_boundary_response> >(repair_meta::get_sync_boundary(std::optional<repair_sync_boundary>)::{lambda(unsigned long)#1}&&) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future.hh:1544
 (inlined by) auto seastar::internal::call_then_impl<seastar::future<unsigned long> >::run<repair_meta::get_sync_boundary(std::optional<repair_sync_boundary>)::{lambda(unsigned long)#1}>(seastar::future<unsigned long>&, repair_meta::get_sync_boundary(std::optional<repair_sync_boundary>)::{lambda(unsigned long)#1}&&) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future.hh:1198
 (inlined by) seastar::future<get_sync_boundary_response> seastar::future<unsigned long>::then<repair_meta::get_sync_boundary(std::optional<repair_sync_boundary>)::{lambda(unsigned long)#1}, seastar::future<get_sync_boundary_response> >(repair_meta::get_sync_boundary(std::optional<repair_sync_boundary>)::{lambda(unsigned long)#1}&&) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future.hh:1468
 (inlined by) repair_meta::get_sync_boundary(std::optional<repair_sync_boundary>) at /jenkins/workspace/scylla-4.2/build/scylla/repair/row_level.cc:1127
 (inlined by) repair_meta::get_sync_boundary_handler(std::optional<repair_sync_boundary>)::{lambda()#1}::operator()() at /jenkins/workspace/scylla-4.2/build/scylla/repair/row_level.cc:1582
 (inlined by) seastar::future<get_sync_boundary_response> seastar::futurize<seastar::future<get_sync_boundary_response> >::invoke<repair_meta::get_sync_boundary_handler(std::optional<repair_sync_boundary>)::{lambda()#1}>(repair_meta::get_sync_boundary_handler(std::optional<repair_sync_boundary>)::{lambda()#1}&&) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future.hh:2026
 (inlined by) auto seastar::futurize_invoke<repair_meta::get_sync_boundary_handler(std::optional<repair_sync_boundary>)::{lambda()#1}>(repair_meta::get_sync_boundary_handler(std::optional<repair_sync_boundary>)::{lambda()#1}&&) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future.hh:2110
 (inlined by) auto seastar::internal::invoke_func_with_gate<repair_meta::get_sync_boundary_handler(std::optional<repair_sync_boundary>)::{lambda()#1}>(seastar::gate&, repair_meta::get_sync_boundary_handler(std::optional<repair_sync_boundary>)::{lambda()#1}&&) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/gate.hh:126
 (inlined by) auto seastar::with_gate<repair_meta::get_sync_boundary_handler(std::optional<repair_sync_boundary>)::{lambda()#1}>(seastar::gate&, repair_meta::get_sync_boundary_handler(std::optional<repair_sync_boundary>)::{lambda()#1}&&) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/gate.hh:144
 (inlined by) repair_meta::get_sync_boundary_handler(std::optional<repair_sync_boundary>) at /jenkins/workspace/scylla-4.2/build/scylla/repair/row_level.cc:1583
repair_meta::get_sync_boundary(gms::inet_address, std::optional<repair_sync_boundary>) at /jenkins/workspace/scylla-4.2/build/scylla/repair/row_level.cc:1572
 (inlined by) row_level_repair::negotiate_sync_boundary(repair_meta&)::{lambda(gms::inet_address const&)#1}::operator()(gms::inet_address const&) const at /jenkins/workspace/scylla-4.2/build/scylla/repair/row_level.cc:2324
 (inlined by) seastar::future<> seastar::futurize<seastar::future<> >::invoke<row_level_repair::negotiate_sync_boundary(repair_meta&)::{lambda(gms::inet_address const&)#1}, gms::inet_address&>(row_level_repair::negotiate_sync_boundary(repair_meta&)::{lambda(gms::inet_address const&)#1}&&, gms::inet_address&) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future.hh:2026
 (inlined by) auto seastar::futurize_invoke<row_level_repair::negotiate_sync_boundary(repair_meta&)::{lambda(gms::inet_address const&)#1}, gms::inet_address&>(row_level_repair::negotiate_sync_boundary(repair_meta&)::{lambda(gms::inet_address const&)#1}&&, gms::inet_address&) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future.hh:2110
 (inlined by) seastar::future<> seastar::parallel_for_each<__gnu_cxx::__normal_iterator<gms::inet_address*, std::vector<gms::inet_address, std::allocator<gms::inet_address> > >, row_level_repair::negotiate_sync_boundary(repair_meta&)::{lambda(gms::inet_address const&)#1}>(__gnu_cxx::__normal_iterator<gms::inet_address*, std::vector<gms::inet_address, std::allocator<gms::inet_address> > >, seastar::future<>, row_level_repair::negotiate_sync_boundary(repair_meta&)::{lambda(gms::inet_address const&)#1}&&) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future-util.hh:159
seastar::future<> seastar::internal::parallel_for_each_impl<std::vector<gms::inet_address, std::allocator<gms::inet_address> >&, row_level_repair::negotiate_sync_boundary(repair_meta&)::{lambda(gms::inet_address const&)#1}>(std::vector<gms::inet_address, std::allocator<gms::inet_address> >&, row_level_repair::negotiate_sync_boundary(repair_meta&)::{lambda(gms::inet_address const&)#1}&&) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future-util.hh:204
 (inlined by) seastar::future<> seastar::futurize<seastar::future<> >::invoke<seastar::future<> (*&)(std::vector<gms::inet_address, std::allocator<gms::inet_address> >&, row_level_repair::negotiate_sync_boundary(repair_meta&)::{lambda(gms::inet_address const&)#1}&&), std::vector<gms::inet_address, std::allocator<gms::inet_address> >&, {lambda(gms::inet_address const&)#1}>(seastar::future<> (*&)(std::vector<gms::inet_address, std::allocator<gms::inet_address> >&, row_level_repair::negotiate_sync_boundary(repair_meta&)::{lambda(gms::inet_address const&)#1}&&), std::vector<gms::inet_address, std::allocator<gms::inet_address> >&, {lambda(gms::inet_address const&)#1}&&) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future.hh:2026
 (inlined by) auto seastar::futurize_invoke<seastar::future<> (*&)(std::vector<gms::inet_address, std::allocator<gms::inet_address> >&, row_level_repair::negotiate_sync_boundary(repair_meta&)::{lambda(gms::inet_address const&)#1}&&), std::vector<gms::inet_address, std::allocator<gms::inet_address> >&, {lambda(gms::inet_address const&)#1}>(seastar::future<> (*&)(std::vector<gms::inet_address, std::allocator<gms::inet_address> >&, row_level_repair::negotiate_sync_boundary(repair_meta&)::{lambda(gms::inet_address const&)#1}&&), std::vector<gms::inet_address, std::allocator<gms::inet_address> >&, {lambda(gms::inet_address const&)#1}&&) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future.hh:2110
 (inlined by) seastar::future<> seastar::parallel_for_each<std::vector<gms::inet_address, std::allocator<gms::inet_address> >&, row_level_repair::negotiate_sync_boundary(repair_meta&)::{lambda(gms::inet_address const&)#1}>(std::vector<gms::inet_address, std::allocator<gms::inet_address> >&, row_level_repair::negotiate_sync_boundary(repair_meta&)::{lambda(gms::inet_address const&)#1}&&) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future-util.hh:216
 (inlined by) row_level_repair::negotiate_sync_boundary(repair_meta&) at /jenkins/workspace/scylla-4.2/build/scylla/repair/row_level.cc:2339
row_level_repair::run()::{lambda()#1}::operator()() const at /jenkins/workspace/scylla-4.2/build/scylla/repair/row_level.cc:2562
void std::__invoke_impl<void, row_level_repair::run()::{lambda()#1}>(std::__invoke_other, row_level_repair::run()::{lambda()#1}&&) at /usr/include/c++/10/bits/invoke.h:60
 (inlined by) std::__invoke_result<row_level_repair::run()::{lambda()#1}>::type std::__invoke<row_level_repair::run()::{lambda()#1}>(std::__invoke_result&&, (row_level_repair::run()::{lambda()#1}&&)...) at /usr/include/c++/10/bits/invoke.h:95
 (inlined by) _ZSt12__apply_implIZN16row_level_repair3runEvEUlvE_St5tupleIJEEJEEDcOT_OT0_St16integer_sequenceImJXspT1_EEE at /usr/include/c++/10/tuple:1723
 (inlined by) _ZSt5applyIZN16row_level_repair3runEvEUlvE_St5tupleIJEEEDcOT_OT0_ at /usr/include/c++/10/tuple:1734
 (inlined by) seastar::future<> seastar::futurize<void>::apply<row_level_repair::run()::{lambda()#1}>(row_level_repair::run()::{lambda()#1}&&, std::tuple<>&&) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future.hh:1976
 (inlined by) seastar::futurize<std::result_of<std::decay<row_level_repair::run()::{lambda()#1}>::type ()>::type>::type seastar::async<row_level_repair::run()::{lambda()#1}>(seastar::thread_attributes, std::decay&&, (std::decay<row_level_repair::run()::{lambda()#1}>::type&&)...)::{lambda()#1}::operator()() const at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/thread.hh:259
 (inlined by) seastar::noncopyable_function<void ()>::direct_vtable_for<seastar::futurize<std::result_of<std::decay<row_level_repair::run()::{lambda()#1}>::type ()>::type>::type seastar::async<row_level_repair::run()::{lambda()#1}>(seastar::thread_attributes, std::decay&&, (std::decay<row_level_repair::run()::{lambda()#1}>::type&&)...)::{lambda()#1}>::call(seastar::noncopyable_function<void ()> const*) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/util/noncopyable_function.hh:101
seastar::noncopyable_function<void ()>::operator()() const at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/util/noncopyable_function.hh:184
 (inlined by) seastar::thread_context::main() at /jenkins/workspace/scylla-4.2/build/scylla/seastar/src/core/thread.cc:297

also was triggere reactor stalls for 650 ms:

2020-09-08T16:29:16+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | scylla: Reactor stalled for 651 ms on shard 10.
2020-09-08T16:29:16+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | scylla: Backtrace:
2020-09-08T16:29:16+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | scylla: 0x0000000002ed6582
2020-09-08T16:29:16+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | scylla: 0x0000000002e7ab00
2020-09-08T16:29:16+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | scylla: 0x0000000002e7afd1
2020-09-08T16:29:16+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | scylla: 0x0000000002e7b2f7
2020-09-08T16:29:16+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | scylla: 0x00007f4c93989a8f
2020-09-08T16:29:16+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | scylla: 0x000000000127c8cf
2020-09-08T16:29:16+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | scylla: 0x000000000146fc32
2020-09-08T16:29:16+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | scylla: 0x0000000001470160
2020-09-08T16:29:16+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | scylla: 0x0000000001471c1c
2020-09-08T16:29:16+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | scylla: 0x0000000002e77ea7
2020-09-08T16:29:16+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | scylla: 0x0000000002e7821e
2020-09-08T16:29:16+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | scylla: 0x0000000002eaf2fd
2020-09-08T16:29:16+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | scylla: 0x0000000002ebecfa
2020-09-08T16:29:16+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | scylla: 0x0000000002e42f7d
2020-09-08T16:29:16+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | scylla: /opt/scylladb/libreloc/libpthread.so.0+0x0000000000009431
2020-09-08T16:29:16+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | scylla: /opt/scylladb/libreloc/libc.so.6+0x0000000000101912

[centos@longevity-100gb-4h-4-2-db-node-74ebd0d8-3 ~]$ addr2line -Cpife /usr/lib/debug/opt/scylladb/libexec/scylla-4.2.rc4-0.20200907.bf0c493c28.x86_64.debug 0x0000000002ed6582 0x0000000002e7ab00 0x0000000002e7afd1 0x0000000002e7b2f7 0x00007f4c93989a8f 0x000000000127c8cf 0x000000000146fc32 0x0000000001470160 0x0000000001471c1c 0x0000000002e77ea7 0x0000000002e7821e 0x0000000002eaf2fd 0x0000000002ebecfa 0x0000000002e42f7d /opt/scylladb/libreloc/libpthread.so.0+0x0000000000009431 /opt/scylladb/libreloc/libc.so.6+0x0000000000101912 
void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) at /usr/include/fmt/format.h:2188
seastar::backtrace_buffer::append_backtrace() at /usr/include/fmt/format.h:2188
 (inlined by) print_with_backtrace at /jenkins/workspace/scylla-4.2/build/scylla/seastar/src/core/reactor.cc:751
seastar::internal::cpu_stall_detector::generate_trace() at /usr/include/fmt/format.h:2188
seastar::internal::cpu_stall_detector::maybe_report() at /usr/include/fmt/format.h:2188
 (inlined by) seastar::internal::cpu_stall_detector::on_signal() at /jenkins/workspace/scylla-4.2/build/scylla/seastar/src/core/reactor.cc:1060
 (inlined by) seastar::reactor::block_notifier(int) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/src/core/reactor.cc:1183
?? ??:0
dht::ring_position_range_vector_sharder::~ring_position_range_vector_sharder() at sstables.cc:?
compaction_manager::trim_to_compact(table*, sstables::compaction_descriptor&) at compaction_manager.cc:?
 (inlined by) operator() at /jenkins/workspace/scylla-4.2/build/scylla/sstables/compaction_manager.cc:164
 (inlined by) operator()<__gnu_cxx::__normal_iterator<seastar::lw_shared_ptr<sstables::sstable>*, std::vector<seastar::lw_shared_ptr<sstables::sstable> > > > at /usr/include/c++/10/bits/predefined_ops.h:316
 (inlined by) __find_if<__gnu_cxx::__normal_iterator<seastar::lw_shared_ptr<sstables::sstable>*, std::vector<seastar::lw_shared_ptr<sstables::sstable> > >, __gnu_cxx::__ops::_Iter_pred<compaction_manager::trim_to_compact(column_family*, sstables::compaction_descriptor&)::<lambda(sstables::shared_sstable&)> > > at /usr/include/c++/10/bits/stl_algobase.h:1932
 (inlined by) __find_if<__gnu_cxx::__normal_iterator<seastar::lw_shared_ptr<sstables::sstable>*, std::vector<seastar::lw_shared_ptr<sstables::sstable> > >, __gnu_cxx::__ops::_Iter_pred<compaction_manager::trim_to_compact(column_family*, sstables::compaction_descriptor&)::<lambda(sstables::shared_sstable&)> > > at /usr/include/c++/10/bits/stl_algobase.h:1977
 (inlined by) __remove_if<__gnu_cxx::__normal_iterator<seastar::lw_shared_ptr<sstables::sstable>*, std::vector<seastar::lw_shared_ptr<sstables::sstable> > >, __gnu_cxx::__ops::_Iter_pred<compaction_manager::trim_to_compact(column_family*, sstables::compaction_descriptor&)::<lambda(sstables::shared_sstable&)> > > at /usr/include/c++/10/bits/stl_algo.h:836
 (inlined by) remove_if<__gnu_cxx::__normal_iterator<seastar::lw_shared_ptr<sstables::sstable>*, std::vector<seastar::lw_shared_ptr<sstables::sstable> > >, compaction_manager::trim_to_compact(column_family*, sstables::compaction_descriptor&)::<lambda(sstables::shared_sstable&)> > at /usr/include/c++/10/bits/stl_algo.h:914
 (inlined by) remove_if<std::vector<seastar::lw_shared_ptr<sstables::sstable> >, compaction_manager::trim_to_compact(column_family*, sstables::compaction_descriptor&)::<lambda(sstables::shared_sstable&)> > at /usr/include/boost/range/algorithm/remove_if.hpp:35
 (inlined by) compaction_manager::trim_to_compact(table*, sstables::compaction_descriptor&) at /jenkins/workspace/scylla-4.2/build/scylla/sstables/compaction_manager.cc:169
compaction_manager::submit(table*)::{lambda()#1}::operator()()::{lambda()#1}::operator()()::{lambda()#1}::operator()() at compaction_manager.cc:?
seastar::lambda_task<auto seastar::with_scheduling_group<compaction_manager::submit(table*)::{lambda()#1}::operator()()::{lambda()#1}::operator()()::{lambda()#1}>(seastar::scheduling_group, compaction_manager::submit(table*)::{lambda()#1}::operator()()::{lambda()#1}::operator()()::{lambda()#1})::{lambda()#1}>::run_and_dispose() at compaction_manager.cc:?
 (inlined by) __invoke<compaction_manager::submit(column_family*)::<lambda()> mutable::<lambda()> mutable::<lambda()>&> at /usr/include/c++/10/bits/invoke.h:96
 (inlined by) __apply_impl<compaction_manager::submit(column_family*)::<lambda()> mutable::<lambda()> mutable::<lambda()>&, std::tuple<> > at /usr/include/c++/10/tuple:1724
 (inlined by) apply<compaction_manager::submit(column_family*)::<lambda()> mutable::<lambda()> mutable::<lambda()>&, std::tuple<> > at /usr/include/c++/10/tuple:1736
 (inlined by) apply<compaction_manager::submit(column_family*)::<lambda()> mutable::<lambda()> mutable::<lambda()>&> at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future.hh:2009
 (inlined by) operator() at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future-util.hh:86
 (inlined by) invoke<seastar::with_scheduling_group<compaction_manager::submit(column_family*)::<lambda()> mutable::<lambda()> mutable::<lambda()>, {}>::<lambda()>&> at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/future.hh:2026
 (inlined by) run_and_dispose at /jenkins/workspace/scylla-4.2/build/scylla/seastar/include/seastar/core/make_task.hh:40
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /usr/include/fmt/format.h:2188
seastar::reactor::run_some_tasks() at /usr/include/fmt/format.h:2188
seastar::reactor::run_some_tasks() at /usr/include/fmt/format.h:2188
 (inlined by) seastar::reactor::run() at /jenkins/workspace/scylla-4.2/build/scylla/seastar/src/core/reactor.cc:2715
seastar::smp::configure(boost::program_options::variables_map, seastar::reactor_config)::{lambda()#3}::operator()() const at /usr/include/fmt/format.h:2188
std::function<void ()>::operator()() const at /usr/include/c++/10/bits/basic_string.h:323
 (inlined by) seastar::posix_thread::start_routine(void*) at /jenkins/workspace/scylla-4.2/build/scylla/seastar/src/core/posix.cc:60
?? ??:0

and row level errors:

2020-09-08T16:32:42+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !WARNING | scylla: [shard 0] gossip - Fail to send EchoMessage to 10.0.1.244: seastar::rpc::closed_error (connection is closed)
2020-09-08T16:32:42+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | scylla: [shard 11] repair - Repair 595 out of 768 ranges, id=2, shard=11, keyspace=cdc_test, table={test_table_scylla_cdc_log, test_table, test_table_postimage_scylla_cdc_log, test_table_preimage, test_table_postimage, test_table_preimage_postimage_scylla_cdc_log, test_table_preimage_postimage, test_table_preimage_scylla_cdc_log}, range=(4803670396404597606, 4803876766903146806]
2020-09-08T16:32:42+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !WARNING | scylla: [shard 11] repair - repair id 2 on shard 11, keyspace=cdc_test, cf=test_table, range=(4717517445675966182, 4738335005828560206], got error in row level repair: seastar::rpc::stream_closed (rpc stream was closed by peer)
2020-09-08T16:32:42+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !WARNING | scylla: [shard 11] repair - repair id 2 on shard 11, keyspace=cdc_test, cf=test_table_scylla_cdc_log, range=(4773113232349414959, 4779170882300103014], got error in row level repair: seastar::rpc::stream_closed (rpc stream was closed by peer)
2020-09-08T16:32:42+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !WARNING | scylla: [shard 11] repair - repair id 2 on shard 11, keyspace=cdc_test, cf=test_table_preimage_scylla_cdc_log, range=(4393088221390039239, 4398403132539001390], got error in row level repair: seastar::rpc::stream_closed (rpc stream was closed by peer)
2020-09-08T16:32:42+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !WARNING | scylla: [shard 11] repair - repair id 2 on shard 11, keyspace=cdc_test, cf=test_table_preimage_postimage, range=(4412947298402880814, 4448203414642076628], got error in row level repair: seastar::rpc::stream_closed (rpc stream was closed by peer)
2020-09-08T16:32:42+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !WARNING | scylla: [shard 11] repair - repair id 2 on shard 11, keyspace=cdc_test, cf=test_table_postimage, range=(4690011385839577211, 4693624813052022549], got error in row level repair: seastar::rpc::closed_error (connection is closed)
2020-09-08T16:32:42+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | scylla: [shard 11] repair - Repair 596 out of 768 ranges, id=2, shard=11, keyspace=cdc_test, table={test_table_scylla_cdc_log, test_table, test_table_postimage_scylla_cdc_log, test_table_preimage, test_table_postimage, test_table_preimage_postimage_scylla_cdc_log, test_table_preimage_postimage, test_table_preimage_scylla_cdc_log}, range=(4803876766903146806, 4808036436625851742]
2020-09-08T16:32:42+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !WARNING | scylla: [shard 11] repair - repair id 2 on shard 11, keyspace=cdc_test, cf=test_table_scylla_cdc_log, range=(4738335005828560206, 4745982132534637953], got error in row level repair: seastar::rpc::stream_closed (rpc stream was closed by peer)
2020-09-08T16:32:42+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !WARNING | scylla: [shard 11] repair - repair id 2 on shard 11, keyspace=cdc_test, cf=test_table_preimage_scylla_cdc_log, range=(4408635723088115664, 4412947298402880814], got error in row level repair: std::runtime_error (get_repair_meta: repair_meta_id 102576 for node 10.0.3.65 does not exist)
2020-09-08T16:32:42+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !WARNING | scylla: [shard 11] repair - repair id 2 on shard 11, keyspace=cdc_test, cf=test_table_preimage_scylla_cdc_log, range=(4398403132539001390, 4408635723088115664], got error in row level repair: std::runtime_error (get_repair_meta: repair_meta_id 102578 for node 10.0.3.65 does not exist)
2020-09-08T16:32:42+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !WARNING | scylla: [shard 11] repair - repair id 2 on shard 11, keyspace=cdc_test, cf=test_table_preimage_scylla_cdc_log, range=(4316461481738982734, 4358344341991429768], got error in row level repair: std::runtime_error (get_repair_meta: repair_meta_id 102518 for node 10.0.3.65 does not exist)
2020-09-08T16:32:42+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !WARNING | scylla: [shard 11] repair - repair id 2 on shard 11, keyspace=cdc_test, cf=test_table_scylla_cdc_log, range=(4803670396404597606, 4803876766903146806], got error in row level repair: std::runtime_error (get_repair_meta: repair_meta_id 102577 for node 10.0.3.65 does not exist)
2020-09-08T16:33:58+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !WARNING | scylla: [shard 4] repair - repair id 2 on shard 4, keyspace=cdc_test, cf=test_table_scylla_cdc_log, range=(4803876766903146806, 4808036436625851742], got error in row level repair: std::runtime_error (get_row_diff: Peer failed to process)
2020-09-08T16:33:58+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !WARNING | scylla: [shard 4] repair - repair id 2 on shard 4, keyspace=cdc_test, cf=test_table_postimage_scylla_cdc_log, range=(4709944579535569981, 4717517445675966182], got error in row level repair: seastar::rpc::closed_error (connection is closed)
2020-09-08T16:33:58+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !WARNING | scylla: [shard 4] repair - repair id 2 on shard 4, keyspace=cdc_test, cf=test_table_postimage_scylla_cdc_log, range=(4693624813052022549, 4699567163703389034], got error in row level repair: seastar::rpc::closed_error (connection is closed)
2020-09-08T16:33:58+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !WARNING | scylla: [shard 4] repair - repair id 2 on shard 4, keyspace=cdc_test, cf=test_table_scylla_cdc_log, range=(4717517445675966182, 4738335005828560206], got error in row level repair: seastar::rpc::closed_error (connection is closed)
2020-09-08T16:33:58+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !WARNING | scylla: [shard 4] repair - repair id 2 on shard 4, keyspace=cdc_test, cf=test_table_scylla_cdc_log, range=(4906824794685605864, 4923013069716399300], got error in row level repair: seastar::rpc::closed_error (connection is closed)
2020-09-08T16:33:58+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !WARNING | scylla: [shard 4] repair - repair id 2 on shard 4, keyspace=cdc_test, cf=test_table_scylla_cdc_log, range=(4779170882300103014, 4782941302533181723], got error in row level repair: seastar::rpc::closed_error (connection is closed)
2020-09-08T16:33:58+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !WARNING | scylla: [shard 4] repair - repair id 2 on shard 4, keyspace=cdc_test, cf=test_table_scylla_cdc_log, range=(4808036436625851742, 4817146485738106632], got error in row level repair: seastar::rpc::closed_error (connection is closed)
2020-09-08T16:33:58+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !WARNING | scylla: [shard 4] repair - repair id 2 on shard 4, keyspace=cdc_test, cf=test_table_scylla_cdc_log, range=(4738335005828560206, 4745982132534637953], got error in row level repair: seastar::rpc::closed_error (connection is closed)
2020-09-08T16:33:58+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | scylla: [shard 4] repair - Repair 599 out of 768 ranges, id=2, shard=4, keyspace=cdc_test, table={test_table_scylla_cdc_log, test_table, test_table_postimage_scylla_cdc_log, test_table_preimage, test_table_postimage, test_table_preimage_postimage_scylla_cdc_log, test_table_preimage_postimage, test_table_preimage_scylla_cdc_log}, range=(4923013069716399300, 4951430655447635422]
2020-09-08T16:33:58+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !WARNING | scylla: [shard 4] seastar - Exceptional future ignored: seastar::rpc::closed_error (connection is closed), backtrace:    0x333167d#012   0x3331990#012   0x3331e19#012   0x2e23967#012   0x207d846#012   0x2e77ea7#012   0x2e7821e#012   0x2eaf2fd#012   0x2ebecfa#012   0x2e42f7d#012   /opt/scylladb/libreloc/libpthread.so.0+0x9431#012   /opt/scylladb/libreloc/libc.so.6+0x101912#012   --------#012   seastar::continuation<seastar::internal::promise_base_with_type<seastar::rpc::sink<repair_stream_cmd>, seastar::rpc::source<repair_hash_with_cmd> >, netw::do_make_sink_source<repair_stream_cmd, repair_hash_with_cmd>(netw::messaging_verb, unsigned int, seastar::shared_ptr<netw::messaging_service::rpc_protocol_client_wrapper>, std::unique_ptr<netw::messaging_service::rpc_protocol_wrapper, std::default_delete<netw::messaging_service::rpc_protocol_wrapper> >&)::{lambda(seastar::rpc::sink<repair_stream_cmd>)#1}::operator()(seastar::rpc::sink<repair_stream_cmd>)::{lambda(seastar::future<seastar::rpc::source<repair_hash_with_cmd> >)#1}::operator()({lambda(seastar::rpc::sink<repair_stream_cmd>)#1})::{lambda()#1}, seastar::future<>::then_impl_nrvo<{lambda(seastar::future<seastar::rpc::source<repair_hash_with_cmd> >)#1}, seastar::future<seastar::rpc::sink<repair_stream_cmd>, seastar::rpc::source<repair_hash_with_cmd> > >({lambda(seastar::future<seastar::rpc::source<repair_hash_with_cmd> >)#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<seastar::rpc::sink<repair_stream_cmd>, seastar::rpc::source<repair_hash_with_cmd> >&, {lambda(seastar::future<seastar::rpc::source<repair_hash_with_cmd> >)#1}&, seastar::future_state<>&&)#1}>
2020-09-08T16:33:58+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !WARNING | scylla: [shard 4] repair - repair id 2 on shard 4, keyspace=cdc_test, cf=test_table, range=(4803670396404597606, 4803876766903146806], got error in row level repair: std::runtime_error (get_repair_meta: repair_meta_id 102172 for node 10.0.3.65 does not exist)
2020-09-08T16:33:58+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !WARNING | scylla: [shard 4] repair - repair id 2 on shard 4, keyspace=cdc_test, cf=test_table_postimage, range=(4621868507791152167, 4634522952713790710], got error in row level repair: std::runtime_error (get_repair_meta: repair_meta_id 102175 for node 10.0.3.65 does not exist)
2020-09-08T17:14:35+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !WARNING | scylla: [shard 4] repair - repair id 2 on shard 4, keyspace=cdc_test, cf=test_table, range=(4951430655447635422, 4980835206617115326], got error in row level repair: std::runtime_error (get_repair_meta: repair_meta_id 103700 for node 10.0.3.65 does not exist)
2020-09-08T17:14:35+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !WARNING | scylla: [shard 4] repair - repair id 2 on shard 4, keyspace=cdc_test, cf=test_table, range=(5311126245992403541, 5342727494702895670], got error in row level repair: std::runtime_error (get_repair_meta: repair_meta_id 103707 for node 10.0.3.65 does not exist)
2020-09-08T17:14:35+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !WARNING | scylla: [shard 4] repair - repair id 2 on shard 4, keyspace=cdc_test, cf=test_table, range=(5062786785311745007, 5078916485120165609], got error in row level repair: std::runtime_error (get_repair_meta: repair_meta_id 103698 for node 10.0.3.65 does not exist)
2020-09-08T17:14:35+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !WARNING | scylla: [shard 4] repair - repair id 2 on shard 4, keyspace=cdc_test, cf=test_table, range=(5160654318799690488, 5165881354230115879], got error in row level repair: std::runtime_error (get_repair_meta: repair_meta_id 103692 for node 10.0.3.65 does not exist)
2020-09-08T17:14:35+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !WARNING | scylla: [shard 4] repair - repair id 2 on shard 4, keyspace=cdc_test, cf=test_table, range=(5280426071292037006, 5293461497827557726], got error in row level repair: std::runtime_error (get_repair_meta: repair_meta_id 103696 for node 10.0.3.65 does not exist)
2020-09-08T17:14:35+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !WARNING | scylla: [shard 4] repair - repair id 2 on shard 4, keyspace=cdc_test, cf=test_table_preimage, range=(5293461497827557726, 5295248718572176917], got error in row level repair: std::runtime_error (get_repair_meta: repair_meta_id 103697 for node 10.0.3.65 does not exist)
2020-09-08T17:14:35+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !WARNING | scylla: [shard 4] repair - repair id 2 on shard 4, keyspace=cdc_test, cf=test_table, range=(4923013069716399300, 4951430655447635422], got error in row level repair: std::runtime_error (get_repair_meta: repair_meta_id 103702 for node 10.0.3.65 does not exist)
2020-09-08T17:14:35+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !WARNING | scylla: [shard 4] repair - repair id 2 on shard 4, keyspace=cdc_test, cf=test_table, range=(5027555586520305352, 5062786785311745007], got error in row level repair: std::runtime_error (get_repair_meta: repair_meta_id 103703 for node 10.0.3.65 does not exist)
2020-09-08T17:14:35+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !WARNING | scylla: [shard 4] repair - repair id 2 on shard 4, keyspace=cdc_test, cf=test_table_preimage_postimage_scylla_cdc_log, range=(4773113232349414959, 4779170882300103014], got error in row level repair: std::runtime_error (get_repair_meta: repair_meta_id 103704 for node 10.0.3.65 does not exist)
2020-09-08T17:22:32+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !WARNING | scylla: [shard 4] repair - repair id 2 on shard 4, keyspace=cdc_test, cf=test_table_preimage_postimage_scylla_cdc_log, range=(4699567163703389034, 4703360424751285851], got error in row level repair: std::runtime_error (get_full_row_hashes: Peer failed to process)
2020-09-08T17:22:32+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !WARNING | scylla: [shard 4] repair - repair id 2 on shard 4, keyspace=cdc_test, cf=test_table_preimage_scylla_cdc_log, range=(4599038303771353929, 4601711493007912312], got error in row level repair: std::runtime_error (get_full_row_hashes: Peer failed to process)

Node log system.log.tar.gz

All logs available: https://cloudius-jenkins-test.s3.amazonaws.com/f4c109de-8085-4b84-99cc-f10a36501ad5/20200908_175718/db-cluster-f4c109de.zip

Also was detected a latency alerts during this process:

2020-09-08 15:23:15.702: (PrometheusAlertManagerEvent Severity.WARNING): alert_name=HighLatencies type=start start=2020-09-08T15:23:09.591Z end=2020-09-08T15:26:09.591Z description=[10.0.3.99] has average high latency for more than 5 minutes. updated=2020-09-08T15:23:10.079Z state=active fingerprint=00a88bcde57fbd84 labels={'alertname': 'HighLatencies', 'instance': '[10.0.3.99]', 'monitor': 'scylla-monitor', 'severity': '1'}
2020-09-08 15:23:15.714: (PrometheusAlertManagerEvent Severity.WARNING): alert_name=HighLatencies type=start start=2020-09-08T15:23:09.591Z end=2020-09-08T15:26:09.591Z description=[10.0.3.189] has average high latency for more than 5 minutes. updated=2020-09-08T15:23:10.079Z state=active fingerprint=44b4482533b8c37c labels={'alertname': 'HighLatencies', 'instance': '[10.0.3.189]', 'monitor': 'scylla-monitor', 'severity': '1'}
2020-09-08 15:23:15.727: (PrometheusAlertManagerEvent Severity.WARNING): alert_name=HighLatencies type=start start=2020-09-08T15:23:09.591Z end=2020-09-08T15:26:09.591Z description=[10.0.1.244] has average high latency for more than 5 minutes. updated=2020-09-08T15:23:10.079Z state=active fingerprint=4a50ef3487beb4ac labels={'alertname': 'HighLatencies', 'instance': '[10.0.1.244]', 'monitor': 'scylla-monitor', 'severity': '1'}
2020-09-08 15:23:15.739: (PrometheusAlertManagerEvent Severity.WARNING): alert_name=HighLatencies type=start start=2020-09-08T15:23:09.591Z end=2020-09-08T15:26:09.591Z description=[10.0.0.103] has average high latency for more than 5 minutes. updated=2020-09-08T15:23:10.079Z state=active fingerprint=f36e784bd12d661b labels={'alertname': 'HighLatencies', 'instance': '[10.0.0.103]', 'monitor': 'scylla-monitor', 'severity': '1'}
2020-09-08 15:30:10.572: (PrometheusAlertManagerEvent Severity.WARNING): alert_name=InstanceDown type=start start=2020-09-08T15:30:09.591Z end=2020-09-08T15:33:09.591Z description=[10.0.3.65] has been down for more than 30 seconds. updated=2020-09-08T15:30:09.594Z state=active fingerprint=d6f45de71616cf7c labels={'alertname': 'InstanceDown', 'instance': '[10.0.3.65]', 'job': 'scylla', 'monitor': 'scylla-monitor', 'severity': '2'}
2020-09-08 15:38:17.722: (PrometheusAlertManagerEvent Severity.WARNING): alert_name=HighLatencies type=end start=2020-09-08T15:23:09.591Z end=2020-09-08T15:40:09.591Z description=[10.0.3.99] has average high latency for more than 5 minutes. updated=2020-09-08T15:37:09.850Z state=active fingerprint=00a88bcde57fbd84 labels={'alertname': 'HighLatencies', 'instance': '[10.0.3.99]', 'monitor': 'scylla-monitor', 'severity': '1'}
2020-09-08 15:38:17.734: (PrometheusAlertManagerEvent Severity.WARNING): alert_name=HighLatencies type=end start=2020-09-08T15:23:09.591Z end=2020-09-08T15:40:09.591Z description=[10.0.3.189] has average high latency for more than 5 minutes. updated=2020-09-08T15:37:09.850Z state=active fingerprint=44b4482533b8c37c labels={'alertname': 'HighLatencies', 'instance': '[10.0.3.189]', 'monitor': 'scylla-monitor', 'severity': '1'}
2020-09-08 15:38:17.746: (PrometheusAlertManagerEvent Severity.WARNING): alert_name=HighLatencies type=end start=2020-09-08T15:23:09.591Z end=2020-09-08T15:40:09.591Z description=[10.0.1.244] has average high latency for more than 5 minutes. updated=2020-09-08T15:37:09.850Z state=active fingerprint=4a50ef3487beb4ac labels={'alertname': 'HighLatencies', 'instance': '[10.0.1.244]', 'monitor': 'scylla-monitor', 'severity': '1'}
2020-09-08 15:38:17.759: (PrometheusAlertManagerEvent Severity.WARNING): alert_name=HighLatencies type=end start=2020-09-08T15:22:09.591Z end=2020-09-08T15:40:09.591Z description=[10.0.3.237] has 95% high latency for more than 5 minutes. updated=2020-09-08T15:37:09.839Z state=active fingerprint=60800c4cb4cc604c labels={'alertname': 'HighLatencies', 'instance': '[10.0.3.237]', 'monitor': 'scylla-monitor', 'severity': '1'}
2020-09-08 15:39:11.886: (PrometheusAlertManagerEvent Severity.WARNING): alert_name=HighLatencies type=start start=2020-09-08T15:23:09.591Z end=2020-09-08T15:42:09.591Z description=[10.0.3.99] has 95% high latency for more than 5 minutes. updated=2020-09-08T15:39:09.854Z state=active fingerprint=00a88bcde57fbd84 labels={'alertname': 'HighLatencies', 'instance': '[10.0.3.99]', 'monitor': 'scylla-monitor', 'severity': '1'}
2020-09-08 15:39:11.900: (PrometheusAlertManagerEvent Severity.WARNING): alert_name=HighLatencies type=start start=2020-09-08T15:23:09.591Z end=2020-09-08T15:42:09.591Z description=[10.0.1.244] has 95% high latency for more than 5 minutes. updated=2020-09-08T15:39:09.854Z state=active fingerprint=4a50ef3487beb4ac labels={'alertname': 'HighLatencies', 'instance': '[10.0.1.244]', 'monitor': 'scylla-monitor', 'severity': '1'}
2020-09-08 15:39:11.912: (PrometheusAlertManagerEvent Severity.WARNING): alert_name=HighLatencies type=start start=2020-09-08T15:22:09.591Z end=2020-09-08T15:42:09.591Z description=[10.0.3.237] has 95% high latency for more than 5 minutes. updated=2020-09-08T15:39:09.854Z state=active fingerprint=60800c4cb4cc604c labels={'alertname': 'HighLatencies', 'instance': '[10.0.3.237]', 'monitor': 'scylla-monitor', 'severity': '1'}
2020-09-08 15:39:11.937: (PrometheusAlertManagerEvent Severity.WARNING): alert_name=HighLatencies type=end start=2020-09-08T15:23:09.591Z end=2020-09-08T15:41:09.591Z description=[10.0.0.103] has 95% high latency for more than 5 minutes. updated=2020-09-08T15:38:09.848Z state=active fingerprint=f36e784bd12d661b labels={'alertname': 'HighLatencies', 'instance': '[10.0.0.103]', 'monitor': 'scylla-monitor', 'severity': '1'}
2020-09-08 15:40:15.148: (PrometheusAlertManagerEvent Severity.WARNING): alert_name=HighLatencies type=end start=2020-09-08T15:23:09.591Z end=2020-09-08T15:42:09.591Z description=[10.0.3.99] has 95% high latency for more than 5 minutes. updated=2020-09-08T15:39:09.854Z state=active fingerprint=00a88bcde57fbd84 labels={'alertname': 'HighLatencies', 'instance': '[10.0.3.99]', 'monitor': 'scylla-monitor', 'severity': '1'}
2020-09-08 15:40:15.163: (PrometheusAlertManagerEvent Severity.WARNING): alert_name=HighLatencies type=end start=2020-09-08T15:23:09.591Z end=2020-09-08T15:42:09.591Z description=[10.0.1.244] has 95% high latency for more than 5 minutes. updated=2020-09-08T15:39:09.854Z state=active fingerprint=4a50ef3487beb4ac labels={'alertname': 'HighLatencies', 'instance': '[10.0.1.244]', 'monitor': 'scylla-monitor', 'severity': '1'}

unnamed

avikivity commented 4 years ago

When you report a large allocation, don't leave out the size.

avikivity commented 4 years ago

Looks like this:

/// \cond internal
class priority_class {
    struct request {
        noncopyable_function<void()> func;
        fair_queue_ticket desc;
    };
    friend class fair_queue;
    uint32_t _shares = 0;
    float _accumulated = 0;
    circular_buffer<request> _queue;
    bool _queued = false;
avikivity commented 4 years ago

This doesn't make sense, we would need a huge amount of reads for this to happen.

avikivity commented 4 years ago

What compaction strategy was used? How many sstables per node?

aleksbykov commented 4 years ago

@avikivity On basic table STCS is used, cdc log tables use TWCS Unfortunately nodes were terminated, and it is difficult to cound how many sstables per node. Do we have any metric in scylla monitoring stack for that?

avikivity commented 4 years ago

@aleksbykov the number of open files is an indication (node_exporter metric)

asias commented 4 years ago

Biggest allocation on node6 is 6M. 

2020-09-08T15:52:23+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !WARNING | scylla: [shard 11] seastar_memory - oversized allocation: 3145728 bytes. This is non-fatal, but could lead to latency and/or fragmentation issues. Please report: at    0x333167d#012   0x3331990#012   0x3331e19#012   0x2e289c3#012   0x2e2d4fb#012   0x2eee2a4#012   0x2eedce4#012   0x3148d14#012   0x3149032#012   0x2e75b0d#012   0x2deca04#012   0x2df71a3#012   0xef8811#012   0xef8b08#012   0x26ed300#012   0x2df4eee#012   0x2e1e8c4#012   0x2e1f881#012   0x12acfed#012   0x12bd71a#012   0x12bfffb#012   0x12ef684#012   0x12f0bfb#012   0x1392f1f#012   0x1393b9e#012   0x13b239a#012   0x1167e5c#012   0x116912a#012   0x116bfa5#012   0x116e51d#012   0x116ea9a#012   0x117aa61#012   0x117b00f#012   0x2e77ea7#012   0x2e7821e#012   0x2eaf2fd#012   0x2ebecfa#012   0x2e42f7d#012   /opt/scylladb/libreloc/libpthread.so.0+0x9431#012   /opt/scylladb/libreloc/libc.so.6+0x101912#012   --------#012   seastar::continuation<seastar::internal::promise_base_with_type<>, restricting_mutation_reader::with_reader<restricting_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda(flat_mutation_reader&)#1}>(restricting_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda(flat_mutation_reader&)#1}, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda(reader_permit::resource_units)#1}, seastar::future<reader_permit>::then_impl_nrvo<reader_permit::resource_units, {lambda(reader_permit::resource_units)#1}<> >(restricting_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda(flat_mutation_reader&)#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<>&, reader_permit::resource_units&, seastar::future_state<reader_permit>&&)#1}, reader_permit>#012   --------#012   seastar::continuation<seastar::internal::promise_base_with_type<seastar::optimized_optional<mutation_fragment> >, flat_mutation_reader::impl::operator()(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}, seastar::future<>::then_impl_nrvo<{lambda()#1}, flat_mutation_reader::impl::operator()(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}<seastar::optimized_optional<mutation_fragment> > >({lambda()#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<seastar::optimized_optional<mutation_fragment> >&, {lambda()#1}&, seastar::future_state<>&&)#1}>#012   --------#012   seastar::continuation<seastar::internal::promise_base_with_type<seastar::bool_class<mutation_reader_merger::needs_merge_tag> >, mutation_reader_merger::prepare_one(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, mutation_reader_merger::reader_and_last_fragment_kind, seastar::bool_class<mutation_reader_merger::reader_galloping_tag>)::{lambda(seastar::optimized_optional<mutation_fragment>)#1}, seastar::future<seastar::optimized_optional<mutation_fragment> >::then_impl_nrvo<{lambda(seastar::optimized_optional<mutation_fragment>)#1}, mutation_reader_merger::prepare_one(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, mutation_reader_merger::reader_and_last_fragment_kind, seastar::bool_class<mutation_reader_merger::reader_galloping_tag>)::{lambda(seastar::optimized_optional<mutation_fragment>)#1}<seastar::bool_class<mutation_reader_merger::needs_merge_tag> > >({lambda(seastar::optimized_optional<mutation_fragment>)#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<seastar::bool_class<mutation_reader_merger::needs_merge_tag> >&, {lambda(seastar::optimized_optional<mutation_fragment>)#1}&, seastar::future_state<seastar::optimized_optional<mutation_fragment> >&&)#1}, seastar::optimized_optional<mutation_fragment> >#012   --------#012   N7seastar12continuationINS_8internal22promise_base_with_typeIJEEEZNS_6futureIJNS_10bool_classIN22mutation_reader_merger15needs_merge_tagEEEEE14discard_resultEvEUlDpOT_E_ZZNS9_14then_impl_nrvoISD_NS4_IJEEEEET0_OT_ENKUlvE_clEvEUlRS3_RSD_ONS_12future_stateIJS8_EEEE_JS8_EEE#012   --------#012   seastar::parallel_for_each_state#012   --------#012   seastar::continuation<seastar::internal::promise_base_with_type<>, mutation_reader_merger::prepare_next(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#2}, seastar::future<>::then_impl_nrvo<{lambda()#2}, seastar::future>({lambda()#2}&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<>&, {lambda()#2}&, seastar::future_state<>&&)#1}>#012   --------#012   seastar::continuation<seastar::internal::promise_base_with_type<boost::iterator_range<mutation_fragment*> >, mutation_reader_merger::operator()(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#3}, seastar::future<>::then_impl_nrvo<{lambda()#3}, mutation_reader_merger::operator()(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#3}<boost::iterator_range<mutation_fragment*> > >({lambda()#3}&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<boost::iterator_range<mutation_fragment*> >&, {lambda()#3}&, seastar::future_state<>&&)#1}>#012   --------#012   seastar::continuation<seastar::internal::promise_base_with_type<>, mutation_fragment_merger<mutation_reader_merger>::fetch(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda(boost::iterator_range<mutation_fragment*>)#1}, seastar::future<boost::iterator_range<mutation_fragment*> >::then_impl_nrvo<{lambda(boost::iterator_range<mutation_fragment*>)#1}, mutation_fragment_merger<mutation_reader_merger>::fetch(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda(boost::iterator_range<mutation_fragment*>)#1}<> >({lambda(boost::iterator_range<mutation_fragment*>)#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<>&, {lambda(boost::iterator_range<mutation_fragment*>)#1}&, seastar::future_state<boost::iterator_range<mutation_fragment*> >&&)#1}, boost::iterator_range<mutation_fragment*> >#012   --------#012   seastar::continuation<seastar::internal::promise_base_with_type<seastar::optimized_optional<mutation_fragment> >, mutation_fragment_merger<mutation_reader_merger>::operator()(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}, seastar::future<>::then_impl_nrvo<{lambda()#1}, mutation_fragment_merger<mutation_reader_merger>::operator()(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}<seastar::optimized_optional<mutation_fragment> > >({lambda()#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<seastar::optimized_optional<mutation_fragment> >&, {lambda()#1}&, seastar::future_state<>&&)#1}>#012   --------#012   seastar::continuation<seastar::internal::promise_base_with_type<seastar::bool_class<seastar::stop_iteration_tag> >, combined_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}::operator()() const::{lambda(seastar::optimized_optional<mutation_fragment>)#1}, seastar::future<mutation_fragment>::then_impl_nrvo<seastar::optimized_optional<mutation_fragment>, {lambda(seastar::optimized_optional<mutation_fragment>)#1}<seastar::bool_class<seastar::stop_iteration_tag> > >(seastar::optimized_optional<mutation_fragm
2020-09-08T15:52:28+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !WARNING | scylla: [shard 12] seastar_memory - oversized allocation: 3145728 bytes. This is non-fatal, but could lead to latency and/or fragmentation issues. Please report: at    0x333167d#012   0x3331990#012   0x3331e19#012   0x2e289c3#012   0x2e2d4fb#012   0x2eee2a4#012   0x2eedce4#012   0x3148d14#012   0x3149032#012   0x2e75b0d#012   0x2deca04#012   0x2df71a3#012   0xef8811#012   0xef8b08#012   0x26ed300#012   0x2df4eee#012   0x2e1e8c4#012   0x2e1f881#012   0x12acfed#012   0x12bd71a#012   0x12bfffb#012   0x12ef684#012   0x12f0bfb#012   0x1392f1f#012   0x1393b9e#012   0x13b239a#012   0x1167e5c#012   0x116912a#012   0x116bfa5#012   0x116e51d#012   0x116ea9a#012   0x117aa61#012   0x117b5c5#012   0x1167e5c#012   0x116912a#012   0x116bfa5#012   0x116e51d#012   0x116ea9a#012   0x11370b4#012   0x116a523#012   0x116aaf7#012   0x116b17d#012   0xfdb0e3#012   0x25c4791#012   0x25c5160#012   0x25c5d83#012   0x25c7208#012   0x25c92d1#012   0x25cae17#012   0x25d4c58#012   0x25d65a3#012   0x313f25c#012   --------#012   N7seastar12continuationINS_8internal22promise_base_with_typeIJEEEZNS_5asyncIZN16row_level_repair3runEvEUlvE_JEEENS_8futurizeINSt9result_ofIFNSt5decayIT_E4typeEDpNS9_IT0_E4typeEEE4typeEE4typeENS_17thread_attributesEOSA_DpOSD_EUlvE0_ZZNS_6futureIJEE14then_impl_nrvoISQ_SS_EET0_SN_ENKUlvE_clEvEUlRS3_RSQ_ONS_12future_stateIJEEEE_JEEE#012   --------#012   seastar::continuation<seastar::internal::promise_base_with_type<>, seastar::future<>::finally_body<seastar::async<row_level_repair::run()::{lambda()#1}>(seastar::thread_attributes, std::decay&&, (std::decay<row_level_repair::run()::{lambda()#1}>::type&&)...)::{lambda()#3}, false>, seastar::future<>::then_wrapped_nrvo<seastar::future<>, {lambda()#3}>({lambda()#3}&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<>&, {lambda()#3}&, seastar::future_state<>&&)#1}>#012   --------#012   seastar::internal::do_with_state<std::tuple<row_level_repair>, seastar::future<> >#012   --------#012   seastar::continuation<seastar::internal::promise_base_with_type<>, seastar::future<>::handle_exception_type<repair_range(repair_info&, nonwrapping_interval<dht::token> const&)::{lambda(auto:1&, auto:2&)#1}::operator()<std::vector<gms::inet_address, std::allocator<gms> >, std::allocator<gms> >(std::vector<gms::inet_address, std::allocator<gms> >&, std::allocator<gms>&) const::{lambda()#2}::operator()() const::{lambda(utils::UUID)#1}::operator()({lambda()#2}) const::{lambda(no_such_column_family&)#1}>(std::vector<gms::inet_address, std::allocator<gms> >&&)::{lambda(no_such_column_family)#1}, seastar::future<>::then_wrapped_nrvo<seastar::future<>, no_such_column_family&>(std::allocator<gms>&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<>&, no_such_column_family&, seastar::future_state<>&&)#1}>#012   --------#012   seastar::continuation<seastar::internal::promise_base_with_type<>, seastar::future<>::handle_exception<repair_range(repair_info&, nonwrapping_interval<dht::token> const&)::{lambda(auto:1&, auto:2&)#1}::operator()<std::vector<gms::inet_address, std::allocator<gms> >, std::allocator<gms> >(std::vector<gms::inet_address, std::allocator<gms> >&, std::allocator<gms>&) const::{lambda()#2}::operator()() const::{lambda(utils::UUID)#1}::operator()({lambda()#2}) const::{lambda(std::__exception_ptr::exception_ptr)#2}>(std::vector<gms::inet_address, std::allocator<gms> >&&)::{lambda(std::__exception_ptr)#1}, seastar::future<>::then_wrapped_nrvo<seastar::future<>, std::__exception_ptr::exception_ptr>(std::allocator<gms>&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<>&, std::__exception_ptr::exception_ptr&, seastar::future_state<>&&)#1}>#012   --------#012   seastar::internal::do_for_each_state<__gnu_cxx::__normal_iterator<utils::UUID*, std::vector<utils::UUID, std::allocator<utils::UUID> > >, repair_range(repair_info&, nonwrapping_interval<dht::token> const&)::{lambda(auto:1&, auto:2&)#1}::operator()<std::vector<gms::inet_address, std::allocator<gms> >, std::allocator<gms> >(std::vector<gms::inet_address, std::allocator<gms> >&, std::allocator<gms>&) const::{lambda()#2}::operator()() const::{lambda(utils::UUID)#1}>#012   --------#012   seastar::internal::do_with_state<std::tuple<std::vector<gms::inet_address, std::allocator<gms::inet_address> >, std::vector<gms::inet_address, std::allocator<gms::inet_address> > >, seastar::future<> >#012   --------#012   N7seastar12continuationINS_8internal22promise_base_with_typeIJEEENS_6futureIJEE12finally_bodyIZZNS_14with_semaphoreINS_33named_semaphore_exception_factoryEZZL16do_repair_rangesNS_13lw_shared_ptrI11repair_infoEEENKUlOT_E_clIR20nonwrapping_intervalIN3dht5tokenEEEEDaSD_EUlvE_NSt6chrono3_V212steady_clockEEENS_8futurizeINSt9result_ofIFT0_vEE4typeEE4typeERNS_15basic_semaphoreISC_T1_EEmOSR_ENUlSC_E_clINS_15semaphore_unitsIS8_SO_EEEEDaSC_EUlvE_Lb0EEEZZNS5_17then_wrapped_nrvoIS5_S17_EENSP_ISC_E4typeES11_ENKUlvE_clEvEUlRS3_RS17_ONS_12future_stateIJEEEE_JEEE
2020-09-08T16:00:40+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !WARNING | scylla: [shard 4] seastar_memory - oversized allocation: 3145728 bytes. This is non-fatal, but could lead to latency and/or fragmentation issues. Please report: at    0x333167d#012   0x3331990#012   0x3331e19#012   0x2e289c3#012   0x2e2d4fb#012   0x2eee2a4#012   0x2eedce4#012   0x3148d14#012   0x3149032#012   0x2e75b0d#012   0x2deca04#012   0x2df71a3#012   0xef8811#012   0xef8b08#012   0x26ed300#012   0x2df4eee#012   0x2e1e8c4#012   0x2e1f881#012   0x12acfed#012   0x12bd71a#012   0x12bfffb#012   0x12ef684#012   0x12f0bfb#012   0x1392f1f#012   0x1393b9e#012   0x13b239a#012   0x1167e5c#012   0x116912a#012   0x116bfa5#012   0x116e51d#012   0x116ea9a#012   0x117aa61#012   0x117b5c5#012   0x1167e5c#012   0x116912a#012   0x116bfa5#012   0x116e51d#012   0x116ea9a#012   0x11370b4#012   0x116a523#012   0x116aaf7#012   0x116b17d#012   0xfdb0e3#012   0x25c4791#012   0x25c5160#012   0x25c5d83#012   0x25c7208#012   0x25c92d1#012   0x25cae17#012   0x25d4c58#012   0x25d65a3#012   0x313f25c#012   --------#012   N7seastar12continuationINS_8internal22promise_base_with_typeIJEEEZNS_5asyncIZN16row_level_repair3runEvEUlvE_JEEENS_8futurizeINSt9result_ofIFNSt5decayIT_E4typeEDpNS9_IT0_E4typeEEE4typeEE4typeENS_17thread_attributesEOSA_DpOSD_EUlvE0_ZZNS_6futureIJEE14then_impl_nrvoISQ_SS_EET0_SN_ENKUlvE_clEvEUlRS3_RSQ_ONS_12future_stateIJEEEE_JEEE#012   --------#012   seastar::continuation<seastar::internal::promise_base_with_type<>, seastar::future<>::finally_body<seastar::async<row_level_repair::run()::{lambda()#1}>(seastar::thread_attributes, std::decay&&, (std::decay<row_level_repair::run()::{lambda()#1}>::type&&)...)::{lambda()#3}, false>, seastar::future<>::then_wrapped_nrvo<seastar::future<>, {lambda()#3}>({lambda()#3}&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<>&, {lambda()#3}&, seastar::future_state<>&&)#1}>#012   --------#012   seastar::internal::do_with_state<std::tuple<row_level_repair>, seastar::future<> >#012   --------#012   seastar::continuation<seastar::internal::promise_base_with_type<>, seastar::future<>::handle_exception_type<repair_range(repair_info&, nonwrapping_interval<dht::token> const&)::{lambda(auto:1&, auto:2&)#1}::operator()<std::vector<gms::inet_address, std::allocator<gms> >, std::allocator<gms> >(std::vector<gms::inet_address, std::allocator<gms> >&, std::allocator<gms>&) const::{lambda()#2}::operator()() const::{lambda(utils::UUID)#1}::operator()({lambda()#2}) const::{lambda(no_such_column_family&)#1}>(std::vector<gms::inet_address, std::allocator<gms> >&&)::{lambda(no_such_column_family)#1}, seastar::future<>::then_wrapped_nrvo<seastar::future<>, no_such_column_family&>(std::allocator<gms>&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<>&, no_such_column_family&, seastar::future_state<>&&)#1}>#012   --------#012   seastar::continuation<seastar::internal::promise_base_with_type<>, seastar::future<>::handle_exception<repair_range(repair_info&, nonwrapping_interval<dht::token> const&)::{lambda(auto:1&, auto:2&)#1}::operator()<std::vector<gms::inet_address, std::allocator<gms> >, std::allocator<gms> >(std::vector<gms::inet_address, std::allocator<gms> >&, std::allocator<gms>&) const::{lambda()#2}::operator()() const::{lambda(utils::UUID)#1}::operator()({lambda()#2}) const::{lambda(std::__exception_ptr::exception_ptr)#2}>(std::vector<gms::inet_address, std::allocator<gms> >&&)::{lambda(std::__exception_ptr)#1}, seastar::future<>::then_wrapped_nrvo<seastar::future<>, std::__exception_ptr::exception_ptr>(std::allocator<gms>&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<>&, std::__exception_ptr::exception_ptr&, seastar::future_state<>&&)#1}>#012   --------#012   seastar::internal::do_for_each_state<__gnu_cxx::__normal_iterator<utils::UUID*, std::vector<utils::UUID, std::allocator<utils::UUID> > >, repair_range(repair_info&, nonwrapping_interval<dht::token> const&)::{lambda(auto:1&, auto:2&)#1}::operator()<std::vector<gms::inet_address, std::allocator<gms> >, std::allocator<gms> >(std::vector<gms::inet_address, std::allocator<gms> >&, std::allocator<gms>&) const::{lambda()#2}::operator()() const::{lambda(utils::UUID)#1}>#012   --------#012   seastar::internal::do_with_state<std::tuple<std::vector<gms::inet_address, std::allocator<gms::inet_address> >, std::vector<gms::inet_address, std::allocator<gms::inet_address> > >, seastar::future<> >#012   --------#012   N7seastar12continuationINS_8internal22promise_base_with_typeIJEEENS_6futureIJEE12finally_bodyIZZNS_14with_semaphoreINS_33named_semaphore_exception_factoryEZZL16do_repair_rangesNS_13lw_shared_ptrI11repair_infoEEENKUlOT_E_clIR20nonwrapping_intervalIN3dht5tokenEEEEDaSD_EUlvE_NSt6chrono3_V212steady_clockEEENS_8futurizeINSt9result_ofIFT0_vEE4typeEE4typeERNS_15basic_semaphoreISC_T1_EEmOSR_ENUlSC_E_clINS_15semaphore_unitsIS8_SO_EEEEDaSC_EUlvE_Lb0EEEZZNS5_17then_wrapped_nrvoIS5_S17_EENSP_ISC_E4typeES11_ENKUlvE_clEvEUlRS3_RS17_ONS_12future_stateIJEEEE_JEEE
2020-09-08T16:00:57+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !WARNING | scylla: [shard 8] seastar_memory - oversized allocation: 3145728 bytes. This is non-fatal, but could lead to latency and/or fragmentation issues. Please report: at    0x333167d#012   0x3331990#012   0x3331e19#012   0x2e289c3#012   0x2e2d4fb#012   0x2eee2a4#012   0x2eedce4#012   0x3148d14#012   0x3149032#012   0x2e75b0d#012   0x2deca04#012   0x2df71a3#012   0xef8811#012   0xef8b08#012   0x26ed300#012   0x2df4eee#012   0x2e1e8c4#012   0x2e1f881#012   0x12acfed#012   0x12bd71a#012   0x12bfffb#012   0x12ef684#012   0x12f0bfb#012   0x1392f1f#012   0x1393b9e#012   0x13b239a#012   0x1167e5c#012   0x116912a#012   0x116bfa5#012   0x116e51d#012   0x116ea9a#012   0x117aa61#012   0x117b00f#012   0x2e77ea7#012   0x2e7821e#012   0x2eaf2fd#012   0x2ebecfa#012   0x2e42f7d#012   /opt/scylladb/libreloc/libpthread.so.0+0x9431#012   /opt/scylladb/libreloc/libc.so.6+0x101912#012   --------#012   seastar::continuation<seastar::internal::promise_base_with_type<>, restricting_mutation_reader::with_reader<restricting_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda(flat_mutation_reader&)#1}>(restricting_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda(flat_mutation_reader&)#1}, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda(reader_permit::resource_units)#1}, seastar::future<reader_permit>::then_impl_nrvo<reader_permit::resource_units, {lambda(reader_permit::resource_units)#1}<> >(restricting_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda(flat_mutation_reader&)#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<>&, reader_permit::resource_units&, seastar::future_state<reader_permit>&&)#1}, reader_permit>#012   --------#012   seastar::continuation<seastar::internal::promise_base_with_type<seastar::optimized_optional<mutation_fragment> >, flat_mutation_reader::impl::operator()(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}, seastar::future<>::then_impl_nrvo<{lambda()#1}, flat_mutation_reader::impl::operator()(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}<seastar::optimized_optional<mutation_fragment> > >({lambda()#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<seastar::optimized_optional<mutation_fragment> >&, {lambda()#1}&, seastar::future_state<>&&)#1}>#012   --------#012   seastar::continuation<seastar::internal::promise_base_with_type<seastar::bool_class<mutation_reader_merger::needs_merge_tag> >, mutation_reader_merger::prepare_one(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, mutation_reader_merger::reader_and_last_fragment_kind, seastar::bool_class<mutation_reader_merger::reader_galloping_tag>)::{lambda(seastar::optimized_optional<mutation_fragment>)#1}, seastar::future<seastar::optimized_optional<mutation_fragment> >::then_impl_nrvo<{lambda(seastar::optimized_optional<mutation_fragment>)#1}, mutation_reader_merger::prepare_one(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, mutation_reader_merger::reader_and_last_fragment_kind, seastar::bool_class<mutation_reader_merger::reader_galloping_tag>)::{lambda(seastar::optimized_optional<mutation_fragment>)#1}<seastar::bool_class<mutation_reader_merger::needs_merge_tag> > >({lambda(seastar::optimized_optional<mutation_fragment>)#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<seastar::bool_class<mutation_reader_merger::needs_merge_tag> >&, {lambda(seastar::optimized_optional<mutation_fragment>)#1}&, seastar::future_state<seastar::optimized_optional<mutation_fragment> >&&)#1}, seastar::optimized_optional<mutation_fragment> >#012   --------#012   N7seastar12continuationINS_8internal22promise_base_with_typeIJEEEZNS_6futureIJNS_10bool_classIN22mutation_reader_merger15needs_merge_tagEEEEE14discard_resultEvEUlDpOT_E_ZZNS9_14then_impl_nrvoISD_NS4_IJEEEEET0_OT_ENKUlvE_clEvEUlRS3_RSD_ONS_12future_stateIJS8_EEEE_JS8_EEE#012   --------#012   seastar::parallel_for_each_state#012   --------#012   seastar::continuation<seastar::internal::promise_base_with_type<>, mutation_reader_merger::prepare_next(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#2}, seastar::future<>::then_impl_nrvo<{lambda()#2}, seastar::future>({lambda()#2}&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<>&, {lambda()#2}&, seastar::future_state<>&&)#1}>#012   --------#012   seastar::continuation<seastar::internal::promise_base_with_type<boost::iterator_range<mutation_fragment*> >, mutation_reader_merger::operator()(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#3}, seastar::future<>::then_impl_nrvo<{lambda()#3}, mutation_reader_merger::operator()(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#3}<boost::iterator_range<mutation_fragment*> > >({lambda()#3}&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<boost::iterator_range<mutation_fragment*> >&, {lambda()#3}&, seastar::future_state<>&&)#1}>#012   --------#012   seastar::continuation<seastar::internal::promise_base_with_type<>, mutation_fragment_merger<mutation_reader_merger>::fetch(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda(boost::iterator_range<mutation_fragment*>)#1}, seastar::future<boost::iterator_range<mutation_fragment*> >::then_impl_nrvo<{lambda(boost::iterator_range<mutation_fragment*>)#1}, mutation_fragment_merger<mutation_reader_merger>::fetch(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda(boost::iterator_range<mutation_fragment*>)#1}<> >({lambda(boost::iterator_range<mutation_fragment*>)#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<>&, {lambda(boost::iterator_range<mutation_fragment*>)#1}&, seastar::future_state<boost::iterator_range<mutation_fragment*> >&&)#1}, boost::iterator_range<mutation_fragment*> >#012   --------#012   seastar::continuation<seastar::internal::promise_base_with_type<seastar::optimized_optional<mutation_fragment> >, mutation_fragment_merger<mutation_reader_merger>::operator()(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}, seastar::future<>::then_impl_nrvo<{lambda()#1}, mutation_fragment_merger<mutation_reader_merger>::operator()(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}<seastar::optimized_optional<mutation_fragment> > >({lambda()#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<seastar::optimized_optional<mutation_fragment> >&, {lambda()#1}&, seastar::future_state<>&&)#1}>#012   --------#012   seastar::continuation<seastar::internal::promise_base_with_type<seastar::bool_class<seastar::stop_iteration_tag> >, combined_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}::operator()() const::{lambda(seastar::optimized_optional<mutation_fragment>)#1}, seastar::future<mutation_fragment>::then_impl_nrvo<seastar::optimized_optional<mutation_fragment>, {lambda(seastar::optimized_optional<mutation_fragment>)#1}<seastar::bool_class<seastar::stop_iteration_tag> > >(seastar::optimized_optional<mutation_fragme
2020-09-08T16:02:27+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !WARNING | scylla: [shard 1] seastar_memory - oversized allocation: 3145728 bytes. This is non-fatal, but could lead to latency and/or fragmentation issues. Please report: at    0x333167d#012   0x3331990#012   0x3331e19#012   0x2e289c3#012   0x2e2d4fb#012   0x2eee2a4#012   0x2eedce4#012   0x3148d14#012   0x3149032#012   0x2e75b0d#012   0x2deca04#012   0x2df71a3#012   0xef8811#012   0xef8b08#012   0x26ed300#012   0x2df4eee#012   0x2e1e8c4#012   0x2e1f881#012   0x12acfed#012   0x12bd71a#012   0x12bfffb#012   0x12ef684#012   0x12f0bfb#012   0x1392f1f#012   0x1393b9e#012   0x13b239a#012   0x1167e5c#012   0x116912a#012   0x116bfa5#012   0x116e51d#012   0x116ea9a#012   0x117aa61#012   0x117b5c5#012   0x1167e5c#012   0x116912a#012   0x116bfa5#012   0x116e51d#012   0x116ea9a#012   0x11370b4#012   0x116a523#012   0x116aaf7#012   0x116b17d#012   0xfdb0e3#012   0x25c4791#012   0x25c5160#012   0x25c5d83#012   0x25c7208#012   0x25c92d1#012   0x25cae17#012   0x25d4c58#012   0x25d65a3#012   0x313f25c#012   --------#012   N7seastar12continuationINS_8internal22promise_base_with_typeIJEEEZNS_5asyncIZN16row_level_repair3runEvEUlvE_JEEENS_8futurizeINSt9result_ofIFNSt5decayIT_E4typeEDpNS9_IT0_E4typeEEE4typeEE4typeENS_17thread_attributesEOSA_DpOSD_EUlvE0_ZZNS_6futureIJEE14then_impl_nrvoISQ_SS_EET0_SN_ENKUlvE_clEvEUlRS3_RSQ_ONS_12future_stateIJEEEE_JEEE#012   --------#012   seastar::continuation<seastar::internal::promise_base_with_type<>, seastar::future<>::finally_body<seastar::async<row_level_repair::run()::{lambda()#1}>(seastar::thread_attributes, std::decay&&, (std::decay<row_level_repair::run()::{lambda()#1}>::type&&)...)::{lambda()#3}, false>, seastar::future<>::then_wrapped_nrvo<seastar::future<>, {lambda()#3}>({lambda()#3}&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<>&, {lambda()#3}&, seastar::future_state<>&&)#1}>#012   --------#012   seastar::internal::do_with_state<std::tuple<row_level_repair>, seastar::future<> >#012   --------#012   seastar::continuation<seastar::internal::promise_base_with_type<>, seastar::future<>::handle_exception_type<repair_range(repair_info&, nonwrapping_interval<dht::token> const&)::{lambda(auto:1&, auto:2&)#1}::operator()<std::vector<gms::inet_address, std::allocator<gms> >, std::allocator<gms> >(std::vector<gms::inet_address, std::allocator<gms> >&, std::allocator<gms>&) const::{lambda()#2}::operator()() const::{lambda(utils::UUID)#1}::operator()({lambda()#2}) const::{lambda(no_such_column_family&)#1}>(std::vector<gms::inet_address, std::allocator<gms> >&&)::{lambda(no_such_column_family)#1}, seastar::future<>::then_wrapped_nrvo<seastar::future<>, no_such_column_family&>(std::allocator<gms>&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<>&, no_such_column_family&, seastar::future_state<>&&)#1}>#012   --------#012   seastar::continuation<seastar::internal::promise_base_with_type<>, seastar::future<>::handle_exception<repair_range(repair_info&, nonwrapping_interval<dht::token> const&)::{lambda(auto:1&, auto:2&)#1}::operator()<std::vector<gms::inet_address, std::allocator<gms> >, std::allocator<gms> >(std::vector<gms::inet_address, std::allocator<gms> >&, std::allocator<gms>&) const::{lambda()#2}::operator()() const::{lambda(utils::UUID)#1}::operator()({lambda()#2}) const::{lambda(std::__exception_ptr::exception_ptr)#2}>(std::vector<gms::inet_address, std::allocator<gms> >&&)::{lambda(std::__exception_ptr)#1}, seastar::future<>::then_wrapped_nrvo<seastar::future<>, std::__exception_ptr::exception_ptr>(std::allocator<gms>&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<>&, std::__exception_ptr::exception_ptr&, seastar::future_state<>&&)#1}>#012   --------#012   seastar::internal::do_for_each_state<__gnu_cxx::__normal_iterator<utils::UUID*, std::vector<utils::UUID, std::allocator<utils::UUID> > >, repair_range(repair_info&, nonwrapping_interval<dht::token> const&)::{lambda(auto:1&, auto:2&)#1}::operator()<std::vector<gms::inet_address, std::allocator<gms> >, std::allocator<gms> >(std::vector<gms::inet_address, std::allocator<gms> >&, std::allocator<gms>&) const::{lambda()#2}::operator()() const::{lambda(utils::UUID)#1}>#012   --------#012   seastar::internal::do_with_state<std::tuple<std::vector<gms::inet_address, std::allocator<gms::inet_address> >, std::vector<gms::inet_address, std::allocator<gms::inet_address> > >, seastar::future<> >#012   --------#012   N7seastar12continuationINS_8internal22promise_base_with_typeIJEEENS_6futureIJEE12finally_bodyIZZNS_14with_semaphoreINS_33named_semaphore_exception_factoryEZZL16do_repair_rangesNS_13lw_shared_ptrI11repair_infoEEENKUlOT_E_clIR20nonwrapping_intervalIN3dht5tokenEEEEDaSD_EUlvE_NSt6chrono3_V212steady_clockEEENS_8futurizeINSt9result_ofIFT0_vEE4typeEE4typeERNS_15basic_semaphoreISC_T1_EEmOSR_ENUlSC_E_clINS_15semaphore_unitsIS8_SO_EEEEDaSC_EUlvE_Lb0EEEZZNS5_17then_wrapped_nrvoIS5_S17_EENSP_ISC_E4typeES11_ENKUlvE_clEvEUlRS3_RS17_ONS_12future_stateIJEEEE_JEEE
2020-09-08T16:02:28+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !WARNING | scylla: [shard 1] seastar_memory - oversized allocation: 6291456 bytes. This is non-fatal, but could lead to latency and/or fragmentation issues. Please report: at    0x333167d#012   0x3331990#012   0x3331e19#012   0x2e289c3#012   0x2e2d4fb#012   0x2eee2a4#012   0x2eedce4#012   0x3148d14#012   0x3149032#012   0x2e75b0d#012   0x2deca04#012   0x2df71a3#012   0xef8811#012   0xef8b08#012   0x26ed300#012   0x2df4eee#012   0x2e1e8c4#012   0x2e1f881#012   0x12acfed#012   0x12bd71a#012   0x12bfffb#012   0x12ef684#012   0x12f0bfb#012   0x1392f1f#012   0x1393b9e#012   0x13b239a#012   0x1167e5c#012   0x116912a#012   0x116bfa5#012   0x116e51d#012   0x116ea9a#012   0x117aa61#012   0x117b5c5#012   0x1167e5c#012   0x116912a#012   0x116bfa5#012   0x116e51d#012   0x116ea9a#012   0x11370b4#012   0x116a523#012   0x116aaf7#012   0x116b17d#012   0xfdb0e3#012   0x25c4791#012   0x25c5160#012   0x25c5d83#012   0x25c7208#012   0x25c92d1#012   0x25cae17#012   0x25d4c58#012   0x25d65a3#012   0x313f25c#012   --------#012   N7seastar12continuationINS_8internal22promise_base_with_typeIJEEEZNS_5asyncIZN16row_level_repair3runEvEUlvE_JEEENS_8futurizeINSt9result_ofIFNSt5decayIT_E4typeEDpNS9_IT0_E4typeEEE4typeEE4typeENS_17thread_attributesEOSA_DpOSD_EUlvE0_ZZNS_6futureIJEE14then_impl_nrvoISQ_SS_EET0_SN_ENKUlvE_clEvEUlRS3_RSQ_ONS_12future_stateIJEEEE_JEEE#012   --------#012   seastar::continuation<seastar::internal::promise_base_with_type<>, seastar::future<>::finally_body<seastar::async<row_level_repair::run()::{lambda()#1}>(seastar::thread_attributes, std::decay&&, (std::decay<row_level_repair::run()::{lambda()#1}>::type&&)...)::{lambda()#3}, false>, seastar::future<>::then_wrapped_nrvo<seastar::future<>, {lambda()#3}>({lambda()#3}&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<>&, {lambda()#3}&, seastar::future_state<>&&)#1}>#012   --------#012   seastar::internal::do_with_state<std::tuple<row_level_repair>, seastar::future<> >#012   --------#012   seastar::continuation<seastar::internal::promise_base_with_type<>, seastar::future<>::handle_exception_type<repair_range(repair_info&, nonwrapping_interval<dht::token> const&)::{lambda(auto:1&, auto:2&)#1}::operator()<std::vector<gms::inet_address, std::allocator<gms> >, std::allocator<gms> >(std::vector<gms::inet_address, std::allocator<gms> >&, std::allocator<gms>&) const::{lambda()#2}::operator()() const::{lambda(utils::UUID)#1}::operator()({lambda()#2}) const::{lambda(no_such_column_family&)#1}>(std::vector<gms::inet_address, std::allocator<gms> >&&)::{lambda(no_such_column_family)#1}, seastar::future<>::then_wrapped_nrvo<seastar::future<>, no_such_column_family&>(std::allocator<gms>&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<>&, no_such_column_family&, seastar::future_state<>&&)#1}>#012   --------#012   seastar::continuation<seastar::internal::promise_base_with_type<>, seastar::future<>::handle_exception<repair_range(repair_info&, nonwrapping_interval<dht::token> const&)::{lambda(auto:1&, auto:2&)#1}::operator()<std::vector<gms::inet_address, std::allocator<gms> >, std::allocator<gms> >(std::vector<gms::inet_address, std::allocator<gms> >&, std::allocator<gms>&) const::{lambda()#2}::operator()() const::{lambda(utils::UUID)#1}::operator()({lambda()#2}) const::{lambda(std::__exception_ptr::exception_ptr)#2}>(std::vector<gms::inet_address, std::allocator<gms> >&&)::{lambda(std::__exception_ptr)#1}, seastar::future<>::then_wrapped_nrvo<seastar::future<>, std::__exception_ptr::exception_ptr>(std::allocator<gms>&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<>&, std::__exception_ptr::exception_ptr&, seastar::future_state<>&&)#1}>#012   --------#012   seastar::internal::do_for_each_state<__gnu_cxx::__normal_iterator<utils::UUID*, std::vector<utils::UUID, std::allocator<utils::UUID> > >, repair_range(repair_info&, nonwrapping_interval<dht::token> const&)::{lambda(auto:1&, auto:2&)#1}::operator()<std::vector<gms::inet_address, std::allocator<gms> >, std::allocator<gms> >(std::vector<gms::inet_address, std::allocator<gms> >&, std::allocator<gms>&) const::{lambda()#2}::operator()() const::{lambda(utils::UUID)#1}>#012   --------#012   seastar::internal::do_with_state<std::tuple<std::vector<gms::inet_address, std::allocator<gms::inet_address> >, std::vector<gms::inet_address, std::allocator<gms::inet_address> > >, seastar::future<> >#012   --------#012   N7seastar12continuationINS_8internal22promise_base_with_typeIJEEENS_6futureIJEE12finally_bodyIZZNS_14with_semaphoreINS_33named_semaphore_exception_factoryEZZL16do_repair_rangesNS_13lw_shared_ptrI11repair_infoEEENKUlOT_E_clIR20nonwrapping_intervalIN3dht5tokenEEEEDaSD_EUlvE_NSt6chrono3_V212steady_clockEEENS_8futurizeINSt9result_ofIFT0_vEE4typeEE4typeERNS_15basic_semaphoreISC_T1_EEmOSR_ENUlSC_E_clINS_15semaphore_unitsIS8_SO_EEEEDaSC_EUlvE_Lb0EEEZZNS5_17then_wrapped_nrvoIS5_S17_EENSP_ISC_E4typeES11_ENKUlvE_clEvEUlRS3_RS17_ONS_12future_stateIJEEEE_JEEE
2020-09-08T16:09:31+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !WARNING | scylla: [shard 6] seastar_memory - oversized allocation: 3145728 bytes. This is non-fatal, but could lead to latency and/or fragmentation issues. Please report: at    0x333167d#012   0x3331990#012   0x3331e19#012   0x2e289c3#012   0x2e2d4fb#012   0x2eee2a4#012   0x2eedce4#012   0x3148d14#012   0x3149032#012   0x2e75b0d#012   0x2deca04#012   0x2df71a3#012   0xef8811#012   0xef8b08#012   0x26ed300#012   0x2df4eee#012   0x2e1e8c4#012   0x2e1f881#012   0x12acfed#012   0x12bd71a#012   0x12bfffb#012   0x12ef684#012   0x12f0bfb#012   0x1392f1f#012   0x1393b9e#012   0x13b239a#012   0x1167e5c#012   0x116912a#012   0x116bfa5#012   0x116e51d#012   0x116ea9a#012   0x117aa61#012   0x117b5c5#012   0x1167e5c#012   0x116912a#012   0x116bfa5#012   0x116e51d#012   0x116ea9a#012   0x11370b4#012   0x116a523#012   0x116aaf7#012   0x116b17d#012   0xfdb0e3#012   0x25c4791#012   0x25c5160#012   0x25c5d83#012   0x25c7208#012   0x25c92d1#012   0x25cae17#012   0x25d4c58#012   0x25d65a3#012   0x313f25c#012   --------#012   N7seastar12continuationINS_8internal22promise_base_with_typeIJEEEZNS_5asyncIZN16row_level_repair3runEvEUlvE_JEEENS_8futurizeINSt9result_ofIFNSt5decayIT_E4typeEDpNS9_IT0_E4typeEEE4typeEE4typeENS_17thread_attributesEOSA_DpOSD_EUlvE0_ZZNS_6futureIJEE14then_impl_nrvoISQ_SS_EET0_SN_ENKUlvE_clEvEUlRS3_RSQ_ONS_12future_stateIJEEEE_JEEE#012   --------#012   seastar::continuation<seastar::internal::promise_base_with_type<>, seastar::future<>::finally_body<seastar::async<row_level_repair::run()::{lambda()#1}>(seastar::thread_attributes, std::decay&&, (std::decay<row_level_repair::run()::{lambda()#1}>::type&&)...)::{lambda()#3}, false>, seastar::future<>::then_wrapped_nrvo<seastar::future<>, {lambda()#3}>({lambda()#3}&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<>&, {lambda()#3}&, seastar::future_state<>&&)#1}>#012   --------#012   seastar::internal::do_with_state<std::tuple<row_level_repair>, seastar::future<> >#012   --------#012   seastar::continuation<seastar::internal::promise_base_with_type<>, seastar::future<>::handle_exception_type<repair_range(repair_info&, nonwrapping_interval<dht::token> const&)::{lambda(auto:1&, auto:2&)#1}::operator()<std::vector<gms::inet_address, std::allocator<gms> >, std::allocator<gms> >(std::vector<gms::inet_address, std::allocator<gms> >&, std::allocator<gms>&) const::{lambda()#2}::operator()() const::{lambda(utils::UUID)#1}::operator()({lambda()#2}) const::{lambda(no_such_column_family&)#1}>(std::vector<gms::inet_address, std::allocator<gms> >&&)::{lambda(no_such_column_family)#1}, seastar::future<>::then_wrapped_nrvo<seastar::future<>, no_such_column_family&>(std::allocator<gms>&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<>&, no_such_column_family&, seastar::future_state<>&&)#1}>#012   --------#012   seastar::continuation<seastar::internal::promise_base_with_type<>, seastar::future<>::handle_exception<repair_range(repair_info&, nonwrapping_interval<dht::token> const&)::{lambda(auto:1&, auto:2&)#1}::operator()<std::vector<gms::inet_address, std::allocator<gms> >, std::allocator<gms> >(std::vector<gms::inet_address, std::allocator<gms> >&, std::allocator<gms>&) const::{lambda()#2}::operator()() const::{lambda(utils::UUID)#1}::operator()({lambda()#2}) const::{lambda(std::__exception_ptr::exception_ptr)#2}>(std::vector<gms::inet_address, std::allocator<gms> >&&)::{lambda(std::__exception_ptr)#1}, seastar::future<>::then_wrapped_nrvo<seastar::future<>, std::__exception_ptr::exception_ptr>(std::allocator<gms>&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<>&, std::__exception_ptr::exception_ptr&, seastar::future_state<>&&)#1}>#012   --------#012   seastar::internal::do_for_each_state<__gnu_cxx::__normal_iterator<utils::UUID*, std::vector<utils::UUID, std::allocator<utils::UUID> > >, repair_range(repair_info&, nonwrapping_interval<dht::token> const&)::{lambda(auto:1&, auto:2&)#1}::operator()<std::vector<gms::inet_address, std::allocator<gms> >, std::allocator<gms> >(std::vector<gms::inet_address, std::allocator<gms> >&, std::allocator<gms>&) const::{lambda()#2}::operator()() const::{lambda(utils::UUID)#1}>#012   --------#012   seastar::internal::do_with_state<std::tuple<std::vector<gms::inet_address, std::allocator<gms::inet_address> >, std::vector<gms::inet_address, std::allocator<gms::inet_address> > >, seastar::future<> >#012   --------#012   N7seastar12continuationINS_8internal22promise_base_with_typeIJEEENS_6futureIJEE12finally_bodyIZZNS_14with_semaphoreINS_33named_semaphore_exception_factoryEZZL16do_repair_rangesNS_13lw_shared_ptrI11repair_infoEEENKUlOT_E_clIR20nonwrapping_intervalIN3dht5tokenEEEEDaSD_EUlvE_NSt6chrono3_V212steady_clockEEENS_8futurizeINSt9result_ofIFT0_vEE4typeEE4typeERNS_15basic_semaphoreISC_T1_EEmOSR_ENUlSC_E_clINS_15semaphore_unitsIS8_SO_EEEEDaSC_EUlvE_Lb0EEEZZNS5_17then_wrapped_nrvoIS5_S17_EENSP_ISC_E4typeES11_ENKUlvE_clEvEUlRS3_RS17_ONS_12future_stateIJEEEE_JEEE
2020-09-08T16:12:47+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !WARNING | scylla: [shard 7] seastar_memory - oversized allocation: 3145728 bytes. This is non-fatal, but could lead to latency and/or fragmentation issues. Please report: at    0x333167d#012   0x3331990#012   0x3331e19#012   0x2e289c3#012   0x2e2d4fb#012   0x2eee2a4#012   0x2eedce4#012   0x3148d14#012   0x3149032#012   0x2e75b0d#012   0x2deca04#012   0x2df71a3#012   0xef8811#012   0xef8b08#012   0x26ed300#012   0x2df4eee#012   0x2e1e8c4#012   0x2e1f881#012   0x12acfed#012   0x12bd71a#012   0x12bfffb#012   0x12ef684#012   0x12f0bfb#012   0x1392f1f#012   0x1393b9e#012   0x13b239a#012   0x1167e5c#012   0x116912a#012   0x116bfa5#012   0x116e51d#012   0x116ea9a#012   0x117aa61#012   0x117b5c5#012   0x1167e5c#012   0x116912a#012   0x116bfa5#012   0x116e51d#012   0x116ea9a#012   0x11370b4#012   0x116a523#012   0x116aaf7#012   0x116b17d#012   0xfdb0e3#012   0x25c4791#012   0x25c5160#012   0x25c5d83#012   0x25c7208#012   0x25c92d1#012   0x25cae17#012   0x25d4c58#012   0x25d65a3#012   0x313f25c#012   --------#012   N7seastar12continuationINS_8internal22promise_base_with_typeIJEEEZNS_5asyncIZN16row_level_repair3runEvEUlvE_JEEENS_8futurizeINSt9result_ofIFNSt5decayIT_E4typeEDpNS9_IT0_E4typeEEE4typeEE4typeENS_17thread_attributesEOSA_DpOSD_EUlvE0_ZZNS_6futureIJEE14then_impl_nrvoISQ_SS_EET0_SN_ENKUlvE_clEvEUlRS3_RSQ_ONS_12future_stateIJEEEE_JEEE#012   --------#012   seastar::continuation<seastar::internal::promise_base_with_type<>, seastar::future<>::finally_body<seastar::async<row_level_repair::run()::{lambda()#1}>(seastar::thread_attributes, std::decay&&, (std::decay<row_level_repair::run()::{lambda()#1}>::type&&)...)::{lambda()#3}, false>, seastar::future<>::then_wrapped_nrvo<seastar::future<>, {lambda()#3}>({lambda()#3}&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<>&, {lambda()#3}&, seastar::future_state<>&&)#1}>#012   --------#012   seastar::internal::do_with_state<std::tuple<row_level_repair>, seastar::future<> >#012   --------#012   seastar::continuation<seastar::internal::promise_base_with_type<>, seastar::future<>::handle_exception_type<repair_range(repair_info&, nonwrapping_interval<dht::token> const&)::{lambda(auto:1&, auto:2&)#1}::operator()<std::vector<gms::inet_address, std::allocator<gms> >, std::allocator<gms> >(std::vector<gms::inet_address, std::allocator<gms> >&, std::allocator<gms>&) const::{lambda()#2}::operator()() const::{lambda(utils::UUID)#1}::operator()({lambda()#2}) const::{lambda(no_such_column_family&)#1}>(std::vector<gms::inet_address, std::allocator<gms> >&&)::{lambda(no_such_column_family)#1}, seastar::future<>::then_wrapped_nrvo<seastar::future<>, no_such_column_family&>(std::allocator<gms>&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<>&, no_such_column_family&, seastar::future_state<>&&)#1}>#012   --------#012   seastar::continuation<seastar::internal::promise_base_with_type<>, seastar::future<>::handle_exception<repair_range(repair_info&, nonwrapping_interval<dht::token> const&)::{lambda(auto:1&, auto:2&)#1}::operator()<std::vector<gms::inet_address, std::allocator<gms> >, std::allocator<gms> >(std::vector<gms::inet_address, std::allocator<gms> >&, std::allocator<gms>&) const::{lambda()#2}::operator()() const::{lambda(utils::UUID)#1}::operator()({lambda()#2}) const::{lambda(std::__exception_ptr::exception_ptr)#2}>(std::vector<gms::inet_address, std::allocator<gms> >&&)::{lambda(std::__exception_ptr)#1}, seastar::future<>::then_wrapped_nrvo<seastar::future<>, std::__exception_ptr::exception_ptr>(std::allocator<gms>&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<>&, std::__exception_ptr::exception_ptr&, seastar::future_state<>&&)#1}>#012   --------#012   seastar::internal::do_for_each_state<__gnu_cxx::__normal_iterator<utils::UUID*, std::vector<utils::UUID, std::allocator<utils::UUID> > >, repair_range(repair_info&, nonwrapping_interval<dht::token> const&)::{lambda(auto:1&, auto:2&)#1}::operator()<std::vector<gms::inet_address, std::allocator<gms> >, std::allocator<gms> >(std::vector<gms::inet_address, std::allocator<gms> >&, std::allocator<gms>&) const::{lambda()#2}::operator()() const::{lambda(utils::UUID)#1}>#012   --------#012   seastar::internal::do_with_state<std::tuple<std::vector<gms::inet_address, std::allocator<gms::inet_address> >, std::vector<gms::inet_address, std::allocator<gms::inet_address> > >, seastar::future<> >#012   --------#012   N7seastar12continuationINS_8internal22promise_base_with_typeIJEEENS_6futureIJEE12finally_bodyIZZNS_14with_semaphoreINS_33named_semaphore_exception_factoryEZZL16do_repair_rangesNS_13lw_shared_ptrI11repair_infoEEENKUlOT_E_clIR20nonwrapping_intervalIN3dht5tokenEEEEDaSD_EUlvE_NSt6chrono3_V212steady_clockEEENS_8futurizeINSt9result_ofIFT0_vEE4typeEE4typeERNS_15basic_semaphoreISC_T1_EEmOSR_ENUlSC_E_clINS_15semaphore_unitsIS8_SO_EEEEDaSC_EUlvE_Lb0EEEZZNS5_17then_wrapped_nrvoIS5_S17_EENSP_ISC_E4typeES11_ENKUlvE_clEvEUlRS3_RS17_ONS_12future_stateIJEEEE_JEEE
2020-09-08T16:14:37+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !WARNING | scylla: [shard 2] seastar_memory - oversized allocation: 3145728 bytes. This is non-fatal, but could lead to latency and/or fragmentation issues. Please report: at    0x333167d#012   0x3331990#012   0x3331e19#012   0x2e289c3#012   0x2e2d4fb#012   0x2eee2a4#012   0x2eedce4#012   0x3148d14#012   0x3149032#012   0x2e75b0d#012   0x2deca04#012   0x2df71a3#012   0xef8811#012   0xef8b08#012   0x26ed300#012   0x2df4eee#012   0x2e1e8c4#012   0x2e1f881#012   0x12acfed#012   0x12bd71a#012   0x12bfffb#012   0x12ef684#012   0x12f0bfb#012   0x1392f1f#012   0x1393b9e#012   0x13b239a#012   0x1167e5c#012   0x116912a#012   0x116bfa5#012   0x116e51d#012   0x116ea9a#012   0x117aa61#012   0x117b5c5#012   0x1167e5c#012   0x116912a#012   0x116bfa5#012   0x116e51d#012   0x116ea9a#012   0x11370b4#012   0x116a523#012   0x116aaf7#012   0x116b17d#012   0xfdb0e3#012   0x25c4791#012   0x25c5160#012   0x25c5d83#012   0x25c7208#012   0x25c92d1#012   0x25cae17#012   0x25d4c58#012   0x25d65a3#012   0x313f25c#012   --------#012   N7seastar12continuationINS_8internal22promise_base_with_typeIJEEEZNS_5asyncIZN16row_level_repair3runEvEUlvE_JEEENS_8futurizeINSt9result_ofIFNSt5decayIT_E4typeEDpNS9_IT0_E4typeEEE4typeEE4typeENS_17thread_attributesEOSA_DpOSD_EUlvE0_ZZNS_6futureIJEE14then_impl_nrvoISQ_SS_EET0_SN_ENKUlvE_clEvEUlRS3_RSQ_ONS_12future_stateIJEEEE_JEEE#012   --------#012   seastar::continuation<seastar::internal::promise_base_with_type<>, seastar::future<>::finally_body<seastar::async<row_level_repair::run()::{lambda()#1}>(seastar::thread_attributes, std::decay&&, (std::decay<row_level_repair::run()::{lambda()#1}>::type&&)...)::{lambda()#3}, false>, seastar::future<>::then_wrapped_nrvo<seastar::future<>, {lambda()#3}>({lambda()#3}&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<>&, {lambda()#3}&, seastar::future_state<>&&)#1}>#012   --------#012   seastar::internal::do_with_state<std::tuple<row_level_repair>, seastar::future<> >#012   --------#012   seastar::continuation<seastar::internal::promise_base_with_type<>, seastar::future<>::handle_exception_type<repair_range(repair_info&, nonwrapping_interval<dht::token> const&)::{lambda(auto:1&, auto:2&)#1}::operator()<std::vector<gms::inet_address, std::allocator<gms> >, std::allocator<gms> >(std::vector<gms::inet_address, std::allocator<gms> >&, std::allocator<gms>&) const::{lambda()#2}::operator()() const::{lambda(utils::UUID)#1}::operator()({lambda()#2}) const::{lambda(no_such_column_family&)#1}>(std::vector<gms::inet_address, std::allocator<gms> >&&)::{lambda(no_such_column_family)#1}, seastar::future<>::then_wrapped_nrvo<seastar::future<>, no_such_column_family&>(std::allocator<gms>&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<>&, no_such_column_family&, seastar::future_state<>&&)#1}>#012   --------#012   seastar::continuation<seastar::internal::promise_base_with_type<>, seastar::future<>::handle_exception<repair_range(repair_info&, nonwrapping_interval<dht::token> const&)::{lambda(auto:1&, auto:2&)#1}::operator()<std::vector<gms::inet_address, std::allocator<gms> >, std::allocator<gms> >(std::vector<gms::inet_address, std::allocator<gms> >&, std::allocator<gms>&) const::{lambda()#2}::operator()() const::{lambda(utils::UUID)#1}::operator()({lambda()#2}) const::{lambda(std::__exception_ptr::exception_ptr)#2}>(std::vector<gms::inet_address, std::allocator<gms> >&&)::{lambda(std::__exception_ptr)#1}, seastar::future<>::then_wrapped_nrvo<seastar::future<>, std::__exception_ptr::exception_ptr>(std::allocator<gms>&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<>&, std::__exception_ptr::exception_ptr&, seastar::future_state<>&&)#1}>#012   --------#012   seastar::internal::do_for_each_state<__gnu_cxx::__normal_iterator<utils::UUID*, std::vector<utils::UUID, std::allocator<utils::UUID> > >, repair_range(repair_info&, nonwrapping_interval<dht::token> const&)::{lambda(auto:1&, auto:2&)#1}::operator()<std::vector<gms::inet_address, std::allocator<gms> >, std::allocator<gms> >(std::vector<gms::inet_address, std::allocator<gms> >&, std::allocator<gms>&) const::{lambda()#2}::operator()() const::{lambda(utils::UUID)#1}>#012   --------#012   seastar::internal::do_with_state<std::tuple<std::vector<gms::inet_address, std::allocator<gms::inet_address> >, std::vector<gms::inet_address, std::allocator<gms::inet_address> > >, seastar::future<> >#012   --------#012   N7seastar12continuationINS_8internal22promise_base_with_typeIJEEENS_6futureIJEE12finally_bodyIZZNS_14with_semaphoreINS_33named_semaphore_exception_factoryEZZL16do_repair_rangesNS_13lw_shared_ptrI11repair_infoEEENKUlOT_E_clIR20nonwrapping_intervalIN3dht5tokenEEEEDaSD_EUlvE_NSt6chrono3_V212steady_clockEEENS_8futurizeINSt9result_ofIFT0_vEE4typeEE4typeERNS_15basic_semaphoreISC_T1_EEmOSR_ENUlSC_E_clINS_15semaphore_unitsIS8_SO_EEEEDaSC_EUlvE_Lb0EEEZZNS5_17then_wrapped_nrvoIS5_S17_EENSP_ISC_E4typeES11_ENKUlvE_clEvEUlRS3_RS17_ONS_12future_stateIJEEEE_JEEE
2020-09-08T16:15:25+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !WARNING | scylla: [shard 3] seastar_memory - oversized allocation: 3145728 bytes. This is non-fatal, but could lead to latency and/or fragmentation issues. Please report: at    0x333167d#012   0x3331990#012   0x3331e19#012   0x2e289c3#012   0x2e2d4fb#012   0x2eee2a4#012   0x2eedce4#012   0x3148d14#012   0x3149032#012   0x2e75b0d#012   0x2deca04#012   0x2df71a3#012   0xef8811#012   0xef8b08#012   0x26ed300#012   0x2df4eee#012   0x2e1e8c4#012   0x2e1f881#012   0x12acfed#012   0x12bd71a#012   0x12bfffb#012   0x12ef684#012   0x12f0bfb#012   0x1392f1f#012   0x1393b9e#012   0x13b239a#012   0x1167e5c#012   0x116912a#012   0x116bfa5#012   0x116e51d#012   0x116ea9a#012   0x117aa61#012   0x117b00f#012   0x2e77ea7#012   0x2e7821e#012   0x2eaf2fd#012   0x2ebecfa#012   0x2e42f7d#012   /opt/scylladb/libreloc/libpthread.so.0+0x9431#012   /opt/scylladb/libreloc/libc.so.6+0x101912#012   --------#012   seastar::continuation<seastar::internal::promise_base_with_type<>, restricting_mutation_reader::with_reader<restricting_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda(flat_mutation_reader&)#1}>(restricting_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda(flat_mutation_reader&)#1}, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda(reader_permit::resource_units)#1}, seastar::future<reader_permit>::then_impl_nrvo<reader_permit::resource_units, {lambda(reader_permit::resource_units)#1}<> >(restricting_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda(flat_mutation_reader&)#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<>&, reader_permit::resource_units&, seastar::future_state<reader_permit>&&)#1}, reader_permit>#012   --------#012   seastar::continuation<seastar::internal::promise_base_with_type<seastar::optimized_optional<mutation_fragment> >, flat_mutation_reader::impl::operator()(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}, seastar::future<>::then_impl_nrvo<{lambda()#1}, flat_mutation_reader::impl::operator()(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}<seastar::optimized_optional<mutation_fragment> > >({lambda()#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<seastar::optimized_optional<mutation_fragment> >&, {lambda()#1}&, seastar::future_state<>&&)#1}>#012   --------#012   seastar::continuation<seastar::internal::promise_base_with_type<seastar::bool_class<mutation_reader_merger::needs_merge_tag> >, mutation_reader_merger::prepare_one(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, mutation_reader_merger::reader_and_last_fragment_kind, seastar::bool_class<mutation_reader_merger::reader_galloping_tag>)::{lambda(seastar::optimized_optional<mutation_fragment>)#1}, seastar::future<seastar::optimized_optional<mutation_fragment> >::then_impl_nrvo<{lambda(seastar::optimized_optional<mutation_fragment>)#1}, mutation_reader_merger::prepare_one(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, mutation_reader_merger::reader_and_last_fragment_kind, seastar::bool_class<mutation_reader_merger::reader_galloping_tag>)::{lambda(seastar::optimized_optional<mutation_fragment>)#1}<seastar::bool_class<mutation_reader_merger::needs_merge_tag> > >({lambda(seastar::optimized_optional<mutation_fragment>)#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<seastar::bool_class<mutation_reader_merger::needs_merge_tag> >&, {lambda(seastar::optimized_optional<mutation_fragment>)#1}&, seastar::future_state<seastar::optimized_optional<mutation_fragment> >&&)#1}, seastar::optimized_optional<mutation_fragment> >#012   --------#012   N7seastar12continuationINS_8internal22promise_base_with_typeIJEEEZNS_6futureIJNS_10bool_classIN22mutation_reader_merger15needs_merge_tagEEEEE14discard_resultEvEUlDpOT_E_ZZNS9_14then_impl_nrvoISD_NS4_IJEEEEET0_OT_ENKUlvE_clEvEUlRS3_RSD_ONS_12future_stateIJS8_EEEE_JS8_EEE#012   --------#012   seastar::parallel_for_each_state#012   --------#012   seastar::continuation<seastar::internal::promise_base_with_type<>, mutation_reader_merger::prepare_next(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#2}, seastar::future<>::then_impl_nrvo<{lambda()#2}, seastar::future>({lambda()#2}&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<>&, {lambda()#2}&, seastar::future_state<>&&)#1}>#012   --------#012   seastar::continuation<seastar::internal::promise_base_with_type<boost::iterator_range<mutation_fragment*> >, mutation_reader_merger::operator()(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#3}, seastar::future<>::then_impl_nrvo<{lambda()#3}, mutation_reader_merger::operator()(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#3}<boost::iterator_range<mutation_fragment*> > >({lambda()#3}&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<boost::iterator_range<mutation_fragment*> >&, {lambda()#3}&, seastar::future_state<>&&)#1}>#012   --------#012   seastar::continuation<seastar::internal::promise_base_with_type<>, mutation_fragment_merger<mutation_reader_merger>::fetch(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda(boost::iterator_range<mutation_fragment*>)#1}, seastar::future<boost::iterator_range<mutation_fragment*> >::then_impl_nrvo<{lambda(boost::iterator_range<mutation_fragment*>)#1}, mutation_fragment_merger<mutation_reader_merger>::fetch(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda(boost::iterator_range<mutation_fragment*>)#1}<> >({lambda(boost::iterator_range<mutation_fragment*>)#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<>&, {lambda(boost::iterator_range<mutation_fragment*>)#1}&, seastar::future_state<boost::iterator_range<mutation_fragment*> >&&)#1}, boost::iterator_range<mutation_fragment*> >#012   --------#012   seastar::continuation<seastar::internal::promise_base_with_type<seastar::optimized_optional<mutation_fragment> >, mutation_fragment_merger<mutation_reader_merger>::operator()(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}, seastar::future<>::then_impl_nrvo<{lambda()#1}, mutation_fragment_merger<mutation_reader_merger>::operator()(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}<seastar::optimized_optional<mutation_fragment> > >({lambda()#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<seastar::optimized_optional<mutation_fragment> >&, {lambda()#1}&, seastar::future_state<>&&)#1}>#012   --------#012   seastar::continuation<seastar::internal::promise_base_with_type<seastar::bool_class<seastar::stop_iteration_tag> >, combined_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}::operator()() const::{lambda(seastar::optimized_optional<mutation_fragment>)#1}, seastar::future<mutation_fragment>::then_impl_nrvo<seastar::optimized_optional<mutation_fragment>, {lambda(seastar::optimized_optional<mutation_fragment>)#1}<seastar::bool_class<seastar::stop_iteration_tag> > >(seastar::optimized_optional<mutation_fragme
2020-09-08T17:08:23+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !WARNING | scylla: [shard 5] seastar_memory - oversized allocation: 3145728 bytes. This is non-fatal, but could lead to latency and/or fragmentation issues. Please report: at    0x333167d#012   0x3331990#012   0x3331e19#012   0x2e289c3#012   0x2e2d4fb#012   0x2eee2a4#012   0x2eedce4#012   0x3148d14#012   0x3149032#012   0x2e75b0d#012   0x2deca04#012   0x2df71a3#012   0xef8811#012   0xef8b08#012   0x26ed300#012   0x2df4eee#012   0x2e1e8c4#012   0x2e1f881#012   0x12acfed#012   0x12bd71a#012   0x12bfffb#012   0x12ef684#012   0x12f0bfb#012   0x1392f1f#012   0x1393b9e#012   0x13b239a#012   0x1167e5c#012   0x116912a#012   0x116bfa5#012   0x116e51d#012   0x116ea9a#012   0x117aa61#012   0x117b5c5#012   0x1167e5c#012   0x116912a#012   0x116bfa5#012   0x116e51d#012   0x116ea9a#012   0x11370b4#012   0x116a523#012   0x116aaf7#012   0x116b17d#012   0xfdb0e3#012   0x25c4791#012   0x25c5160#012   0x25c5d83#012   0x25c7208#012   0x25c92d1#012   0x25cae17#012   0x25d4c58#012   0x25d65a3#012   0x313f25c#012   --------#012   N7seastar12continuationINS_8internal22promise_base_with_typeIJEEEZNS_5asyncIZN16row_level_repair3runEvEUlvE_JEEENS_8futurizeINSt9result_ofIFNSt5decayIT_E4typeEDpNS9_IT0_E4typeEEE4typeEE4typeENS_17thread_attributesEOSA_DpOSD_EUlvE0_ZZNS_6futureIJEE14then_impl_nrvoISQ_SS_EET0_SN_ENKUlvE_clEvEUlRS3_RSQ_ONS_12future_stateIJEEEE_JEEE#012   --------#012   seastar::continuation<seastar::internal::promise_base_with_type<>, seastar::future<>::finally_body<seastar::async<row_level_repair::run()::{lambda()#1}>(seastar::thread_attributes, std::decay&&, (std::decay<row_level_repair::run()::{lambda()#1}>::type&&)...)::{lambda()#3}, false>, seastar::future<>::then_wrapped_nrvo<seastar::future<>, {lambda()#3}>({lambda()#3}&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<>&, {lambda()#3}&, seastar::future_state<>&&)#1}>#012   --------#012   seastar::internal::do_with_state<std::tuple<row_level_repair>, seastar::future<> >#012   --------#012   seastar::continuation<seastar::internal::promise_base_with_type<>, seastar::future<>::handle_exception_type<repair_range(repair_info&, nonwrapping_interval<dht::token> const&)::{lambda(auto:1&, auto:2&)#1}::operator()<std::vector<gms::inet_address, std::allocator<gms> >, std::allocator<gms> >(std::vector<gms::inet_address, std::allocator<gms> >&, std::allocator<gms>&) const::{lambda()#2}::operator()() const::{lambda(utils::UUID)#1}::operator()({lambda()#2}) const::{lambda(no_such_column_family&)#1}>(std::vector<gms::inet_address, std::allocator<gms> >&&)::{lambda(no_such_column_family)#1}, seastar::future<>::then_wrapped_nrvo<seastar::future<>, no_such_column_family&>(std::allocator<gms>&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<>&, no_such_column_family&, seastar::future_state<>&&)#1}>#012   --------#012   seastar::continuation<seastar::internal::promise_base_with_type<>, seastar::future<>::handle_exception<repair_range(repair_info&, nonwrapping_interval<dht::token> const&)::{lambda(auto:1&, auto:2&)#1}::operator()<std::vector<gms::inet_address, std::allocator<gms> >, std::allocator<gms> >(std::vector<gms::inet_address, std::allocator<gms> >&, std::allocator<gms>&) const::{lambda()#2}::operator()() const::{lambda(utils::UUID)#1}::operator()({lambda()#2}) const::{lambda(std::__exception_ptr::exception_ptr)#2}>(std::vector<gms::inet_address, std::allocator<gms> >&&)::{lambda(std::__exception_ptr)#1}, seastar::future<>::then_wrapped_nrvo<seastar::future<>, std::__exception_ptr::exception_ptr>(std::allocator<gms>&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<>&, std::__exception_ptr::exception_ptr&, seastar::future_state<>&&)#1}>#012   --------#012   seastar::internal::do_for_each_state<__gnu_cxx::__normal_iterator<utils::UUID*, std::vector<utils::UUID, std::allocator<utils::UUID> > >, repair_range(repair_info&, nonwrapping_interval<dht::token> const&)::{lambda(auto:1&, auto:2&)#1}::operator()<std::vector<gms::inet_address, std::allocator<gms> >, std::allocator<gms> >(std::vector<gms::inet_address, std::allocator<gms> >&, std::allocator<gms>&) const::{lambda()#2}::operator()() const::{lambda(utils::UUID)#1}>#012   --------#012   seastar::internal::do_with_state<std::tuple<std::vector<gms::inet_address, std::allocator<gms::inet_address> >, std::vector<gms::inet_address, std::allocator<gms::inet_address> > >, seastar::future<> >#012   --------#012   N7seastar12continuationINS_8internal22promise_base_with_typeIJEEENS_6futureIJEE12finally_bodyIZZNS_14with_semaphoreINS_33named_semaphore_exception_factoryEZZL16do_repair_rangesNS_13lw_shared_ptrI11repair_infoEEENKUlOT_E_clIR20nonwrapping_intervalIN3dht5tokenEEEEDaSD_EUlvE_NSt6chrono3_V212steady_clockEEENS_8futurizeINSt9result_ofIFT0_vEE4typeEE4typeERNS_15basic_semaphoreISC_T1_EEmOSR_ENUlSC_E_clINS_15semaphore_unitsIS8_SO_EEEEDaSC_EUlvE_Lb0EEEZZNS5_17then_wrapped_nrvoIS5_S17_EENSP_ISC_E4typeES11_ENKUlvE_clEvEUlRS3_RS17_ONS_12future_stateIJEEEE_JEEE
2020-09-08T17:22:33+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !WARNING | scylla: [shard 4] seastar_memory - oversized allocation: 6291456 bytes. This is non-fatal, but could lead to latency and/or fragmentation issues. Please report: at    0x333167d#012   0x3331990#012   0x3331e19#012   0x2e289c3#012   0x2e2d4fb#012   0x2eee2a4#012   0x2eedce4#012   0x3148d14#012   0x3149032#012   0x2e75b0d#012   0x2deca04#012   0x2df71a3#012   0xef8811#012   0xef8b08#012   0x26ed300#012   0x2df4eee#012   0x2e1e8c4#012   0x2e1f881#012   0x12acfed#012   0x12bd71a#012   0x12bfffb#012   0x12ef684#012   0x12f0bfb#012   0x1392f1f#012   0x1393b9e#012   0x13b239a#012   0x1167e5c#012   0x116912a#012   0x116bfa5#012   0x116e51d#012   0x116ea9a#012   0x117aa61#012   0x117b5c5#012   0x1167e5c#012   0x116912a#012   0x116bfa5#012   0x116e51d#012   0x116ea9a#012   0x11370b4#012   0x116a523#012   0x116aaf7#012   0x116b17d#012   0xfdb0e3#012   0x25c4791#012   0x25c5160#012   0x25c5d83#012   0x25c7208#012   0x25c92d1#012   0x25cae17#012   0x25d4c58#012   0x25d65a3#012   0x313f25c#012   --------#012   N7seastar12continuationINS_8internal22promise_base_with_typeIJEEEZNS_5asyncIZN16row_level_repair3runEvEUlvE_JEEENS_8futurizeINSt9result_ofIFNSt5decayIT_E4typeEDpNS9_IT0_E4typeEEE4typeEE4typeENS_17thread_attributesEOSA_DpOSD_EUlvE0_ZZNS_6futureIJEE14then_impl_nrvoISQ_SS_EET0_SN_ENKUlvE_clEvEUlRS3_RSQ_ONS_12future_stateIJEEEE_JEEE#012   --------#012   seastar::continuation<seastar::internal::promise_base_with_type<>, seastar::future<>::finally_body<seastar::async<row_level_repair::run()::{lambda()#1}>(seastar::thread_attributes, std::decay&&, (std::decay<row_level_repair::run()::{lambda()#1}>::type&&)...)::{lambda()#3}, false>, seastar::future<>::then_wrapped_nrvo<seastar::future<>, {lambda()#3}>({lambda()#3}&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<>&, {lambda()#3}&, seastar::future_state<>&&)#1}>#012   --------#012   seastar::internal::do_with_state<std::tuple<row_level_repair>, seastar::future<> >#012   --------#012   seastar::continuation<seastar::internal::promise_base_with_type<>, seastar::future<>::handle_exception_type<repair_range(repair_info&, nonwrapping_interval<dht::token> const&)::{lambda(auto:1&, auto:2&)#1}::operator()<std::vector<gms::inet_address, std::allocator<gms> >, std::allocator<gms> >(std::vector<gms::inet_address, std::allocator<gms> >&, std::allocator<gms>&) const::{lambda()#2}::operator()() const::{lambda(utils::UUID)#1}::operator()({lambda()#2}) const::{lambda(no_such_column_family&)#1}>(std::vector<gms::inet_address, std::allocator<gms> >&&)::{lambda(no_such_column_family)#1}, seastar::future<>::then_wrapped_nrvo<seastar::future<>, no_such_column_family&>(std::allocator<gms>&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<>&, no_such_column_family&, seastar::future_state<>&&)#1}>#012   --------#012   seastar::continuation<seastar::internal::promise_base_with_type<>, seastar::future<>::handle_exception<repair_range(repair_info&, nonwrapping_interval<dht::token> const&)::{lambda(auto:1&, auto:2&)#1}::operator()<std::vector<gms::inet_address, std::allocator<gms> >, std::allocator<gms> >(std::vector<gms::inet_address, std::allocator<gms> >&, std::allocator<gms>&) const::{lambda()#2}::operator()() const::{lambda(utils::UUID)#1}::operator()({lambda()#2}) const::{lambda(std::__exception_ptr::exception_ptr)#2}>(std::vector<gms::inet_address, std::allocator<gms> >&&)::{lambda(std::__exception_ptr)#1}, seastar::future<>::then_wrapped_nrvo<seastar::future<>, std::__exception_ptr::exception_ptr>(std::allocator<gms>&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<>&, std::__exception_ptr::exception_ptr&, seastar::future_state<>&&)#1}>#012   --------#012   seastar::internal::do_for_each_state<__gnu_cxx::__normal_iterator<utils::UUID*, std::vector<utils::UUID, std::allocator<utils::UUID> > >, repair_range(repair_info&, nonwrapping_interval<dht::token> const&)::{lambda(auto:1&, auto:2&)#1}::operator()<std::vector<gms::inet_address, std::allocator<gms> >, std::allocator<gms> >(std::vector<gms::inet_address, std::allocator<gms> >&, std::allocator<gms>&) const::{lambda()#2}::operator()() const::{lambda(utils::UUID)#1}>#012   --------#012   seastar::internal::do_with_state<std::tuple<std::vector<gms::inet_address, std::allocator<gms::inet_address> >, std::vector<gms::inet_address, std::allocator<gms::inet_address> > >, seastar::future<> >#012   --------#012   N7seastar12continuationINS_8internal22promise_base_with_typeIJEEENS_6futureIJEE12finally_bodyIZZNS_14with_semaphoreINS_33named_semaphore_exception_factoryEZZL16do_repair_rangesNS_13lw_shared_ptrI11repair_infoEEENKUlOT_E_clIR20nonwrapping_intervalIN3dht5tokenEEEEDaSD_EUlvE_NSt6chrono3_V212steady_clockEEENS_8futurizeINSt9result_ofIFT0_vEE4typeEE4typeERNS_15basic_semaphoreISC_T1_EEmOSR_ENUlSC_E_clINS_15semaphore_unitsIS8_SO_EEEEDaSC_EUlvE_Lb0EEEZZNS5_17then_wrapped_nrvoIS5_S17_EENSP_ISC_E4typeES11_ENKUlvE_clEvEUlRS3_RS17_ONS_12future_stateIJEEEE_JEEE
asias commented 4 years ago

The tests has a procedure to take a node down. The "storage_proxy - exception during mutation write to 10.0.3.65: seastar::named_semaphore_timed_out (Semaphore timed out: smp_service_group#2 2->13 semaphore)" issue is very similar to https://github.com/scylladb/scylla/issues/7025 which is caused by hinted hand-off, which @eliransin is looking into it.

asias commented 4 years ago

The only stall (Reactor stalled for 651 ms on shard 10.) in node6.log is caused by remove_if below. We need to futurize trim_to_compact.

$ cat 6.txt |grep 'Reactor stalled for'
2020-09-08T16:29:16+00:00  longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO    | scylla: Reactor stalled for 651 ms on shard 10.
147 int compaction_manager::trim_to_compact(column_family* cf, sstables::compaction_descriptor& descriptor) {
148     int weight = calculate_weight(descriptor.sstables);
149     // NOTE: a compaction job with level > 0 cannot be trimmed because leveled
150     // compaction relies on higher levels having no overlapping sstables.
151     if (descriptor.level != 0 || descriptor.sstables.empty()) {
152         return weight;
153     }
154 
155     uint64_t total_size = get_total_size(descriptor.sstables);
156     int min_threshold = cf->min_compaction_threshold();
157     auto compacting_run_identifiers = boost::copy_range<std::unordered_set<utils::UUID>>(descriptor.sstables
158         | boost::adaptors::transformed(std::mem_fn(&sstables::sstable::run_identifier)));
159 
160     while (compacting_run_identifiers.size() > size_t(min_threshold)) {
161         if (_weight_tracker.count(weight)) {
162             auto run_id_to_remove = descriptor.sstables.back()->run_identifier();
163             auto e = boost::range::remove_if(descriptor.sstables, [&] (sstables::shared_sstable& sst) -> bool {
164                 if (sst->run_identifier() != run_id_to_remove) {
165                     return false;
166                 }
167                 total_size -= sst->data_size();
168                 return true;
169             });
170             descriptor.sstables.erase(e, descriptor.sstables.end());
171             compacting_run_identifiers.erase(run_id_to_remove);
172             weight = calculate_weight(total_size);
173         } else {
174             break;
175         }
176     }
177     return weight;
178 }
asias commented 4 years ago

compaction_manager::trim_to_compact is removed in commit 99b75d1f6378cccf03d43f687863d64fbc0c97f9 (compaction: Improve compaction efficiency by killing the procedure that trims jobs). Should we backport to 4.2 to avoid the stall, @raphaelsc @avikivity ?

slivne commented 4 years ago

thanks asias - reassigned @raphaelsc / @bhalevy / @avikivity

avikivity commented 4 years ago

compaction_manager::trim_to_compact is removed in commit 99b75d1 (compaction: Improve compaction efficiency by killing the procedure that trims jobs). Should we backport to 4.2 to avoid the stall, @raphaelsc @avikivity ?

Such backports are dangerous. We commit to master and backport later in order to allow time to find problems.

I don't see why remove_if() should be so slow.

asias commented 4 years ago

O(N), N is the number of sstables. We have a lot of them.

raphaelsc commented 4 years ago

O(N), N is the number of sstables. We have a lot of them.

trim_to_compact() runs on the SSTables selected for compaction, which is usually only a small subset (unless you do major compaction or something). So I wouldn't expect such a stall coming from trim_to_compact() itself. 651ms is too much.

raphaelsc commented 4 years ago

I am seeing awkward things like this in the log:

2020-09-08T16:14:18+00:00 longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO | scylla: [shard 6] compaction - Compacted 6 sstables to [/var/lib/scylla/data/cdc_test/test_table_scylla_cdc_log-de936812f1d211ea8b07000000000008/mc-27446-big-Data.db:level=0, ]. 71kB to 34kB (~48% of original) in 2600422ms = 13 bytes/s. ~768 total partitions merged to 6.

Compaction taking 2600 seconds to compact 71kB of data?

Could it be an inverse effect of the backlog controller bug affecting TWCS (https://github.com/scylladb/scylla/issues/6054)? Have to investigate this.

I am seeing big compaction jobs running at roughly the same time of stall detection: 2020-09-08T15:16:49+00:00 longevity-cdc-100gb-4h-4-2-db-node-f4c109de-6 !INFO | scylla: [shard 7] compaction - Compacted 972 sstables to [/var/lib/scylla/data/cdc_test/test_table_scylla_cdc_log-de936812f1d211ea8b07000000000008/mc-17717-big-Data.db:level=0, ]. 36MB to 36kB (~0% of original) in 344ms = 104kB/s. ~129792 total partitions merged to 32.

Please keep in mind that those SSTables aren't actually compacted because they're fully expired.

I don't think trim_to_compact() alone is the source of stall. Prior to trim_to_compact() call, we: 1) copied non-compacting ssts into a candidates set 2) called compaction strategy to get a set of sstables to compact, that includes some expensive operations like finding fully expired sstables. Also sstable_set is now fully copied into the compaction descriptor (that was previously done in compaction, not in the strategy). 3) finally calling trim_to_compact() which is also kind of expensive because it copies all run UUIDs into a set, and remove the SSTs from the set of SSTables to compact.

Even though trim_to_compact() was the last reported action, the stall comes from all those operations combined. I think that step 2 is the probably the major factor of the stall, not step 3 (trim_to_compact()).

raphaelsc commented 4 years ago

BTW, with versioned_sstable_set (https://github.com/scylladb/scylla/pull/7272), we no longer have to actually copy the set into compaction, reducing significantly the stall observed here.

slivne commented 3 years ago

sstable_set work will hopefully be part of 4.5

raphaelsc commented 3 years ago

we no longer have trim_to_compact() after 99b75d1f6378cccf03

raphaelsc commented 1 year ago

@aleksbykov do you remember if we hit this in the last few months?

mykaul commented 3 months ago

@aleksbykov ?