search

redpanda-data / redpanda

Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
https://redpanda.com
9.41k stars 577 forks source link

Failed to allocate crash #17914

Open DrorBuhnik opened 5 months ago

DrorBuhnik commented 5 months ago

Version & Environment

Version: v23.3.10 Git ref: 255020e51d Build date: 2024-03-28T07:53:07Z OS/Arch: linux/arm64 Go version: go1.21.3

Redpanda Cluster node-0 v23.3.10 - 255020e51d9ed9b7f90a67872bfc81d62923e3e9 node-1 v23.3.10 - 255020e51d9ed9b7f90a67872bfc81d62923e3e9 node-2 v23.3.10 - 255020e51d9ed9b7f90a67872bfc81d62923e3e9

Kubernetes: v1.24.17-eks-508b6b3 (AWS EKS)

Kafka Client: kafakaJS (https://kafka.js.org/)

What went wrong?

Once in a while, might be a few days or might be a few weeks, one of the Redpanda instances (out of 3) crash with this log:

2024-04-17 13:57:32.600 INFO  2024-04-17 10:57:32,600 [shard  8:main] storage - segment.cc:765 - Creating new segment /var/lib/redpanda/data/kafka/__consumer_offsets/1_70/52672520690-1-v1.log
2024-04-17 13:57:35.088 ERROR 2024-04-17 10:57:35,086 [shard  8:main] seastar_memory - Dumping seastar memory diagnostics
2024-04-17 13:57:35.089 Used memory:   1960M
2024-04-17 13:57:35.089 Free memory:   2M
2024-04-17 13:57:35.089 Total memory:  2G
2024-04-17 13:57:35.089 Hard failures: 1
2024-04-17 13:57:35.089 
2024-04-17 13:57:35.089 Alloc sites legend:
2024-04-17 13:57:35.089     size: the estimated total size of all allocations at this stack (i.e., adjusted up from observed samples)
2024-04-17 13:57:35.089     count: the number of live samples at this stack (i.e., NOT adjusted up from observed samples)
2024-04-17 13:57:35.089     at: the backtrace for this allocation site
2024-04-17 13:57:35.089 Top-N alloc sites:
2024-04-17 13:57:35.089 size: 516006364 count: 172 at: 0x7decd3b 0x7af6bd7 0x7b02af7 0x70502e7 0x7047397 0x715ccc3 0x715dabb 0x6f4e65f 0x7ba481b 0x7ba6ecf 0x7bf0233 0x7b44ecb /opt/redpanda/lib/libc.so.6+0x843b7 /opt/redpanda/lib/libc.so.6+0xef2db
2024-04-17 13:57:35.089 size: 441005439 count: 147 at: 0x7decd3b 0x7af6bd7 0x7b02af7 0x705034b 0x7047397 0x715ccc3 0x715dabb 0x6f4e65f 0x7ba481b 0x7ba6ecf 0x7bf0233 0x7b44ecb /opt/redpanda/lib/libc.so.6+0x843b7 /opt/redpanda/lib/libc.so.6+0xef2db
2024-04-17 13:57:35.089 size: 432005328 count: 144 at: 0x7decd3b 0x7af6bd7 0x7b02af7 0x7050437 0x7047397 0x715ccc3 0x715dabb 0x6f4e65f 0x7ba481b 0x7ba6ecf 0x7bf0233 0x7b44ecb /opt/redpanda/lib/libc.so.6+0x843b7 /opt/redpanda/lib/libc.so.6+0xef2db
2024-04-17 13:57:35.089 size: 153001887 count: 51 at: 0x7decd3b 0x7b0d9d7 0x7af86af 0x7aeb813 0x2f12813 0x2f12f13 0x7ba481b 0x7ba6ecf 0x7bf0233 0x7b44ecb /opt/redpanda/lib/libc.so.6+0x843b7 /opt/redpanda/lib/libc.so.6+0xef2db
2024-04-17 13:57:35.089 size: 120001480 count: 40 at: 0x7decd3b 0x7af749f 0x7b02af7 0x71a0f9b 0x2d5f937 0x7ba481b 0x7ba6ecf 0x7bf0233 0x7b44ecb /opt/redpanda/lib/libc.so.6+0x843b7 /opt/redpanda/lib/libc.so.6+0xef2db
2024-04-17 13:57:35.089 size: 45000555 count: 15 at: 0x7decd3b 0x7af6bd7 0x7b02af7 0x700752b 0x7009e77 0x7ba481b 0x7ba6ecf 0x7bf0233 0x7b44ecb /opt/redpanda/lib/libc.so.6+0x843b7 /opt/redpanda/lib/libc.so.6+0xef2db
2024-04-17 13:57:35.089 size: 33554432 count: 2 at: 0x7decd3b 0x7aef613 0x7af8dd3 0x7aeb813 0x71b6297 0x71b6cdb 0x71b5b7b 0x2c9d677 0x7ba481b 0x7ba6ecf 0x7bf0233 0x7b44ecb /opt/redpanda/lib/libc.so.6+0x843b7 /opt/redpanda/lib/libc.so.6+0xef2db
2024-04-17 13:57:35.089 size: 18000222 count: 6 at: 0x7decd3b 0x7af6bd7 0x7b02af7 0x705034b 0x7047397 0x7165d63 0x36fdf6f 0x7ba481b 0x7ba6ecf 0x7bf0233 0x7b44ecb /opt/redpanda/lib/libc.so.6+0x843b7 /opt/redpanda/lib/libc.so.6+0xef2db
2024-04-17 13:57:35.089 size: 18000222 count: 6 at: 0x7decd3b 0x7af6bd7 0x7b02af7 0x71130ef 0x2d5f937 0x7ba481b 0x7ba6ecf 0x7bf0233 0x7b44ecb /opt/redpanda/lib/libc.so.6+0x843b7 /opt/redpanda/lib/libc.so.6+0xef2db
2024-04-17 13:57:35.089 size: 16777216 count: 1 at: 0x7decd3b 0x7aef613 0x7af8dd3 0x7aeb813 0x71b1d9f 0x2c9d663 0x7ba481b 0x7ba6ecf 0x7bf0233 0x7b44ecb /opt/redpanda/lib/libc.so.6+0x843b7 /opt/redpanda/lib/libc.so.6+0xef2db
2024-04-17 13:57:35.089 If you work at Redpanda please refer to https://vectorizedio.atlassian.net/l/cp/iuEMd2NN
2024-04-17 13:57:35.089 Small pools:
2024-04-17 13:57:35.089 objsz spansz usedobj memory unused wst%
2024-04-17 13:57:35.089     8     4K      2k    40K    21K   52
2024-04-17 13:57:35.089    10     4K       1     8K     8K   99
2024-04-17 13:57:35.089    12     4K       3     8K     8K   99
2024-04-17 13:57:35.089    14     4K       2     8K     8K   99
2024-04-17 13:57:35.089    16     4K      2k    40K    13K   31
2024-04-17 13:57:35.089    32     4K    129k     4M     6K    0
2024-04-17 13:57:35.089    32     4K    172k     5M     8K    0
2024-04-17 13:57:35.089    32     4K      3k   116K     7K    6
2024-04-17 13:57:35.089    32     4K      3k   812K   719K   88
2024-04-17 13:57:35.089    48     4K      7k   360K    46K   12
2024-04-17 13:57:35.089    48     4K      3k   860K   733K   85
2024-04-17 13:57:35.089    64     4K      2k   172K    38K   22
2024-04-17 13:57:35.089    64     4K    143k     9M     2K    0
2024-04-17 13:57:35.089    80     4K     55k     4M   960B    0
2024-04-17 13:57:35.089    96     4K     50k     5M     4K    0
2024-04-17 13:57:35.089   112     4K     361   156K   116K   74
2024-04-17 13:57:35.089   128     4K     448   340K   284K   83
2024-04-17 13:57:35.089   160     4K     600   116K    22K   19
2024-04-17 13:57:35.089   192     4K      5k     1M    34K    3
2024-04-17 13:57:35.089   224     4K     383   108K    24K   22
2024-04-17 13:57:35.089   256     4K     621   396K   241K   60
2024-04-17 13:57:35.089   320     8K     247   184K   107K   57
2024-04-17 13:57:35.089   384     8K     179   152K    85K   55
2024-04-17 13:57:35.089   448     4K     57k    25M    41K    0
2024-04-17 13:57:35.089   512     4K     205   204K   102K   49
2024-04-17 13:57:35.089   640    16K      95   240K   181K   75
2024-04-17 13:57:35.089   768    16K     630   768K   296K   38
2024-04-17 13:57:35.089   896     8K     228   288K    88K   30
2024-04-17 13:57:35.089  1024     4K     56k    55M   128K    0
2024-04-17 13:57:35.089  1280    32K     211   672K   408K   60
2024-04-17 13:57:35.089  1536    32K      12   512K   494K   96
2024-04-17 13:57:35.089  1792    16K      79   496K   357K   71
2024-04-17 13:57:35.089  2048     8K      72   464K   320K   68
2024-04-17 13:57:35.089  2560    64K       7   512K   493K   96
2024-04-17 13:57:35.089  3072    64K      74     1M   864K   79
2024-04-17 13:57:35.089  3584    32K      11   704K   665K   94
2024-04-17 13:57:35.089  4096    16K     230     2M   792K   46
2024-04-17 13:57:35.089  5120   128K       6   896K   865K   96
2024-04-17 13:57:35.089  6144   128K       5     1M     1M   97
2024-04-17 13:57:35.089  7168    64K      44     2M     2M   83
2024-04-17 13:57:35.089  8192    32K    169k  1318M   600K    0
2024-04-17 13:57:35.089 10240    64K      15     2M     2M   91
2024-04-17 13:57:35.089 12288    64K       9     2M     2M   93
2024-04-17 13:57:35.089 14336   128K       4     3M     3M   98
2024-04-17 13:57:35.089 16384    64K     12k   185M     2M    1
2024-04-17 13:57:35.089 
2024-04-17 13:57:35.089 Page spans:
2024-04-17 13:57:35.089 index  size  free  used spans
2024-04-17 13:57:35.089     0    4K    4K  111M   28k
2024-04-17 13:57:35.089     1    8K    2M    1M   374
2024-04-17 13:57:35.089     2   16K  256K    4M   252
2024-04-17 13:57:35.089     3   32K    0B 1546M   49k
2024-04-17 13:57:35.089     4   64K    0B  196M    3k
2024-04-17 13:57:35.089     5  128K    0B   10M    81
2024-04-17 13:57:35.089     6  256K    0B    1M     5
2024-04-17 13:57:35.089     7  512K    0B    1M     2
2024-04-17 13:57:35.089     8    1M    0B    0B     0
2024-04-17 13:57:35.089     9    2M    0B    2M     1
2024-04-17 13:57:35.089    10    4M    0B    8M     2
2024-04-17 13:57:35.089    11    8M    0B    0B     0
2024-04-17 13:57:35.089    12   16M    0B   48M     3
2024-04-17 13:57:35.089    13   32M    0B   32M     1
2024-04-17 13:57:35.089    14   64M    0B    0B     0
2024-04-17 13:57:35.089    15  128M    0B    0B     0
2024-04-17 13:57:35.089    16  256M    0B    0B     0
2024-04-17 13:57:35.089    17  512M    0B    0B     0
2024-04-17 13:57:35.089    18    1G    0B    0B     0
2024-04-17 13:57:35.089    19    2G    0B    0B     0
2024-04-17 13:57:35.089    20    4G    0B    0B     0
2024-04-17 13:57:35.089    21    8G    0B    0B     0
2024-04-17 13:57:35.089    22   16G    0B    0B     0
2024-04-17 13:57:35.089    23   32G    0B    0B     0
2024-04-17 13:57:35.089    24   64G    0B    0B     0
2024-04-17 13:57:35.089    25  128G    0B    0B     0
2024-04-17 13:57:35.089    26  256G    0B    0B     0
2024-04-17 13:57:35.089    27  512G    0B    0B     0
2024-04-17 13:57:35.089    28    1T    0B    0B     0
2024-04-17 13:57:35.089    29    2T    0B    0B     0
2024-04-17 13:57:35.089    30    4T    0B    0B     0
2024-04-17 13:57:35.089    31    8T    0B    0B     0
2024-04-17 13:57:35.089 
2024-04-17 13:57:35.089 ERROR 2024-04-17 10:57:35,088 [shard  8:main] seastar - Failed to allocate 30736 bytes
2024-04-17 13:57:35.089 Aborting on shard 8.
2024-04-17 13:57:35.089 Backtrace:
2024-04-17 13:57:35.089   0x7ba9ef3
2024-04-17 13:57:35.089   0x7beec43
2024-04-17 13:57:35.089   linux-vdso.so.1+0x84f
2024-04-17 13:57:35.089   /opt/redpanda/lib/libc.so.6+0x86067
2024-04-17 13:57:35.089   /opt/redpanda/lib/libc.so.6+0x3e87f
2024-04-17 13:57:35.089   /opt/redpanda/lib/libc.so.6+0x2aef7
2024-04-17 13:57:35.089   0x7af95ff
2024-04-17 13:57:35.089   0x7af7d0b
2024-04-17 13:57:35.089   0x7b02af7
2024-04-17 13:57:35.089   0x46905db
2024-04-17 13:57:35.089   0x6eebf8b
2024-04-17 13:57:35.089   0x6f233eb
2024-04-17 13:57:35.089   0x7ba481b
2024-04-17 13:57:35.089   0x7ba6ecf
2024-04-17 13:57:35.089   0x7bf0233
2024-04-17 13:57:35.089   0x7b44ecb
2024-04-17 13:57:35.089   /opt/redpanda/lib/libc.so.6+0x843b7
2024-04-17 13:57:35.089   /opt/redpanda/lib/libc.so.6+0xef2db

It is during regular work on the cluster, nothing special happens in the producers/consumers at the time, no management changes, no infrastructure changes Once the instance got this first "failed to allocate" crash I will get it on that instance again and again during the automatic k8s restarts. my only "fix" right now is to delete the cluster (including the disks) and start fresh until the next crash

Also, I updated Redpanda from v23.2.7 in hope that it will miraculously fix it but sadly it didn't

What should have happened instead?

I would like to understand why my cluster crash Also, I would like to know why seastar thinks he has only 2G memory while each Redpanda instance has 24G

How to reproduce the issue?

I don't have a reliable way to reproduce it, it seems to happen at a random times (until I could figure out what is the issue here)

Additional information

This is the redpanda.yml:

redpanda:
    data_directory: /var/lib/redpanda/data
    empty_seed_starts_cluster: false
    seed_servers:
        - host:
            address: redpanda-0.redpanda.infra.svc.cluster.local.
            port: 33145
    rpc_server:
        address: 0.0.0.0
        port: 33145
    kafka_api:
        - address: 0.0.0.0
          port: 9093
          name: internal
    kafka_api_tls:
        - name: internal
    admin:
        - address: 0.0.0.0
          port: 9644
          name: admin
    admin_api_tls:
        - {}
    advertised_rpc_api:
        address: redpanda-0.redpanda.infra.svc.cluster.local.
        port: 33145
    advertised_kafka_api:
        - address: redpanda-0.redpanda.infra.svc.cluster.local.
          port: 9093
          name: internal
    auto_create_topics_enabled: false
    compacted_log_segment_size: 67108864
    default_topic_partitions: 60
    default_topic_replications: 3
    delete_retention_ms: 21600000
    enable_rack_awareness: false
    enable_sasl: false
    group_offset_retention_sec: 21600
    group_topic_partitions: 16
    kafka_batch_max_bytes: 1048576
    kafka_connection_rate_limit: 1000
    log_cleanup_policy: delete
    log_segment_size: 134217728
    log_segment_size_max: 268435456
    log_segment_size_min: 16777216
    max_compacted_log_segment_size: 536870912
    node_status_interval: 200
    retention_local_target_bytes_default: 102005473280
    rpc_server_tls:
        enabled: false
    storage_min_free_bytes: 5368709120
    topic_partitions_per_shard: 1000
rpk:
    enable_memory_locking: true
    tune_aio_events: true
pandaproxy:
    pandaproxy_api:
        - address: 0.0.0.0
          port: 8082
          name: internal
    pandaproxy_api_tls:
        - name: internal
    advertised_pandaproxy_api:
        - address: redpanda-0.redpanda.infra.svc.cluster.local.
          port: 8082
          name: internal
config_file: /etc/redpanda/redpanda.yaml

The command line to start redpanda:

rpk redpanda start --smp=12 --memory=23G --reserve-memory=220M --default-log-level=info --advertise-rpc-addr=$(SERVICE_NAME).redpanda.infra.svc.cluster.local.:33145 --advertise-kafka-addr=internal://$(SERVICE_NAME).redpanda.infra.svc.cluster.local.:9093

The init container command:

cp /tmp/base-config/redpanda.yaml /etc/redpanda/redpanda.yaml
chown 101:101 /etc/redpanda/redpanda.yaml
rpk redpanda tune all

JIRA Link: CORE-2403

michael-redpanda commented 5 months ago

Hi @DrorBuhnik thanks for the bug report. Can you please confirm which version of Redpanda the above stack trace was generated from? I tried decoding it for v23.3.10 and v23.2.7 but it didn't appear to be valid.

DrorBuhnik commented 5 months ago

yes, this is a stack trace from v23.3.10 this is the output of rpk version:

Version: v23.3.10
Git ref: https://github.com/redpanda-data/redpanda/commit/255020e51d9ed9b7f90a67872bfc81d62923e3e9
Build date: 2024-03-28T07:53:07Z
OS/Arch: linux/arm64
Go version: go1.21.3

Redpanda Cluster
node-0 v23.3.10 - https://github.com/redpanda-data/redpanda/commit/255020e51d9ed9b7f90a67872bfc81d62923e3e9
node-1 v23.3.10 - https://github.com/redpanda-data/redpanda/commit/255020e51d9ed9b7f90a67872bfc81d62923e3e9
node-2 v23.3.10 - https://github.com/redpanda-data/redpanda/commit/255020e51d9ed9b7f90a67872bfc81d62923e3e9

I'm using docker image redpandadata/redpanda:v23.3.10

dotnwat commented 5 months ago

Also, I would like to know why seastar thinks he has only 2G memory while each Redpanda instance has 24G

@DrorBuhnik the 2G number is per-core. So I'm guessing that you have 12 cores in this machine?

dotnwat commented 5 months ago 
2024-04-17 13:57:35.089 size: 516006364 count: 172 at:
[Backtrace #0]
void seastar::backtrace<seastar::current_backtrace_tasklocal()::$_0>(seastar::current_backtrace_tasklocal()::$_0&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/util/backtrace.hh:64
 (inlined by) seastar::current_backtrace_tasklocal() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/util/backtrace.cc:98
seastar::memory::get_backtrace() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:929
 (inlined by) seastar::memory::get_allocation_site() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:939
 (inlined by) seastar::memory::cpu_pages::add_alloc_site(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:845
 (inlined by) void* seastar::memory::allocate_from_sampled_small_pool<(seastar::memory::alignment_t)1>(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1580
 (inlined by) seastar::memory::allocate_slowpath(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1638
seastar::memory::allocate(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1658
 (inlined by) operator new(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:2355
void* std::__1::__libcpp_operator_new[abi:v160004]<unsigned long>(unsigned long) at /vectorized/llvm/bin/../include/c++/v1/new:266
 (inlined by) std::__1::__libcpp_allocate[abi:v160004](unsigned long, unsigned long) at /vectorized/llvm/bin/../include/c++/v1/new:292
 (inlined by) std::__1::allocator<unsigned int>::allocate[abi:v160004](unsigned long) at /vectorized/llvm/bin/../include/c++/v1/__memory/allocator.h:115
 (inlined by) std::__1::__allocation_result<std::__1::allocator_traits<std::__1::allocator<unsigned int>>::pointer> std::__1::__allocate_at_least[abi:v160004]<std::__1::allocator<unsigned int>>(std::__1::allocator<unsigned int>&, unsigned long) at /vectorized/llvm/bin/../include/c++/v1/__memory/allocate_at_least.h:55
 (inlined by) std::__1::__split_buffer<unsigned int, std::__1::allocator<unsigned int>&>::__split_buffer(unsigned long, unsigned long, std::__1::allocator<unsigned int>&) at /vectorized/llvm/bin/../include/c++/v1/__split_buffer:323
 (inlined by) std::__1::vector<unsigned int, std::__1::allocator<unsigned int>>::reserve(unsigned long) at /vectorized/llvm/bin/../include/c++/v1/vector:1503
 (inlined by) fragmented_vector<unsigned int, 8192ul>::add_capacity() at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ee3cd99e5b23244c-1/redpanda/redpanda/src/v/utils/fragmented_vector.h:508
 (inlined by) fragmented_vector<unsigned int, 8192ul>::maybe_add_capacity() at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ee3cd99e5b23244c-1/redpanda/redpanda/src/v/utils/fragmented_vector.h:486
 (inlined by) void fragmented_vector<unsigned int, 8192ul>::push_back<unsigned int&>(unsigned int&) at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ee3cd99e5b23244c-1/redpanda/redpanda/src/v/utils/fragmented_vector.h:159
 (inlined by) storage::index_state::add_entry(unsigned int, storage::offset_time_index, unsigned long) at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ee3cd99e5b23244c-1/redpanda/redpanda/src/v/storage/index_state.h:157
storage::index_state::maybe_index(unsigned long, unsigned long, unsigned long, detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>, detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>, model::timestamp, model::timestamp, std::__1::optional<model::timestamp>, bool, unsigned long) at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ee3cd99e5b23244c-1/redpanda/redpanda/src/v/storage/index_state.cc:109
storage::internal::copy_data_segment_reducer::filter_and_append(model::compression, model::record_batch) at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ee3cd99e5b23244c-1/redpanda/redpanda/src/v/storage/compaction_reducers.cc:317
storage::internal::copy_data_segment_reducer::operator()(model::record_batch) at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ee3cd99e5b23244c-1/redpanda/redpanda/src/v/storage/compaction_reducers.cc:348
auto model::record_batch_reader::impl::do_consume<storage::internal::copy_data_segment_reducer>(storage::internal::copy_data_segment_reducer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::'lambda'(storage::internal::copy_data_segment_reducer&)::operator()(storage::internal::copy_data_segment_reducer&) const at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ee3cd99e5b23244c-1/redpanda/redpanda/src/v/model/record_batch_reader.h:142
 (inlined by) auto model::record_batch_reader::impl::do_action<storage::internal::copy_data_segment_reducer, auto model::record_batch_reader::impl::do_consume<storage::internal::copy_data_segment_reducer>(storage::internal::copy_data_segment_reducer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::'lambda'(storage::internal::copy_data_segment_reducer&)>(storage::internal::copy_data_segment_reducer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>, auto model::record_batch_reader::impl::do_consume<storage::internal::copy_data_segment_reducer>(storage::internal::copy_data_segment_reducer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::'lambda'(storage::internal::copy_data_segment_reducer&)&&)::'lambda'()::operator()() const at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ee3cd99e5b23244c-1/redpanda/redpanda/src/v/model/record_batch_reader.h:155
 (inlined by) seastar::future<seastar::bool_class<seastar::stop_iteration_tag>> seastar::futurize<seastar::future<seastar::bool_class<seastar::stop_iteration_tag>>>::invoke<auto model::record_batch_reader::impl::do_action<storage::internal::copy_data_segment_reducer, auto model::record_batch_reader::impl::do_consume<storage::internal::copy_data_segment_reducer>(storage::internal::copy_data_segment_reducer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::'lambda'(storage::internal::copy_data_segment_reducer&)>(storage::internal::copy_data_segment_reducer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>, auto model::record_batch_reader::impl::do_consume<storage::internal::copy_data_segment_reducer>(storage::internal::copy_data_segment_reducer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::'lambda'(storage::internal::copy_data_segment_reducer&)&&)::'lambda'()&>(storage::internal::copy_data_segment_reducer&&) at /vectorized/include/seastar/core/future.hh:2006
 (inlined by) auto seastar::futurize_invoke<auto model::record_batch_reader::impl::do_action<storage::internal::copy_data_segment_reducer, auto model::record_batch_reader::impl::do_consume<storage::internal::copy_data_segment_reducer>(storage::internal::copy_data_segment_reducer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::'lambda'(storage::internal::copy_data_segment_reducer&)>(storage::internal::copy_data_segment_reducer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>, auto model::record_batch_reader::impl::do_consume<storage::internal::copy_data_segment_reducer>(storage::internal::copy_data_segment_reducer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::'lambda'(storage::internal::copy_data_segment_reducer&)&&)::'lambda'()&>(storage::internal::copy_data_segment_reducer&&) at /vectorized/include/seastar/core/future.hh:2037
 (inlined by) seastar::internal::repeater<auto model::record_batch_reader::impl::do_action<storage::internal::copy_data_segment_reducer, auto model::record_batch_reader::impl::do_consume<storage::internal::copy_data_segment_reducer>(storage::internal::copy_data_segment_reducer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::'lambda'(storage::internal::copy_data_segment_reducer&)>(storage::internal::copy_data_segment_reducer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>, auto model::record_batch_reader::impl::do_consume<storage::internal::copy_data_segment_reducer>(storage::internal::copy_data_segment_reducer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::'lambda'(storage::internal::copy_data_segment_reducer&)&&)::'lambda'()>::run_and_dispose() at /vectorized/include/seastar/core/loop.hh:79
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2750
 (inlined by) seastar::reactor::run_some_tasks() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3213
seastar::reactor::do_run() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3397
seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2::operator()() const at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:4624
 (inlined by) decltype(std::declval<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2&>()()) std::__1::__invoke[abi:v160004]<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2&>(seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2&) at /vectorized/llvm/bin/../include/c++/v1/__functional/invoke.h:394
 (inlined by) void std::__1::__invoke_void_return_wrapper<void, true>::__call<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2&>(seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2&) at /vectorized/llvm/bin/../include/c++/v1/__functional/invoke.h:487
 (inlined by) std::__1::__function::__alloc_func<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2, std::__1::allocator<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2>, void ()>::operator()[abi:v160004]() at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:185
 (inlined by) std::__1::__function::__func<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2, std::__1::allocator<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2>, void ()>::operator()() at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:356
std::__1::__function::__value_func<void ()>::operator()[abi:v160004]() const at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:510
 (inlined by) std::__1::function<void ()>::operator()() const at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:1156
 (inlined by) seastar::posix_thread::start_routine(void*) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/posix.cc:90
LLVMSymbolizer: error reading file: No such file or directory
?? at ??:0
?? at ??:0

2024-04-17 13:57:35.089 size: 441005439 count: 147 at:
[Backtrace #1]
void seastar::backtrace<seastar::current_backtrace_tasklocal()::$_0>(seastar::current_backtrace_tasklocal()::$_0&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/util/backtrace.hh:64
 (inlined by) seastar::current_backtrace_tasklocal() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/util/backtrace.cc:98
seastar::memory::get_backtrace() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:929
 (inlined by) seastar::memory::get_allocation_site() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:939
 (inlined by) seastar::memory::cpu_pages::add_alloc_site(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:845
 (inlined by) void* seastar::memory::allocate_from_sampled_small_pool<(seastar::memory::alignment_t)1>(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1580
 (inlined by) seastar::memory::allocate_slowpath(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1638
seastar::memory::allocate(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1658
 (inlined by) operator new(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:2355
void* std::__1::__libcpp_operator_new[abi:v160004]<unsigned long>(unsigned long) at /vectorized/llvm/bin/../include/c++/v1/new:266
 (inlined by) std::__1::__libcpp_allocate[abi:v160004](unsigned long, unsigned long) at /vectorized/llvm/bin/../include/c++/v1/new:292
 (inlined by) std::__1::allocator<unsigned int>::allocate[abi:v160004](unsigned long) at /vectorized/llvm/bin/../include/c++/v1/__memory/allocator.h:115
 (inlined by) std::__1::__allocation_result<std::__1::allocator_traits<std::__1::allocator<unsigned int>>::pointer> std::__1::__allocate_at_least[abi:v160004]<std::__1::allocator<unsigned int>>(std::__1::allocator<unsigned int>&, unsigned long) at /vectorized/llvm/bin/../include/c++/v1/__memory/allocate_at_least.h:55
 (inlined by) std::__1::__split_buffer<unsigned int, std::__1::allocator<unsigned int>&>::__split_buffer(unsigned long, unsigned long, std::__1::allocator<unsigned int>&) at /vectorized/llvm/bin/../include/c++/v1/__split_buffer:323
 (inlined by) std::__1::vector<unsigned int, std::__1::allocator<unsigned int>>::reserve(unsigned long) at /vectorized/llvm/bin/../include/c++/v1/vector:1503
 (inlined by) fragmented_vector<unsigned int, 8192ul>::add_capacity() at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ee3cd99e5b23244c-1/redpanda/redpanda/src/v/utils/fragmented_vector.h:508
 (inlined by) fragmented_vector<unsigned int, 8192ul>::maybe_add_capacity() at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ee3cd99e5b23244c-1/redpanda/redpanda/src/v/utils/fragmented_vector.h:486
 (inlined by) void fragmented_vector<unsigned int, 8192ul>::push_back<unsigned int>(unsigned int&&) at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ee3cd99e5b23244c-1/redpanda/redpanda/src/v/utils/fragmented_vector.h:159
 (inlined by) storage::index_state::add_entry(unsigned int, storage::offset_time_index, unsigned long) at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ee3cd99e5b23244c-1/redpanda/redpanda/src/v/storage/index_state.h:158
storage::index_state::maybe_index(unsigned long, unsigned long, unsigned long, detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>, detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>, model::timestamp, model::timestamp, std::__1::optional<model::timestamp>, bool, unsigned long) at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ee3cd99e5b23244c-1/redpanda/redpanda/src/v/storage/index_state.cc:109
storage::internal::copy_data_segment_reducer::filter_and_append(model::compression, model::record_batch) at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ee3cd99e5b23244c-1/redpanda/redpanda/src/v/storage/compaction_reducers.cc:317
storage::internal::copy_data_segment_reducer::operator()(model::record_batch) at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ee3cd99e5b23244c-1/redpanda/redpanda/src/v/storage/compaction_reducers.cc:348
auto model::record_batch_reader::impl::do_consume<storage::internal::copy_data_segment_reducer>(storage::internal::copy_data_segment_reducer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::'lambda'(storage::internal::copy_data_segment_reducer&)::operator()(storage::internal::copy_data_segment_reducer&) const at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ee3cd99e5b23244c-1/redpanda/redpanda/src/v/model/record_batch_reader.h:142
 (inlined by) auto model::record_batch_reader::impl::do_action<storage::internal::copy_data_segment_reducer, auto model::record_batch_reader::impl::do_consume<storage::internal::copy_data_segment_reducer>(storage::internal::copy_data_segment_reducer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::'lambda'(storage::internal::copy_data_segment_reducer&)>(storage::internal::copy_data_segment_reducer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>, auto model::record_batch_reader::impl::do_consume<storage::internal::copy_data_segment_reducer>(storage::internal::copy_data_segment_reducer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::'lambda'(storage::internal::copy_data_segment_reducer&)&&)::'lambda'()::operator()() const at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ee3cd99e5b23244c-1/redpanda/redpanda/src/v/model/record_batch_reader.h:155
 (inlined by) seastar::future<seastar::bool_class<seastar::stop_iteration_tag>> seastar::futurize<seastar::future<seastar::bool_class<seastar::stop_iteration_tag>>>::invoke<auto model::record_batch_reader::impl::do_action<storage::internal::copy_data_segment_reducer, auto model::record_batch_reader::impl::do_consume<storage::internal::copy_data_segment_reducer>(storage::internal::copy_data_segment_reducer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::'lambda'(storage::internal::copy_data_segment_reducer&)>(storage::internal::copy_data_segment_reducer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>, auto model::record_batch_reader::impl::do_consume<storage::internal::copy_data_segment_reducer>(storage::internal::copy_data_segment_reducer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::'lambda'(storage::internal::copy_data_segment_reducer&)&&)::'lambda'()&>(storage::internal::copy_data_segment_reducer&&) at /vectorized/include/seastar/core/future.hh:2006
 (inlined by) auto seastar::futurize_invoke<auto model::record_batch_reader::impl::do_action<storage::internal::copy_data_segment_reducer, auto model::record_batch_reader::impl::do_consume<storage::internal::copy_data_segment_reducer>(storage::internal::copy_data_segment_reducer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::'lambda'(storage::internal::copy_data_segment_reducer&)>(storage::internal::copy_data_segment_reducer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>, auto model::record_batch_reader::impl::do_consume<storage::internal::copy_data_segment_reducer>(storage::internal::copy_data_segment_reducer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::'lambda'(storage::internal::copy_data_segment_reducer&)&&)::'lambda'()&>(storage::internal::copy_data_segment_reducer&&) at /vectorized/include/seastar/core/future.hh:2037
 (inlined by) seastar::internal::repeater<auto model::record_batch_reader::impl::do_action<storage::internal::copy_data_segment_reducer, auto model::record_batch_reader::impl::do_consume<storage::internal::copy_data_segment_reducer>(storage::internal::copy_data_segment_reducer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::'lambda'(storage::internal::copy_data_segment_reducer&)>(storage::internal::copy_data_segment_reducer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>, auto model::record_batch_reader::impl::do_consume<storage::internal::copy_data_segment_reducer>(storage::internal::copy_data_segment_reducer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::'lambda'(storage::internal::copy_data_segment_reducer&)&&)::'lambda'()>::run_and_dispose() at /vectorized/include/seastar/core/loop.hh:79
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2750
 (inlined by) seastar::reactor::run_some_tasks() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3213
seastar::reactor::do_run() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3397
seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2::operator()() const at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:4624
 (inlined by) decltype(std::declval<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2&>()()) std::__1::__invoke[abi:v160004]<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2&>(seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2&) at /vectorized/llvm/bin/../include/c++/v1/__functional/invoke.h:394
 (inlined by) void std::__1::__invoke_void_return_wrapper<void, true>::__call<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2&>(seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2&) at /vectorized/llvm/bin/../include/c++/v1/__functional/invoke.h:487
 (inlined by) std::__1::__function::__alloc_func<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2, std::__1::allocator<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2>, void ()>::operator()[abi:v160004]() at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:185
 (inlined by) std::__1::__function::__func<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2, std::__1::allocator<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2>, void ()>::operator()() at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:356
std::__1::__function::__value_func<void ()>::operator()[abi:v160004]() const at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:510
 (inlined by) std::__1::function<void ()>::operator()() const at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:1156
 (inlined by) seastar::posix_thread::start_routine(void*) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/posix.cc:90
?? at ??:0
?? at ??:0

2024-04-17 13:57:35.089 size: 432005328 count: 144 at:
[Backtrace #2]
void seastar::backtrace<seastar::current_backtrace_tasklocal()::$_0>(seastar::current_backtrace_tasklocal()::$_0&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/util/backtrace.hh:64
 (inlined by) seastar::current_backtrace_tasklocal() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/util/backtrace.cc:98
seastar::memory::get_backtrace() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:929
 (inlined by) seastar::memory::get_allocation_site() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:939
 (inlined by) seastar::memory::cpu_pages::add_alloc_site(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:845
 (inlined by) void* seastar::memory::allocate_from_sampled_small_pool<(seastar::memory::alignment_t)1>(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1580
 (inlined by) seastar::memory::allocate_slowpath(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1638
seastar::memory::allocate(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1658
 (inlined by) operator new(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:2355
void* std::__1::__libcpp_operator_new[abi:v160004]<unsigned long>(unsigned long) at /vectorized/llvm/bin/../include/c++/v1/new:266
 (inlined by) std::__1::__libcpp_allocate[abi:v160004](unsigned long, unsigned long) at /vectorized/llvm/bin/../include/c++/v1/new:292
 (inlined by) std::__1::allocator<unsigned long>::allocate[abi:v160004](unsigned long) at /vectorized/llvm/bin/../include/c++/v1/__memory/allocator.h:115
 (inlined by) std::__1::__allocation_result<std::__1::allocator_traits<std::__1::allocator<unsigned long>>::pointer> std::__1::__allocate_at_least[abi:v160004]<std::__1::allocator<unsigned long>>(std::__1::allocator<unsigned long>&, unsigned long) at /vectorized/llvm/bin/../include/c++/v1/__memory/allocate_at_least.h:55
 (inlined by) std::__1::__split_buffer<unsigned long, std::__1::allocator<unsigned long>&>::__split_buffer(unsigned long, unsigned long, std::__1::allocator<unsigned long>&) at /vectorized/llvm/bin/../include/c++/v1/__split_buffer:323
 (inlined by) std::__1::vector<unsigned long, std::__1::allocator<unsigned long>>::reserve(unsigned long) at /vectorized/llvm/bin/../include/c++/v1/vector:1503
 (inlined by) fragmented_vector<unsigned long, 8192ul>::add_capacity() at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ee3cd99e5b23244c-1/redpanda/redpanda/src/v/utils/fragmented_vector.h:508
 (inlined by) fragmented_vector<unsigned long, 8192ul>::maybe_add_capacity() at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ee3cd99e5b23244c-1/redpanda/redpanda/src/v/utils/fragmented_vector.h:486
 (inlined by) void fragmented_vector<unsigned long, 8192ul>::push_back<unsigned long&>(unsigned long&) at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ee3cd99e5b23244c-1/redpanda/redpanda/src/v/utils/fragmented_vector.h:159
 (inlined by) storage::index_state::add_entry(unsigned int, storage::offset_time_index, unsigned long) at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ee3cd99e5b23244c-1/redpanda/redpanda/src/v/storage/index_state.h:159
storage::index_state::maybe_index(unsigned long, unsigned long, unsigned long, detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>, detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true>>, model::timestamp, model::timestamp, std::__1::optional<model::timestamp>, bool, unsigned long) at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ee3cd99e5b23244c-1/redpanda/redpanda/src/v/storage/index_state.cc:109
storage::internal::copy_data_segment_reducer::filter_and_append(model::compression, model::record_batch) at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ee3cd99e5b23244c-1/redpanda/redpanda/src/v/storage/compaction_reducers.cc:317
storage::internal::copy_data_segment_reducer::operator()(model::record_batch) at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ee3cd99e5b23244c-1/redpanda/redpanda/src/v/storage/compaction_reducers.cc:348
auto model::record_batch_reader::impl::do_consume<storage::internal::copy_data_segment_reducer>(storage::internal::copy_data_segment_reducer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::'lambda'(storage::internal::copy_data_segment_reducer&)::operator()(storage::internal::copy_data_segment_reducer&) const at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ee3cd99e5b23244c-1/redpanda/redpanda/src/v/model/record_batch_reader.h:142
 (inlined by) auto model::record_batch_reader::impl::do_action<storage::internal::copy_data_segment_reducer, auto model::record_batch_reader::impl::do_consume<storage::internal::copy_data_segment_reducer>(storage::internal::copy_data_segment_reducer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::'lambda'(storage::internal::copy_data_segment_reducer&)>(storage::internal::copy_data_segment_reducer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>, auto model::record_batch_reader::impl::do_consume<storage::internal::copy_data_segment_reducer>(storage::internal::copy_data_segment_reducer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::'lambda'(storage::internal::copy_data_segment_reducer&)&&)::'lambda'()::operator()() const at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ee3cd99e5b23244c-1/redpanda/redpanda/src/v/model/record_batch_reader.h:155
 (inlined by) seastar::future<seastar::bool_class<seastar::stop_iteration_tag>> seastar::futurize<seastar::future<seastar::bool_class<seastar::stop_iteration_tag>>>::invoke<auto model::record_batch_reader::impl::do_action<storage::internal::copy_data_segment_reducer, auto model::record_batch_reader::impl::do_consume<storage::internal::copy_data_segment_reducer>(storage::internal::copy_data_segment_reducer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::'lambda'(storage::internal::copy_data_segment_reducer&)>(storage::internal::copy_data_segment_reducer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>, auto model::record_batch_reader::impl::do_consume<storage::internal::copy_data_segment_reducer>(storage::internal::copy_data_segment_reducer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::'lambda'(storage::internal::copy_data_segment_reducer&)&&)::'lambda'()&>(storage::internal::copy_data_segment_reducer&&) at /vectorized/include/seastar/core/future.hh:2006
 (inlined by) auto seastar::futurize_invoke<auto model::record_batch_reader::impl::do_action<storage::internal::copy_data_segment_reducer, auto model::record_batch_reader::impl::do_consume<storage::internal::copy_data_segment_reducer>(storage::internal::copy_data_segment_reducer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::'lambda'(storage::internal::copy_data_segment_reducer&)>(storage::internal::copy_data_segment_reducer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>, auto model::record_batch_reader::impl::do_consume<storage::internal::copy_data_segment_reducer>(storage::internal::copy_data_segment_reducer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::'lambda'(storage::internal::copy_data_segment_reducer&)&&)::'lambda'()&>(storage::internal::copy_data_segment_reducer&&) at /vectorized/include/seastar/core/future.hh:2037
 (inlined by) seastar::internal::repeater<auto model::record_batch_reader::impl::do_action<storage::internal::copy_data_segment_reducer, auto model::record_batch_reader::impl::do_consume<storage::internal::copy_data_segment_reducer>(storage::internal::copy_data_segment_reducer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::'lambda'(storage::internal::copy_data_segment_reducer&)>(storage::internal::copy_data_segment_reducer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>, auto model::record_batch_reader::impl::do_consume<storage::internal::copy_data_segment_reducer>(storage::internal::copy_data_segment_reducer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::'lambda'(storage::internal::copy_data_segment_reducer&)&&)::'lambda'()>::run_and_dispose() at /vectorized/include/seastar/core/loop.hh:79
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2750
 (inlined by) seastar::reactor::run_some_tasks() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3213
seastar::reactor::do_run() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3397
seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2::operator()() const at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:4624
 (inlined by) decltype(std::declval<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2&>()()) std::__1::__invoke[abi:v160004]<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2&>(seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2&) at /vectorized/llvm/bin/../include/c++/v1/__functional/invoke.h:394
 (inlined by) void std::__1::__invoke_void_return_wrapper<void, true>::__call<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2&>(seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2&) at /vectorized/llvm/bin/../include/c++/v1/__functional/invoke.h:487
 (inlined by) std::__1::__function::__alloc_func<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2, std::__1::allocator<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2>, void ()>::operator()[abi:v160004]() at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:185
 (inlined by) std::__1::__function::__func<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2, std::__1::allocator<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2>, void ()>::operator()() at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:356
std::__1::__function::__value_func<void ()>::operator()[abi:v160004]() const at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:510
 (inlined by) std::__1::function<void ()>::operator()() const at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:1156
 (inlined by) seastar::posix_thread::start_routine(void*) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/posix.cc:90
?? at ??:0
?? at ??:0
DrorBuhnik commented 5 months ago

Also, I would like to know why seastar thinks he has only 2G memory while each Redpanda instance has 24G

@DrorBuhnik the 2G number is per-core. So I'm guessing that you have 12 cores in this machine?

ha! you are correct! that explains it

until we can pin point and fix the issue - do you think that adding more memory/reducing the number of cpu core would help mitigate/"fix" the issue?

michael-redpanda commented 5 months ago

Crash backtrace:

Backtrace:
[Backtrace #0]
void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::'lambda'(seastar::frame)>(seastar::backtrace_buffer::append_backtrace()::'lambda'(seastar::frame)&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/util/backtrace.hh:64
 (inlined by) seastar::backtrace_buffer::append_backtrace() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:824
 (inlined by) seastar::print_with_backtrace(seastar::backtrace_buffer&, bool) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:854
 (inlined by) seastar::print_with_backtrace(char const*, bool) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:866
seastar::sigabrt_action() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:4155
 (inlined by) void seastar::install_oneshot_signal_handler<6, (void (*)())(&seastar::sigabrt_action())>()::'lambda'(int, siginfo_t*, void*)::operator()(int, siginfo_t*, void*) const at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:4131
 (inlined by) void seastar::install_oneshot_signal_handler<6, (void (*)())(&seastar::sigabrt_action())>()::'lambda'(int, siginfo_t*, void*)::__invoke(int, siginfo_t*, void*) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:4127
LLVMSymbolizer: error reading file: No such file or directory
?? at ??:0
LLVMSymbolizer: error reading file: No such file or directory
?? at ??:0
?? at ??:0
?? at ??:0
seastar::memory::on_allocation_failure(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:2090
seastar::memory::finish_allocation(void*, unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1608
 (inlined by) seastar::memory::allocate_slowpath(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1647
seastar::memory::allocate(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1658
 (inlined by) operator new(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:2355
seastar::chunked_fifo<seastar::internal::abortable_fifo<seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>::entry, seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>::expiry_handler>::entry, 128ul>::back_chunk_new() at /vectorized/include/seastar/core/chunked_fifo.hh:414
 (inlined by) seastar::chunked_fifo<seastar::internal::abortable_fifo<seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>::entry, seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>::expiry_handler>::entry, 128ul>::ensure_room_back() at /vectorized/include/seastar/core/chunked_fifo.hh:435
 (inlined by) void seastar::chunked_fifo<seastar::internal::abortable_fifo<seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>::entry, seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>::expiry_handler>::entry, 128ul>::emplace_back<>() at /vectorized/include/seastar/core/chunked_fifo.hh:471
 (inlined by) seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>::entry& seastar::internal::abortable_fifo<seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>::entry, seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>::expiry_handler>::emplace_back<seastar::promise<void>, unsigned long&>(seastar::promise<void>&&, unsigned long&) at /vectorized/include/seastar/core/abortable_fifo.hh:214
 (inlined by) seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>::wait(std::__1::chrono::time_point<std::__1::chrono::steady_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>, unsigned long) at /vectorized/include/seastar/core/semaphore.hh:288
 (inlined by) seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>::wait(unsigned long) at /vectorized/include/seastar/core/semaphore.hh:263
 (inlined by) seastar::future<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>> seastar::get_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long) at /vectorized/include/seastar/core/semaphore.hh:580
seastar::futurize<std::__1::invoke_result<storage::disk_log_impl::disk_usage_and_reclaimable_space(storage::gc_config)::$_5::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()>::type>::type seastar::with_semaphore<seastar::semaphore_default_exception_factory, storage::disk_log_impl::disk_usage_and_reclaimable_space(storage::gc_config)::$_5::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'(), std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, storage::disk_log_impl::disk_usage_and_reclaimable_space(storage::gc_config)::$_5::operator()(seastar::lw_shared_ptr<storage::segment> const&) const::'lambda'()&&) at /vectorized/include/seastar/core/semaphore.hh:726
 (inlined by) storage::disk_log_impl::disk_usage_and_reclaimable_space(storage::gc_config)::$_5::operator()(seastar::lw_shared_ptr<storage::segment> const&) const at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ee3cd99e5b23244c-1/redpanda/redpanda/src/v/storage/disk_log_impl.cc:2540
 (inlined by) seastar::future<storage::usage> seastar::futurize<seastar::future<storage::usage>>::invoke<storage::disk_log_impl::disk_usage_and_reclaimable_space(storage::gc_config)::$_5&, seastar::lw_shared_ptr<storage::segment>&>(storage::disk_log_impl::disk_usage_and_reclaimable_space(storage::gc_config)::$_5&, seastar::lw_shared_ptr<storage::segment>&) at /vectorized/include/seastar/core/future.hh:2006
 (inlined by) auto seastar::futurize_invoke<storage::disk_log_impl::disk_usage_and_reclaimable_space(storage::gc_config)::$_5&, seastar::lw_shared_ptr<storage::segment>&>(storage::disk_log_impl::disk_usage_and_reclaimable_space(storage::gc_config)::$_5&, seastar::lw_shared_ptr<storage::segment>&) at /vectorized/include/seastar/core/future.hh:2037
 (inlined by) seastar::future<storage::usage> seastar::map_reduce<fragmented_vector<seastar::lw_shared_ptr<storage::segment>, 8192ul>::iter<false>, storage::disk_log_impl::disk_usage_and_reclaimable_space(storage::gc_config)::$_5, storage::usage, storage::disk_log_impl::disk_usage_and_reclaimable_space(storage::gc_config)::$_6>(fragmented_vector<seastar::lw_shared_ptr<storage::segment>, 8192ul>::iter<false>, fragmented_vector<seastar::lw_shared_ptr<storage::segment>, 8192ul>::iter<false>, storage::disk_log_impl::disk_usage_and_reclaimable_space(storage::gc_config)::$_5&&, storage::usage, storage::disk_log_impl::disk_usage_and_reclaimable_space(storage::gc_config)::$_6) at /vectorized/include/seastar/core/map_reduce.hh:192
 (inlined by) seastar::future<storage::usage> seastar::map_reduce<fragmented_vector<seastar::lw_shared_ptr<storage::segment>, 8192ul>&, storage::disk_log_impl::disk_usage_and_reclaimable_space(storage::gc_config)::$_5, storage::usage, storage::disk_log_impl::disk_usage_and_reclaimable_space(storage::gc_config)::$_6>(fragmented_vector<seastar::lw_shared_ptr<storage::segment>, 8192ul>&, storage::disk_log_impl::disk_usage_and_reclaimable_space(storage::gc_config)::$_5&&, storage::usage, storage::disk_log_impl::disk_usage_and_reclaimable_space(storage::gc_config)::$_6) at /vectorized/include/seastar/core/map_reduce.hh:262
 (inlined by) storage::disk_log_impl::disk_usage_and_reclaimable_space(storage::gc_config) (.resume) at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ee3cd99e5b23244c-1/redpanda/redpanda/src/v/storage/disk_log_impl.cc:2537
std::__1::coroutine_handle<seastar::internal::coroutine_traits_base<std::__1::pair<storage::usage, storage::reclaim_size_limits>>::promise_type>::resume[abi:v160004]() const at /vectorized/llvm/bin/../include/c++/v1/__coroutine/coroutine_handle.h:169
 (inlined by) seastar::internal::coroutine_traits_base<std::__1::pair<storage::usage, storage::reclaim_size_limits>>::promise_type::run_and_dispose() at /vectorized/include/seastar/core/coroutine.hh:83
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2750
 (inlined by) seastar::reactor::run_some_tasks() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3213
seastar::reactor::do_run() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3397
seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2::operator()() const at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:4624
 (inlined by) decltype(std::declval<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2&>()()) std::__1::__invoke[abi:v160004]<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2&>(seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2&) at /vectorized/llvm/bin/../include/c++/v1/__functional/invoke.h:394
 (inlined by) void std::__1::__invoke_void_return_wrapper<void, true>::__call<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2&>(seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2&) at /vectorized/llvm/bin/../include/c++/v1/__functional/invoke.h:487
 (inlined by) std::__1::__function::__alloc_func<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2, std::__1::allocator<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2>, void ()>::operator()[abi:v160004]() at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:185
 (inlined by) std::__1::__function::__func<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2, std::__1::allocator<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2>, void ()>::operator()() at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:356
std::__1::__function::__value_func<void ()>::operator()[abi:v160004]() const at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:510
 (inlined by) std::__1::function<void ()>::operator()() const at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:1156
 (inlined by) seastar::posix_thread::start_routine(void*) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/posix.cc:90
?? at ??:0
?? at ??:0
dotnwat commented 5 months ago

until we can pin point and fix the issue - do you think that adding more memory/reducing the number of cpu core would help mitigate/"fix" the issue?

Yes, probably. 2GB/core is the minimum recommended amount. I think in practice it is usually quite a bit more. The lower it is, the more likely we hit these allocation issues which are hard to predict/discover.

DrorBuhnik commented 4 months ago

Great! I will adjust my cluster resources to bump the cpu-memory ratio Thank you!

github-actions[bot] commented 1 month ago

This issue hasn't seen activity in 3 months. If you want to keep it open, post a comment or remove the stale label – otherwise this will be closed in two weeks.

DrorBuhnik commented 1 month ago

A small update - bumping the memory/cpu-core ration (I reached 84GB/6cpu => 14GB per core) helped a bit, the rate of the crashes but didn't solved it We are still running it but seems like that the "root cause" was cpu starvation. now we are running with 48GB/12cpu and it seems much better, so far zero crash in 5 days :crossed_fingers: