search

redpanda-data / redpanda

Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
https://redpanda.com
9.46k stars 579 forks source link

Seastar Failed to allocate 2162672 bytes | RP v23.2.8 | Broker crash/restart #19840

Closed tj-redpanda closed 2 days ago

tj-redpanda commented 3 months ago

Version & Environment

Redpanda version: (use rpk version):

Version: v23.2.8 (rev 2f0de10)

Other info: 3 Broker Cluster

What went wrong?

Redpanda crashed/stopped, restarted which interrupted clients ERROR 2024-06-12 18:00:34,223 [shard 6] seastar_memory - Dumping seastar memory diagnostics ERROR 2024-06-12 18:00:34,231 [shard 6] seastar - Failed to allocate 2162672 bytes

What should have happened instead?

Requested memory be allocated, if appropriate

How to reproduce the issue?

No Reproduction, It's only occurred once

Additional information


INFO  2024-06-12 18:00:34,054 [shard  6] raft - [group_id:957, {kafka/tradewinds_prod_pcap_events/0}] consensus.cc:1932 - Truncating log in term: 2, Request previous log index: 4727936937 is earlier than log end offset: 4727942657. Truncating to: 4727936938
INFO  2024-06-12 18:00:34,054 [shard  6] offset_translator - ntp: {kafka/tradewinds_prod_pcap_events/0} - offset_translator.cc:243 - truncate at offset: 4727936938, new state: {base offset/delta: {3365809312}/1, map size: 1, last delta: 1}
INFO  2024-06-12 18:00:34,181 [shard  6] storage - segment.cc:759 - Creating new segment /var/lib/redpanda/data/kafka/tradewinds_prod_pcap_events/0_208719/4727936938-2-v1.log
ERROR 2024-06-12 18:00:34,223 [shard  6] seastar_memory - Dumping seastar memory diagnostics
Used memory:   983M
Free memory:   4627M
Total memory:  5G
Hard failures: 1
Top-N alloc sites - size count stack:
630007770 210 0x708b59a 0x6d244d1 0x6d0d447 0x6cffa8e 0x23af1d1 0x23af9fc 0x6dd3e6f 0x6dd7511 0x6e307d5 0x6d61d3f /opt/redpanda/lib/libc.so.6+0x91016 /opt/redpanda/lib/libc.so.6+0x1166cf
48000592 16 0x708b59a 0x6d0c185 0x6d18900 0x64fc0cb 0x64ff57d 0x631da7b 0x631f772 0x29319d9 0x2c3c703 0x6dd3e6f 0x6dd7511 0x6e307d5 0x6d61d3f /opt/redpanda/lib/libc.so.6+0x91016 /opt/redpanda/lib/libc.so.6+0x1166cf
33554432 2 0x708b59a 0x6d03656 0x6d0dce6 0x6cffa8e 0x6742fd7 0x6743d26 0x67428b3 0x21d4613 0x6dd3e6f 0x6dd7511 0x6e307d5 0x6d61d3f /opt/redpanda/lib/libc.so.6+0x91016 /opt/redpanda/lib/libc.so.6+0x1166cf
16777216 1 0x708b59a 0x6d03656 0x6d0dce6 0x6cffa8e 0x673d81c 0x21d4600 0x6dd3e6f 0x6dd7511 0x6e307d5 0x6d61d3f /opt/redpanda/lib/libc.so.6+0x91016 /opt/redpanda/lib/libc.so.6+0x1166cf
15000185 5 0x708b59a 0x6d0b643 0x6d18900 0x64f034c 0x64e7197 0x64b2bb3 0x647025d 0x6471aff 0x63a8ac3 0x298f9d8 0x299019d 0x6dd3e6f 0x6dd7511 0x6e307d5 0x6d61d3f /opt/redpanda/lib/libc.so.6+0x91016 /opt/redpanda/lib/libc.so.6+0x1166cf
12000148 4 0x708b59a 0x6d0b643 0x6d18900 0x64b3bb9 0x647b192 0x6dd3e6f 0x6dd7511 0x6e307d5 0x6d61d3f /opt/redpanda/lib/libc.so.6+0x91016 /opt/redpanda/lib/libc.so.6+0x1166cf
12000148 4 0x708b59a 0x6d0b643 0x6d18900 0x64f034c 0x64e7197 0x64b2bb3 0x648e5cb 0x6dd3e6f 0x6dd7511 0x6e307d5 0x6d61d3f /opt/redpanda/lib/libc.so.6+0x91016 /opt/redpanda/lib/libc.so.6+0x1166cf
12000148 4 0x708b59a 0x6d0b643 0x6d18900 0x64efd19 0x64e7197 0x64b2bb3 0x648e5cb 0x6dd3e6f 0x6dd7511 0x6e307d5 0x6d61d3f /opt/redpanda/lib/libc.so.6+0x91016 /opt/redpanda/lib/libc.so.6+0x1166cf
12000148 4 0x708b59a 0x6d0b643 0x6d18900 0x64effe4 0x64e7197 0x64b2bb3 0x648e5cb 0x6dd3e6f 0x6dd7511 0x6e307d5 0x6d61d3f /opt/redpanda/lib/libc.so.6+0x91016 /opt/redpanda/lib/libc.so.6+0x1166cf
12000148 4 0x708b59a 0x6d0b643 0x6d171d1 0x3faa4bc 0x35f88fe 0x35f61b6 0x35dc1bb 0x35d7328 0x324c8f1 0x329891d 0x32783fe 0x327794f 0x327016e 0x326b011 0x328bfba 0x229854a 0x6dd3e6f 0x6dd7511 0x6e307d5 0x6d61d3f /opt/redpanda/lib/libc.so.6+0x91016 /opt/redpanda/lib/libc.so.6+0x1166cf
If you work at Redpanda please refer to https://vectorizedio.atlassian.net/l/cp/iuEMd2NN
Small pools:
objsz spansz usedobj memory unused wst%
    8     4K     842   108K   101K   93
   10     4K      57     8K     7K   93
   12     4K      54    16K    15K   96
   14     4K      82     8K     7K   85
   16     4K      3k   124K    83K   66
   32     4K      8k   348K   109K   31
   32     4K     78k     2M     5K    0
   32     4K      6k   448K   265K   59
   32     4K     14k     4M  3998K   90
   48     4K     47k     2M     3K    0
   48     4K     51k     9M  6510K   73
   64     4K     18k     2M  1006K   47
   64     4K    105k     7M  1091K   14
   80     4K     57k     4M     8K    0
   96     4K     41k     5M   981K   20
  112     4K     356   432K   393K   90
  128     4K      9k     3M     2M   63
  160     4K     16k     3M   145K    5
  192     4K      5k     1M   170K   16
  224     4K      8k     2M   131K    7
  256     4K     655   688K   524K   76
  320     8K     771   888K   647K   72
  384     8K     726   880K   608K   69
  448     4K      3k     2M   691K   35
  512     4K     536   764K   496K   64
  640    16K     826     1M   684K   56
  768    16K     44k    32M     9K    0
  896     8K      2k     3M  1103K   35
 1024     4K     171   760K   589K   77
 1280    32K     518     2M  1528K   70
 1536    32K     249     1M   746K   66
 1792    16K     142     2M     1M   84
 2048     8K     134   832K   564K   67
 2560    64K     136     2M  1963K   85
 3072    64K     212     3M     3M   80
 3584    32K      1k     6M     3M   43
 4096    16K     151     2M     1M   65
 5120   128K     261    10M     9M   87
 6144   128K      97     4M     4M   86
 7168    64K     526    10M     6M   63
 8192    32K     10k    90M     8M    9
10240    64K      22     3M     2M   91
12288    64K     360     7M     3M   41
14336   128K       7     4M     4M   97
16384    64K     36k   561M     3M    0
Page spans:
index  size  free  used spans
    0    4K   25M   50M   19k
    1    8K   49M    6M    7k
    2   16K   71M   37M    7k
    3   32K  301M  148M   14k
    4   64K  576M  595M   19k
    5  128K  945M   22M    8k
    6  256K 1179M    1M    5k
    7  512K 1047M  512K    2k
    8    1M  414M    0B   414
    9    2M   20M    4M    12
   10    4M    0B    8M     2
   11    8M    0B    0B     0
   12   16M    0B   48M     3
   13   32M    0B    0B     0
   14   64M    0B   64M     1
   15  128M    0B    0B     0
   16  256M    0B    0B     0
   17  512M    0B    0B     0
   18    1G    0B    0B     0
   19    2G    0B    0B     0
   20    4G    0B    0B     0
   21    8G    0B    0B     0
   22   16G    0B    0B     0
   23   32G    0B    0B     0
   24   64G    0B    0B     0
   25  128G    0B    0B     0
   26  256G    0B    0B     0
   27  512G    0B    0B     0
   28    1T    0B    0B     0
   29    2T    0B    0B     0
   30    4T    0B    0B     0
   31    8T    0B    0B     0
ERROR 2024-06-12 18:00:34,231 [shard  6] seastar - Failed to allocate 2162672 bytes
Aborting on shard 6.
Backtrace:
  0x6ddb7a3
  0x6e2eccb
  /opt/redpanda/lib/libc.so.6+0x42abf
  /opt/redpanda/lib/libc.so.6+0x92e3b
  /opt/redpanda/lib/libc.so.6+0x42a15
  /opt/redpanda/lib/libc.so.6+0x2c82e
  0x6d0e5ca
  0x6d0ca6c
  0x6d18900
  0x329601e
  0x3295f54
  0x327db3f
  0x327a135
  0x32787fa
  0x327794f
  0x327016e
  0x326b011
  0x328bfba
  0x229854a
  0x6dd3e6f
  0x6dd7511
  0x6e307d5
  0x6d61d3f
  /opt/redpanda/lib/libc.so.6+0x91016
  /opt/redpanda/lib/libc.so.6+0x1166cf
redpanda.service: Main process exited, code=killed, status=6/ABRT
redpanda.service: Failed with result 'signal'.
redpanda.service: Scheduled restart job, restart counter is at 1.
Stopped Redpanda, the fastest queue in the West..
Starting Redpanda, the fastest queue in the West....
System check - STARTED
System check - PASSED
We'd love to hear about your experience with Redpanda:
https://redpanda.com/feedback
Starting redpanda...
Running:
/opt/redpanda/bin/redpanda redpanda --redpanda-cfg /etc/redpanda/redpanda.yaml --lock-memory=false --io-properties-file=/etc/redpanda/io-config.yaml```

JIRA Link: [CORE-4193](https://redpandadata.atlassian.net/browse/CORE-4193)
github-actions[bot] commented 2 weeks ago

This issue hasn't seen activity in 3 months. If you want to keep it open, post a comment or remove the stale label – otherwise this will be closed in two weeks.

github-actions[bot] commented 2 days ago

This issue was closed due to lack of activity. Feel free to reopen if it's still relevant.