Oversized allocation

[weeco]

Version & Environment

Redpanda version: (use rpk version):

Version: v24.1.6 Git ref: 5e880f6fd1 Build date: 2024-05-31T17:33:43Z OS/Arch: linux/arm64 Go version: go1.22.2

Redpanda Cluster node-0 v24.1.6 - 5e880f6fd1a610d0991b00e32c012a03b14888ca redpanda@2a7e5bbf36a9:/$

What went wrong?

I used the Quickstart / Docker compose from our docs and tried to authenticate with SASL against Redpanda. I got the following log message:

WARN  2024-06-05 22:31:05,339 [shard 0:admi] seastar_memory - oversized allocation: 196608 bytes. This is non-fatal, but could lead to latency and/or fragmentation issues. Please report: at 0x831e503 0x8028c67 0x8034603 0x2df94a3 /opt/redpanda/lib/libc++.so.1+0x64617 /opt/redpanda/lib/libc++.so.1+0x68a43 0x376c39f 0x376b7e3 0x376b147 0x37670ef 0x333404b 0x8207a93 0x8207603 0x355c07b 0x820f493 0x81d102b 0x81e299b 0x81e09bb 0x81dddb3 0x81e64a7 0x80d632b 0x80d89df 0x80d6bb3 0x7ff9ec7 0x7ff8a1b 0x2d206bb 0x83692ff /opt/redpanda/lib/libc.so.6+0x2b1c7 /opt/redpanda/lib/libc.so.6+0x2b29f 0x2d195ef
   --------
   seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::httpd::connection::read_one()::$_0, seastar::future<void> seastar::future<void>::then_impl_nrvo<seastar::httpd::connection::read_one()::$_0, seastar::future<void>>(seastar::httpd::connection::read_one()::$_0&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, seastar::httpd::connection::read_one()::$_0&, seastar::future_state<seastar::internal::monostate>&&), void>
   --------
   seastar::internal::do_until_state<seastar::httpd::connection::read()::$_0, seastar::httpd::connection::read()::$_1>
   --------
   seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::httpd::connection::read()::$_2, seastar::futurize<seastar::future<void>>::type seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::httpd::connection::read()::$_2>(seastar::httpd::connection::read()::$_2&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, seastar::httpd::connection::read()::$_2&, seastar::future_state<seastar::internal::monostate>&&), void>
WARN  2024-06-05 22:31:05,707 [shard 0:main] seastar_memory - oversized allocation: 217088 bytes. This is non-fatal, but could lead to latency and/or fragmentation issues. Please report: at 0x831e503 0x8028c67 0x8032efb /opt/redpanda/lib/libgnutls.so.30+0xc5ca3 /opt/redpanda/lib/libgnutls.so.30+0x12a9e3 /opt/redpanda/lib/libgnutls.so.30+0x813df 0x82d10f7 0x81a6e83
Rate-limit: suppressed 74 backtraces on shard 0

What should have happened instead?

1. No oversize alloc
2. The "Please report at" dumps a hex string and I assume it should be something else / human readable?

How to reproduce the issue?

Not entirely sure whether it's easier to reproduce, but if needed please reach out to me quickly and I can try to reproduce this in my setup.

1. 3. 4.

Additional information

Please attach any relevant logs, backtraces, or metric charts.

JIRA Link: CORE-3235

[StephanDollberg]

This is a known issue. It comes from within gnutls. Nothing we can about at this point (will be replaced by openssl) but also not critical as it's on startup.

[nvartolomei]

@StephanDollberg the first one seems to be about seastar httpd handler. The second one is gnutls.

WARN  2024-07-31 12:37:27,228 [shard 0:admi] seastar_memory - oversized allocation: 196608 bytes. This is non-fatal, but could lead to latency and/or fragmentation issues. Please report: at 0xa4edd03 0xa1336bd 0xa13ff78 0x33a29a7 /home/nv/redpanda/vbuild/release/clang/rp_deps_install/lib/libc++.so.1+0x63aa7 /home/nv/redpanda/vbuild/release/clang/rp_deps_install/lib/libc++.so.1+0x6778f 0x3ffa3aa 0x3ff9477 0x3ff8c13 0x3ff3085 0x3a05fa2 0xa39a303 0xa399cd8 0xa399c4c 0x3cd5b51 0xa3a2df6 0xa357033 0xa36dd6c 0xa36b1cf 0xa36722b 0xa372b31 0xa22029f 0xa2239c1 0xa220b86 0xa0fe3b0 0xa0fc7a8 0x32968b6 0xa76f039 /lib/x86_64-linux-gnu/libc.so.6+0x27249 /lib/x86_64-linux-gnu/libc.so.6+0x27304 0x328f0e0
   --------
   seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::httpd::connection::read_one()::$_0, seastar::future<void> seastar::future<void>::then_impl_nrvo<seastar::httpd::connection::read_one()::$_0, seastar::future<void>>(seastar::httpd::connection::read_one()::$_0&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, seastar::httpd::connection::read_one()::$_0&, seastar::future_state<seastar::internal::monostate>&&), void>
   --------
   seastar::internal::do_until_state<seastar::httpd::connection::read()::$_0, seastar::httpd::connection::read()::$_1>
   --------
   seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::httpd::connection::read()::$_2, seastar::futurize<seastar::future<void>>::type seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::httpd::connection::read()::$_2>(seastar::httpd::connection::read()::$_2&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, seastar::httpd::connection::read()::$_2&, seastar::future_state<seastar::internal::monostate>&&), void>

I still get it on any admin endpoint without any TLS setup.

[StephanDollberg]

Ah didn't see that there was two in there. No idea about the httpd one. Probably something in the admin API that needs fixing.

[nvartolomei]

In my case it was triggered by hitting the cluster config endpoint with a PUT for a single setting change.