search

redpanda-data / redpanda

Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
https://redpanda.com
9.16k stars 559 forks source link

Oversized allocation: 409600 bytes in `kafka::create_topics_api` #16758

Closed ballard26 closed 3 months ago

ballard26 commented 4 months ago

Occurs while sending a topic creation request for 4k+ topics in https://github.com/redpanda-data/redpanda/pull/16463 . Backtrace;

/home/ubuntu/tests/results/2024-02-26--002/ManyPartitionsTest/test_many_topics_throughput/1/RedpandaService-0-139946360787280/ip-172-31-1-54/redpanda.log:861:WARN  2024-02-26 21:05:38,101 [shard 0:main] seastar_memory - oversized allocation: 409600 bytes. This is non-fatal, but could lead to latency and/or fragmentation issues. Please report: at
[Backtrace #53]
{/opt/redpanda/libexec/redpanda} 0x93c83d3: void seastar::backtrace<seastar::current_backtrace_tasklocal()::$_0>(seastar::current_backtrace_tasklocal()::$_0&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/util/backtrace.hh:64
 (inlined by) seastar::current_backtrace_tasklocal() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/util/backtrace.cc:98
 (inlined by) seastar::current_tasktrace() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/util/backtrace.cc:149
 (inlined by) seastar::current_backtrace() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/util/backtrace.cc:182
{/opt/redpanda/libexec/redpanda} 0x902fefc: seastar::memory::cpu_pages::warn_large_allocation(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:839
 (inlined by) seastar::memory::cpu_pages::check_large_allocation(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:904
 (inlined by) seastar::memory::cpu_pages::allocate_large(unsigned int, bool) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:911
 (inlined by) seastar::memory::allocate_large(unsigned long, bool) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1521
 (inlined by) seastar::memory::allocate_slowpath(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1645
{/opt/redpanda/libexec/redpanda} 0x903d1d0: seastar::memory::allocate(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1658
 (inlined by) operator new(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:2355
{/opt/redpanda/libexec/redpanda} 0x4737618: void* std::__1::__libcpp_operator_new[abi:v160004]<unsigned long>(unsigned long) at /vectorized/llvm/bin/../include/c++/v1/new:266
 (inlined by) std::__1::__libcpp_allocate[abi:v160004](unsigned long, unsigned long) at /vectorized/llvm/bin/../include/c++/v1/new:292
 (inlined by) std::__1::allocator<cluster::custom_assignable_topic_configuration>::allocate[abi:v160004](unsigned long) at /vectorized/llvm/bin/../include/c++/v1/__memory/allocator.h:115
 (inlined by) std::__1::__allocation_result<std::__1::allocator_traits<std::__1::allocator<cluster::custom_assignable_topic_configuration> >::pointer> std::__1::__allocate_at_least[abi:v160004]<std::__1::allocator<cluster::custom_assignable_topic_configuration> >(std::__1::allocator<cluster::custom_assignable_topic_configuration>&, unsigned long) at /vectorized/llvm/bin/../include/c++/v1/__memory/allocate_at_least.h:55
 (inlined by) std::__1::__split_buffer<cluster::custom_assignable_topic_configuration, std::__1::allocator<cluster::custom_assignable_topic_configuration>&>::__split_buffer(unsigned long, unsigned long, std::__1::allocator<cluster::custom_assignable_topic_configuration>&) at /vectorized/llvm/bin/../include/c++/v1/__split_buffer:323
 (inlined by) std::__1::vector<cluster::custom_assignable_topic_configuration, std::__1::allocator<cluster::custom_assignable_topic_configuration> >::reserve(unsigned long) at /vectorized/llvm/bin/../include/c++/v1/vector:1503
 (inlined by) std::__1::vector<decltype(to_cluster_type(*(fp))), std::__1::allocator<decltype(to_cluster_type(*(fp)))> > kafka::to_cluster_type<fragmented_vector<kafka::creatable_topic, 18446744073709551615ul>::iter<false> >(fragmented_vector<kafka::creatable_topic, 18446744073709551615ul>::iter<false>, fragmented_vector<kafka::creatable_topic, 18446744073709551615ul>::iter<false>) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-07246cfc3e72245bc-1/redpanda/redpanda/src/v/kafka/server/handlers/topics/topic_utils.h:127
 (inlined by) kafka::handler_template<kafka::create_topics_api, (short)0, (short)6, seastar::future<seastar::foreign_ptr<std::__1::unique_ptr<kafka::response, std::__1::default_delete<kafka::response> > > >, &(kafka::default_estimate_adaptor(unsigned long, kafka::connection_context&))>::handle(kafka::request_context, seastar::smp_service_group) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-07246cfc3e72245bc-1/redpanda/redpanda/src/v/kafka/server/handlers/create_topics.cc:320
{/opt/redpanda/libexec/redpanda} 0x43ab80a: std::__1::coroutine_handle<seastar::internal::coroutine_traits_base<seastar::foreign_ptr<std::__1::unique_ptr<kafka::response, std::__1::default_delete<kafka::response> > > >::promise_type>::resume[abi:v160004]() const at /vectorized/llvm/bin/../include/c++/v1/__coroutine/coroutine_handle.h:169
 (inlined by) seastar::internal::coroutine_traits_base<seastar::foreign_ptr<std::__1::unique_ptr<kafka::response, std::__1::default_delete<kafka::response> > > >::promise_type::run_and_dispose() at /vectorized/include/seastar/core/coroutine.hh:83
{/opt/redpanda/libexec/redpanda} 0x90fd3ff: seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2750
 (inlined by) seastar::reactor::run_some_tasks() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3213
{/opt/redpanda/libexec/redpanda} 0x9100b71: seastar::reactor::do_run() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3397
{/opt/redpanda/libexec/redpanda} 0x90fdd06: seastar::reactor::run() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3265
{/opt/redpanda/libexec/redpanda} 0x8ffce80: seastar::app_template::run_deprecated(int, char**, std::__1::function<void ()>&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/app-template.cc:276
{/opt/redpanda/libexec/redpanda} 0x8ffb278: seastar::app_template::run(int, char**, std::__1::function<seastar::future<int> ()>&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/app-template.cc:167
{/opt/redpanda/libexec/redpanda} 0x2ccffb6: application::run(int, char**) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-07246cfc3e72245bc-1/redpanda/redpanda/src/v/redpanda/application.cc:416
{/opt/redpanda/libexec/redpanda} 0x941e759: main at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-07246cfc3e72245bc-1/redpanda/redpanda/src/v/redpanda/main.cc:22
{/opt/redpanda/lib/libc.so.6} 0x2d58f: ?? at ??:0
{/opt/redpanda/lib/libc.so.6} 0x2d648: __libc_start_main at ??:0
{/opt/redpanda/libexec/redpanda} 0x2cc87e4: _start at ??:0
piyushredpanda commented 4 months ago

We want to get this fixed within the next two weeks, @michael-redpanda and @aanthony-rp