oversized allocation in lz4_frame_compressor::compress

[andijcr]

LZ4F_decompress internally can allocate a +4MiB buffer and that lead to this crash

this happened in the segment::append path and it's a known problem https://github.com/redpanda-data/redpanda/blob/7d8d220aff01b9a2b1eb6e2c8a6ca039857d179c/src/v/storage/segment.cc#L478-L486

https://github.com/redpanda-data/redpanda/blob/7d8d220aff01b9a2b1eb6e2c8a6ca039857d179c/src/v/compression/internal/lz4_frame_compressor.cc#L212-L218

https://github.com/lz4/lz4/blob/4cf83dd1952898e2a8c5fcd689ce459c53f22ff0/lib/lz4frame.c#L1683 https://github.com/lz4/lz4/blob/4cf83dd1952898e2a8c5fcd689ce459c53f22ff0/lib/lz4frame.c#L1425

ERROR 2024-01-19 10:54:47,863 [shard 0] seastar - Failed to allocate 4194308 bytes

{redpanda} 0x6f50683: void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) alla /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/util/backtrace.hh:60
 (posta inline da) seastar::backtrace_buffer::append_backtrace() alla /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:786
 (posta inline da) seastar::print_with_backtrace(seastar::backtrace_buffer&, bool) alla /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:816
 (posta inline da) seastar::print_with_backtrace(char const*, bool) alla /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:828
{redpanda} 0x6fa3bab: seastar::sigabrt_action() alla /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:4065
 (posta inline da) seastar::install_oneshot_signal_handler<6, (void (*)())(&seastar::sigabrt_action)>()::{lambda(int, siginfo_t*, void*)#1}::operator()(int, siginfo_t*, void*) const alla /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:4041
 (posta inline da) seastar::install_oneshot_signal_handler<6, (void (*)())(&seastar::sigabrt_action)>()::{lambda(int, siginfo_t*, void*)#1}::__invoke(int, siginfo_t*, void*) alla /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:4037
addr2line: "/opt/redpanda/lib/libc.so.6": questo file non esiste
{/opt/redpanda/lib/libc.so.6} 0x42abf: /opt/redpanda/lib/libc.so.6 0x42abf 
{/opt/redpanda/lib/libc.so.6} 0x92e3b: /opt/redpanda/lib/libc.so.6 0x92e3b 
{/opt/redpanda/lib/libc.so.6} 0x42a15: /opt/redpanda/lib/libc.so.6 0x42a15 
{/opt/redpanda/lib/libc.so.6} 0x2c82e: /opt/redpanda/lib/libc.so.6 0x2c82e 
{redpanda} 0x6e834aa: seastar::memory::on_allocation_failure(unsigned long) alla /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:2054
{redpanda} 0x6e8194c: seastar::memory::finish_allocation(void*, unsigned long) alla /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1579
 (posta inline da) seastar::memory::allocate_slowpath(unsigned long) alla /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1618
{redpanda} 0x6e8c0b1: seastar::memory::allocate(unsigned long) alla /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1629
 (posta inline da) malloc alla /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:2103
{/opt/redpanda/lib/liblz4.so.1} 0x36a4b: LZ4F_malloc alla /v/build/v_deps_build/lz4-prefix/src/lz4/lib/lz4frame.c:123
 (posta inline da) LZ4F_decompress alla /v/build/v_deps_build/lz4-prefix/src/lz4/lib/lz4frame.c:1628
{redpanda} 0x68b6d65: compression::internal::lz4_frame_compressor::uncompress(iobuf const&) alla /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-086684ed5624ba7fd-1/redpanda/redpanda/src/v/compression/internal/lz4_frame_compressor.cc:212
{redpanda} 0x689ad2c: compression::compressor::uncompress(iobuf const&, model::compression) alla /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-086684ed5624ba7fd-1/redpanda/redpanda/src/v/compression/compression.cc:67
{redpanda} 0x67d60d3: storage::internal::maybe_decompress_batch_sync(model::record_batch const&) alla /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-086684ed5624ba7fd-1/redpanda/redpanda/src/v/storage/parser_utils.cc:59
 (posta inline da) seastar::future<model::record_batch> seastar::futurize<model::record_batch>::invoke<model::record_batch (&)(model::record_batch const&), model::record_batch const&>(model::record_batch (&)(model::record_batch const&), model::record_batch const&) alla /vectorized/include/seastar/core/future.hh:2001
 (posta inline da) auto seastar::futurize_invoke<model::record_batch (&)(model::record_batch const&), model::record_batch const&>(model::record_batch (&)(model::record_batch const&), model::record_batch const&) alla /vectorized/include/seastar/core/future.hh:2030
 (posta inline da) storage::internal::decompress_batch(model::record_batch const&) alla /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-086684ed5624ba7fd-1/redpanda/redpanda/src/v/storage/parser_utils.cc:41
{redpanda} 0x65cc85c: storage::segment::compaction_index_batch(model::record_batch const&) alla /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-086684ed5624ba7fd-1/redpanda/redpanda/src/v/storage/segment.cc:488
{redpanda} 0x65ce368: storage::segment::do_append(model::record_batch const&) alla /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-086684ed5624ba7fd-1/redpanda/redpanda/src/v/storage/segment.cc:544
{redpanda} 0x65cfb6f: storage::segment::append(model::record_batch const&) alla /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-086684ed5624ba7fd-1/redpanda/redpanda/src/v/storage/segment.cc:602
{redpanda} 0x6505c33: storage::disk_log_appender::append_batch_to_segment(model::record_batch const&) alla /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-086684ed5624ba7fd-1/redpanda/redpanda/src/v/storage/disk_log_appender.cc:136
 (posta inline da) storage::disk_log_appender::operator()(model::record_batch&) alla /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-086684ed5624ba7fd-1/redpanda/redpanda/src/v/storage/disk_log_appender.cc:113
{redpanda} 0x2a04208: storage::log_appender::operator()(model::record_batch&) alla /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-086684ed5624ba7fd-1/redpanda/redpanda/src/v/storage/log_appender.h:62
 (posta inline da) raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer::operator()(model::record_batch&) alla /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-086684ed5624ba7fd-1/redpanda/redpanda/src/v/raft/consensus.cc:2550
 (posta inline da) model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}::operator()(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&) const::{lambda(model::record_batch_reader::foreign_data_t&)#1}::operator()(model::record_batch_reader::foreign_data_t&) const alla /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-086684ed5624ba7fd-1/redpanda/redpanda/src/v/model/record_batch_reader.h:134
 (posta inline da) decltype (((std::declval<seastar::internal::variant_visitor<model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}::operator()(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&) const::{lambda(seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >&)#1}, model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}::operator()(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&) const::{lambda(model::record_batch_reader::foreign_data_t&)#1}> >)())((std::declval<model::record_batch_reader::foreign_data_t&>)())) std::__1::__invoke[abi:v160004]<seastar::internal::variant_visitor<model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}::operator()(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&) const::{lambda(seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >&)#1}, model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}::operator()(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&) const::{lambda(model::record_batch_reader::foreign_data_t&)#1}>, model::record_batch_reader::foreign_data_t&>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&&, model::record_batch_reader::foreign_data_t&) alla /vectorized/llvm/bin/../include/c++/v1/__functional/invoke.h:394
 (posta inline da) decltype(auto) std::__1::__variant_detail::__visitation::__variant::__value_visitor<seastar::internal::variant_visitor<model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}::operator()(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&) const::{lambda(seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >&)#1}, model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}::operator()(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&) const::{lambda(model::record_batch_reader::foreign_data_t&)#1}> >::operator()[abi:v160004]<std::__1::__variant_detail::__alt<1ul, model::record_batch_reader::foreign_data_t>&>(std::__1::__variant_detail::__alt<1ul, model::record_batch_reader::foreign_data_t>&) const alla /vectorized/llvm/bin/../include/c++/v1/variant:689
 (posta inline da) decltype (((std::declval<std::__1::__variant_detail::__visitation::__variant::__value_visitor<seastar::internal::variant_visitor<model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}::operator()(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&) const::{lambda(seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >&)#1}, model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}::operator()(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&) const::{lambda(model::record_batch_reader::foreign_data_t&)#1}> > >)())((std::declval<std::__1::__variant_detail::__alt<1ul, model::record_batch_reader::foreign_data_t>&>)())) std::__1::__invoke[abi:v160004]<std::__1::__variant_detail::__visitation::__variant::__value_visitor<seastar::internal::variant_visitor<model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}::operator()(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&) const::{lambda(seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >&)#1}, model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}::operator()(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&) const::{lambda(model::record_batch_reader::foreign_data_t&)#1}> >, std::__1::__variant_detail::__alt<1ul, model::record_batch_reader::foreign_data_t>&>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&&, std::__1::__variant_detail::__alt<1ul, model::record_batch_reader::foreign_data_t>&) alla /vectorized/llvm/bin/../include/c++/v1/__functional/invoke.h:394
 (posta inline da) decltype(auto) std::__1::__variant_detail::__visitation::__base::__dispatcher<1ul>::__dispatch[abi:v160004]<std::__1::__variant_detail::__visitation::__variant::__value_visitor<seastar::internal::variant_visitor<model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}::operator()(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&) const::{lambda(seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >&)#1}, model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}::operator()(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&) const::{lambda(model::record_batch_reader::foreign_data_t&)#1}> >&&, std::__1::__variant_detail::__base<(std::__1::__variant_detail::_Trait)1, seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >, model::record_batch_reader::foreign_data_t>&>(std::__1::__variant_detail::__visitation::__variant::__value_visitor<seastar::internal::variant_visitor<model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}::operator()(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&) const::{lambda(seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >&)#1}, model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}::operator()(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&) const::{lambda(model::record_batch_reader::foreign_data_t&)#1}> >&&, std::__1::__variant_detail::__base<(std::__1::__variant_detail::_Trait)1, seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >, model::record_batch_reader::foreign_data_t>&) alla /vectorized/llvm/bin/../include/c++/v1/variant:569
{redpanda} 0x2a049cd: decltype(auto) std::__1::__variant_detail::__visitation::__base::__visit_alt[abi:v160004]<std::__1::__variant_detail::__visitation::__variant::__value_visitor<seastar::internal::variant_visitor<model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}::operator()(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&) const::{lambda(seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >&)#1}, model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}::operator()(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&) const::{lambda(model::record_batch_reader::foreign_data_t&)#1}> >, std::__1::__variant_detail::__impl<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >, model::record_batch_reader::foreign_data_t>&>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&&, std::__1::__variant_detail::__impl<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >, model::record_batch_reader::foreign_data_t>&) alla /vectorized/llvm/bin/../include/c++/v1/variant:532
 (posta inline da) decltype(auto) std::__1::__variant_detail::__visitation::__variant::__visit_alt[abi:v160004]<std::__1::__variant_detail::__visitation::__variant::__value_visitor<seastar::internal::variant_visitor<model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}::operator()(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&) const::{lambda(seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >&)#1}, model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}::operator()(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&) const::{lambda(model::record_batch_reader::foreign_data_t&)#1}> >, std::__1::variant<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >, model::record_batch_reader::foreign_data_t>&>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&&, std::__1::variant<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >, model::record_batch_reader::foreign_data_t>&) alla /vectorized/llvm/bin/../include/c++/v1/variant:639
 (posta inline da) decltype(auto) std::__1::__variant_detail::__visitation::__variant::__visit_value[abi:v160004]<seastar::internal::variant_visitor<model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}::operator()(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&) const::{lambda(seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >&)#1}, model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}::operator()(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&) const::{lambda(model::record_batch_reader::foreign_data_t&)#1}>, std::__1::variant<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >, model::record_batch_reader::foreign_data_t>&>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&&, std::__1::variant<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >, model::record_batch_reader::foreign_data_t>&) alla /vectorized/llvm/bin/../include/c++/v1/variant:658
 (posta inline da) decltype(auto) std::__1::visit[abi:v160004]<seastar::internal::variant_visitor<model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}::operator()(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&) const::{lambda(seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >&)#1}, model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}::operator()(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&) const::{lambda(model::record_batch_reader::foreign_data_t&)#1}>, std::__1::variant<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >, model::record_batch_reader::foreign_data_t>&, void>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&&, std::__1::variant<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >, model::record_batch_reader::foreign_data_t>&) alla /vectorized/llvm/bin/../include/c++/v1/variant:1756
 (posta inline da) auto seastar::visit<std::__1::variant<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >, model::record_batch_reader::foreign_data_t>&, model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}::operator()(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&) const::{lambda(seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >&)#1}, model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}::operator()(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&) const::{lambda(model::record_batch_reader::foreign_data_t&)#1}>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&&, model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}::operator()(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&) const::{lambda(seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >&)#1}&&, model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}::operator()(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&) const::{lambda(model::record_batch_reader::foreign_data_t&)#1}&&) alla /vectorized/include/seastar/util/variant_utils.hh:71
 (posta inline da) model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}::operator()(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&) const alla /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-086684ed5624ba7fd-1/redpanda/redpanda/src/v/model/record_batch_reader.h:126
 (posta inline da) model::record_batch_reader::impl::do_action<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer, model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >, model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}&&)::{lambda()#1}::operator()() const alla /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-086684ed5624ba7fd-1/redpanda/redpanda/src/v/model/record_batch_reader.h:154
 (posta inline da) seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > seastar::futurize<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > >::invoke<model::record_batch_reader::impl::do_action<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer, model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >, model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}&&)::{lambda()#1}&>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&&) alla /vectorized/include/seastar/core/future.hh:1999
 (posta inline da) auto seastar::futurize_invoke<model::record_batch_reader::impl::do_action<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer, model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >, model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}&&)::{lambda()#1}&>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&&) alla /vectorized/include/seastar/core/future.hh:2030
 (posta inline da) seastar::internal::repeater<model::record_batch_reader::impl::do_action<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer, model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >, model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}&&)::{lambda()#1}>::run_and_dispose() alla /vectorized/include/seastar/core/loop.hh:75
{redpanda} 0x6f48d4f: seastar::reactor::run_tasks(seastar::reactor::task_queue&) alla /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2660
 (posta inline da) seastar::reactor::run_some_tasks() alla /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3123
{redpanda} 0x6f4c3f1: seastar::reactor::do_run() alla /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3307
{redpanda} 0x6f495b6: seastar::reactor::run() alla /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3175
{redpanda} 0x6e4bb40: seastar::app_template::run_deprecated(int, char**, std::__1::function<void ()>&&) alla /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/app-template.cc:265
{redpanda} 0x6e49f38: seastar::app_template::run(int, char**, std::__1::function<seastar::future<int> ()>&&) alla /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/app-template.cc:156
{redpanda} 0x21ec9e5: application::run(int, char**) alla /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-086684ed5624ba7fd-1/redpanda/redpanda/src/v/redpanda/application.cc:347
{redpanda} 0x72b520b: main alla /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-086684ed5624ba7fd-1/redpanda/redpanda/src/v/redpanda/main.cc:22
{/opt/redpanda/lib/libc.so.6} 0x2d58f: /opt/redpanda/lib/libc.so.6 0x2d58f 
{/opt/redpanda/lib/libc.so.6} 0x2d648: /opt/redpanda/lib/libc.so.6 0x2d648 
{redpanda} 0x21e65e4: _start alla ??:?

JIRA Link: CORE-1722

[abhijat]

LZ4 can allocate a maximum of 4MiB + 4 bytes buffer while decompressing, as mentioned in the ticket details.

This allocation is outside of redpanda control, but registering a custom allocator with LZ4F_CustomMem can allow us to control how the allocation happens, and allows us to make use of a reusable buffer and avoid this dynamic allocation for compressed records.

Implementation would require a per shard preallocated buffer which is used by the custom allocator.

[dotnwat]

Does lz4 support fragmented memory at all or streaming operations? If so, we could probably write an adapter of some sort like we've done for zstd streaming. Otherwise, we already have the infrastructure to allocate large contiguous scratch spaces on bootup for this purpose. I think since we've adjusted zstd to do streaming that we don't use it anymore, but it's there and could probalby be used here for lz4.

[travisdowns]

Does lz4 support fragmented memory at all or streaming operations?

I think probably "no" to the first and "yes" to the second.

Streaming helps avoiding needing to allocate large output buffers, since we can accept the output in chunks (and it helps avoid stalls). However, I think this 4MB is an internal allocation during decompression that it will make regardless of streaming or not, for a block which it needs to treat as one unit. I don't think it can use a fragmented allocation for this block.

So yes, it's the pre-allocate route. Probably we don't need many buffers per shard since it's a CPU bound activity which doesn't seem useful to do in parallel (unless maybe we are doing big decompressions in a streaming way while reading from disk: then it does seem like we'd want to overlap them). @ballard26 mentioned that he had an object-pooling thing he wrote for zstd which might be useful here?

[abhijat]

It looks like per call to LZ4F_decompress there are two potential mallocs which can happen. These are guaranteed to happen together, one for the input scratch space and one for the output scratch space. Both will require roughly similar sizes depending on the input. The maximum sizes for the two are:

• input: 4MiB + 4 bytes
• output: 4MiB + 128KiB - here the +128KiB happens when the block mode for the input is linked IE block compressions have been performed relative to each other, rather than independently. Redpanda and kafka explicitly set this to unlinked during compression but I'm not sure if assuming this mode will always be unlinked is safe, it might be safer to assume the mode will always be linked and allocate the extra 128KiB chunk.

The sizes will depend on the input frame header as defined in https://github.com/lz4/lz4/blob/dev/doc/lz4_Frame_format.md (Block Maximum Size section).

So before each call to frame_compressor::uncompress we should have enough space for the above two buffers.

object-pooling

Yes, it looks like it can be used here, the pool can used be for preallocated buffers.

[abhijat]

As an optimization to avoid having all decompress operations waiting for a semaphore even when they require small amounts of space, we can use LZ4F_getFrameInfo, which is already called before starting decompression (when enough data is available).

If it succeeds it gives us information about the frame which contains the block size id. We only need to wait for a large contiguous chunk when the block size id corresponds to 4MiB, in other cases we can fallback to a fast path of normal allocation without waiting for resources.

[abhijat]

The use of a custom memory manager requires static linking for lz4, from the relevant header lz4frame.h

/* These declarations are not stable and may change in the future.
 * They are therefore only safe to depend on
 * when the caller is statically linked against the library.
 * To access their declarations, define LZ4F_STATIC_LINKING_ONLY.
 *
 * By default, these symbols aren't published into shared/dynamic libraries.
 * You can override this behavior and force them to be published
 * by defining LZ4F_PUBLISH_STATIC_FUNCTIONS.
 * Use at your own risk.
 */