search

redpanda-data / redpanda

Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
https://redpanda.com
9.74k stars 592 forks source link

Segmentation fault after finding manifest #3517

Closed NyaliaLui closed 2 years ago

NyaliaLui commented 2 years ago

Version & Environment

Redpanda version: v21.11.3-si-beta8

The following backtrace was seen on BYOC during long running tests for shadow indexing.

Backtrace

/opt/redpanda/libexec/redpanda: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /opt/redpanda/lib/ld.so, for GNU/Linux 3.2.0, BuildID[sha1]=77ef482aaedafa88c022a72d524e984372eb0fea, stripped

[Backtrace #0]
void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/util/backtrace.hh:59
 (inlined by) seastar::backtrace_buffer::append_backtrace() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:754
 (inlined by) seastar::print_with_backtrace(seastar::backtrace_buffer&, bool) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:784
 (inlined by) seastar::print_with_backtrace(char const*, bool) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:796
 (inlined by) seastar::sigsegv_action() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3568
 (inlined by) operator() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3554
 (inlined by) __invoke at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3550
?? ??:0
?? ??:0
rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >::SetStringRaw(rapidjson::GenericStringRef<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>&) at /vectorized/include/rapidjson/document.h:2439
 (inlined by) GenericValue at /vectorized/include/rapidjson/document.h:843
 (inlined by) rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>, rapidjson::CrtAllocator>::String(char const*, unsigned int, bool) at /vectorized/include/rapidjson/document.h:2822
 (inlined by) rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>, rapidjson::CrtAllocator>::Key(char const*, unsigned int, bool) at /vectorized/include/rapidjson/document.h:2830
 (inlined by) void rapidjson::GenericReader<rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::CrtAllocator>::ParseString<0u, rapidjson::BasicIStreamWrapper<std::__1::basic_istream<char, std::__1::char_traits<char> > >, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>, rapidjson::CrtAllocator> >(rapidjson::BasicIStreamWrapper<std::__1::basic_istream<char, std::__1::char_traits<char> > >&, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>, rapidjson::CrtAllocator>&, bool) at /vectorized/include/rapidjson/reader.h:982
void rapidjson::GenericReader<rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::CrtAllocator>::ParseObject<0u, rapidjson::BasicIStreamWrapper<std::__1::basic_istream<char, std::__1::char_traits<char> > >, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>, rapidjson::CrtAllocator> >(rapidjson::BasicIStreamWrapper<std::__1::basic_istream<char, std::__1::char_traits<char> > >&, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>, rapidjson::CrtAllocator>&) at /vectorized/include/rapidjson/reader.h:760
void rapidjson::GenericReader<rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::CrtAllocator>::ParseObject<0u, rapidjson::BasicIStreamWrapper<std::__1::basic_istream<char, std::__1::char_traits<char> > >, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>, rapidjson::CrtAllocator> >(rapidjson::BasicIStreamWrapper<std::__1::basic_istream<char, std::__1::char_traits<char> > >&, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>, rapidjson::CrtAllocator>&) at /vectorized/include/rapidjson/reader.h:772
void rapidjson::GenericReader<rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::CrtAllocator>::ParseObject<0u, rapidjson::BasicIStreamWrapper<std::__1::basic_istream<char, std::__1::char_traits<char> > >, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>, rapidjson::CrtAllocator> >(rapidjson::BasicIStreamWrapper<std::__1::basic_istream<char, std::__1::char_traits<char> > >&, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>, rapidjson::CrtAllocator>&) at /vectorized/include/rapidjson/reader.h:772
rapidjson::ParseResult rapidjson::GenericReader<rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::CrtAllocator>::Parse<0u, rapidjson::BasicIStreamWrapper<std::__1::basic_istream<char, std::__1::char_traits<char> > >, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>, rapidjson::CrtAllocator> >(rapidjson::BasicIStreamWrapper<std::__1::basic_istream<char, std::__1::char_traits<char> > >&, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>, rapidjson::CrtAllocator>&) at /vectorized/include/rapidjson/reader.h:575
 (inlined by) rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>, rapidjson::CrtAllocator>& rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>, rapidjson::CrtAllocator>::ParseStream<0u, rapidjson::UTF8<char>, rapidjson::BasicIStreamWrapper<std::__1::basic_istream<char, std::__1::char_traits<char> > > >(rapidjson::BasicIStreamWrapper<std::__1::basic_istream<char, std::__1::char_traits<char> > >&) at /vectorized/include/rapidjson/document.h:2630
rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>, rapidjson::CrtAllocator>& rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>, rapidjson::CrtAllocator>::ParseStream<rapidjson::BasicIStreamWrapper<std::__1::basic_istream<char, std::__1::char_traits<char> > > >(rapidjson::BasicIStreamWrapper<std::__1::basic_istream<char, std::__1::char_traits<char> > >&) at /vectorized/include/rapidjson/document.h:2656
 (inlined by) cloud_storage::manifest::update(seastar::input_stream<char>) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-09f40697b81d3241f-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/cloud_storage/manifest.cc:272
std::experimental::coroutines_v1::coroutine_handle<void>::resume() at /vectorized/llvm/bin/../include/c++/v1/experimental/coroutine:121
 (inlined by) seastar::internal::coroutine_traits_base<void>::promise_type::run_and_dispose() at /vectorized/include/seastar/core/coroutine.hh:108
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2230
 (inlined by) seastar::reactor::run_some_tasks() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2639
seastar::reactor::do_run() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2808
operator() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3996
 (inlined by) decltype ((std::__1::forward<seastar::smp::configure(boost::program_options::variables_map, seastar::reactor_config)::$_92&>({parm#1}))()) std::__1::__invoke<seastar::smp::configure(boost::program_options::variables_map, seastar::reactor_config)::$_92&>(seastar::smp::configure(boost::program_options::variables_map, seastar::reactor_config)::$_92&) at /vectorized/llvm/bin/../include/c++/v1/type_traits:3694
 (inlined by) void std::__1::__invoke_void_return_wrapper<void, true>::__call<seastar::smp::configure(boost::program_options::variables_map, seastar::reactor_config)::$_92&>(seastar::smp::configure(boost::program_options::variables_map, seastar::reactor_config)::$_92&) at /vectorized/llvm/bin/../include/c++/v1/__functional_base:348
 (inlined by) std::__1::__function::__alloc_func<seastar::smp::configure(boost::program_options::variables_map, seastar::reactor_config)::$_92, std::__1::allocator<seastar::smp::configure(boost::program_options::variables_map, seastar::reactor_config)::$_92>, void ()>::operator()() at /vectorized/llvm/bin/../include/c++/v1/functional:1558
 (inlined by) std::__1::__function::__func<seastar::smp::configure(boost::program_options::variables_map, seastar::reactor_config)::$_92, std::__1::allocator<seastar::smp::configure(boost::program_options::variables_map, seastar::reactor_config)::$_92>, void ()>::operator()() at /vectorized/llvm/bin/../include/c++/v1/functional:1732
std::__1::__function::__value_func<void ()>::operator()() const at /vectorized/llvm/bin/../include/c++/v1/functional:1885
 (inlined by) std::__1::function<void ()>::operator()() const at /vectorized/llvm/bin/../include/c++/v1/functional:2560
 (inlined by) seastar::posix_thread::start_routine(void*) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/posix.cc:60
/opt/redpanda/lib/libpthread.so.0: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=7b0cdaf878ab4f99078439d864af70a5fd7b5a2c, for GNU/Linux 3.2.0, stripped

__libpthread_freeres at ??:?
/opt/redpanda/lib/libc.so.6: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=6e3c087aca9b39549e4ba92c451f1e399b586e28, for GNU/Linux 3.2.0, stripped

clone at ??:?
NyaliaLui commented 2 years ago

The grafana logs before the segfault:

    2022-01-16T16:40:21.470906162Z stderr F INFO  2022-01-16 16:40:21,470 [shard 0] archival - [fiber1] - service.cc:293 - Found manifest for partition {kafka/test-2k-e/1125}
    2022-01-16T16:40:21.472525671Z stderr F INFO  2022-01-16 16:40:21,472 [shard 3] archival - [fiber1] - service.cc:293 - Found manifest for partition {kafka/test-2k-e/1515}
    2022-01-16T16:40:21.473230971Z stderr F INFO  2022-01-16 16:40:21,473 [shard 0] archival - [fiber1] - service.cc:293 - Found manifest for partition {kafka/test-2k-e/1093}
    2022-01-16T16:40:21.473269582Z stderr F INFO  2022-01-16 16:40:21,473 [shard 10] archival - [fiber1] - service.cc:293 - Found manifest for partition {kafka/test-2k-e/1113}
    2022-01-16T16:40:21.474132512Z stderr F INFO  2022-01-16 16:40:21,474 [shard 3] archival - [fiber1] - service.cc:293 - Found manifest for partition {kafka/test-2k-e/43}
    2022-01-16T16:40:21.474697653Z stderr F INFO  2022-01-16 16:40:21,474 [shard 13] archival - [fiber1] - service.cc:293 - Found manifest for partition {kafka/test-2k-e/1887}
    2022-01-16T16:40:21.475543796Z stderr F INFO  2022-01-16 16:40:21,475 [shard 0] archival - [fiber1] - service.cc:293 - Found manifest for partition {kafka/test-2k-e/613}
    2022-01-16T16:40:21.47792319Z stderr F INFO  2022-01-16 16:40:21,477 [shard 12] archival - [fiber1] - service.cc:293 - Found manifest for partition {kafka/test-2k-e/1853}
    2022-01-16T16:40:21.478872659Z stderr F INFO  2022-01-16 16:40:21,478 [shard 7] archival - [fiber1] - service.cc:293 - Found manifest for partition {kafka/test-2k-e/691}
    2022-01-16T16:40:21.480276527Z stderr F INFO  2022-01-16 16:40:21,480 [shard 7] archival - [fiber1] - service.cc:293 - Found manifest for partition {kafka/test-2k-e/1427}
NyaliaLui commented 2 years ago

Brokers were configured with the new storage_read_buffer_size and storage_read_readahead_count configs: Do kubectl edit cluster -n <namespace> <cluster name> to get

spec:
   additionalConfiguration:
     redpanda.default_topic_replications: "3"
     redpanda.id_allocator_replication: "3"
     redpanda.storage_read_buffer_size: "32768"
     redpanda.storage_read_readahead_count: "2"

The topic:

SUMMARY
=======
NAME        test-2k-e
PARTITIONS  2048
REPLICAS    3

CONFIGS
=======
KEY                     VALUE                          SOURCE
cleanup.policy          delete                         DYNAMIC_TOPIC_CONFIG
compression.type        producer                       DEFAULT_CONFIG
message.timestamp.type  CreateTime                     DEFAULT_CONFIG
partition_count         2048                           DYNAMIC_TOPIC_CONFIG
redpanda.datapolicy     function_name:  script_name:   DEFAULT_CONFIG
redpanda.remote.read    true                           DYNAMIC_TOPIC_CONFIG
redpanda.remote.write   true                           DYNAMIC_TOPIC_CONFIG
replication_factor      3                              DYNAMIC_TOPIC_CONFIG
retention.bytes         2147483648                     DYNAMIC_TOPIC_CONFIG
retention.ms            604800000                      DEFAULT_CONFIG
segment.bytes           1073741824                     DYNAMIC_TOPIC_CONFIG
jcsp commented 2 years ago

This is a duplicate of: