search

yugabyte / yugabyte-db

YugabyteDB - the cloud native distributed SQL database for mission-critical applications.
https://www.yugabyte.com
Other
8.99k stars 1.07k forks source link

[YSQL] Postgres process get rashed during sequential PITR restore test #17847

Closed qvad closed 1 month ago

qvad commented 1 year ago

Jira Link: DB-6935

Description

Core dump occured during PITR stress test. In this test we do 20 sequential restores with data consistency checks.

Screenshot 2023-06-20 at 11 37 57 
(lldb) target create "/home/yugabyte/yb-software/yugabyte-2.18.1.0-b84-centos-x86_64/postgres/bin/postgres" --core "/home/yugabyte/cores/core_17961_1687178127_!home!yugabyte!yb-software!yugabyte-2.18.1.0-b84-centos-x86_64!postgres!bin!postgres"
Core file '/home/yugabyte/cores/core_17961_1687178127_!home!yugabyte!yb-software!yugabyte-2.18.1.0-b84-centos-x86_64!postgres!bin!postgres' (x86_64) was loaded.
(lldb) bt all
warning: This version of LLDB has no plugin for the language "assembler". Inspection of frame variables will be limited.
warning: This version of LLDB has no plugin for the language "assembler". Inspection of frame variables will be limited.
* thread #1, name = 'postgres', stop reason = signal SIGSEGV
  * frame #0: 0x00007fbb7d1ccd85 libyb_util.so`std::__1::__hash_const_iterator<std::__1::__hash_node<std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, void*>*> std::__1::__hash_table<std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::__unordered_map_hasher<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, true>, std::__1::__unordered_map_equal<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, true>, std::__1::allocator<std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>>>::find<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>(this=<unavailable>, __k=<unavailable>) const at __hash_table:2338:31
    frame #1: 0x00007fbb7d1cc98b libyb_util.so`yb::PrometheusWriter::WriteSingleEntry(std::__1::unordered_map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, long, yb::AggregationFunction, char const*, char const*) [inlined] std::__1::unordered_map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>>>::find[abi:v15007](this=0x00007fbb7e896400, __k="table_id") const at unordered_map:1445:69
    frame #2: 0x00007fbb7d1cc97b libyb_util.so`yb::PrometheusWriter::WriteSingleEntry(this=0x00007fbb71b06828, attr=0x00007fbb7e896400, name=<unavailable>, value=3189, aggregation_function=kSum, type="", description="") at metrics_writer.cc:125:18
    frame #3: 0x00007fbb7e85df4a libserver_process.so`yb::PgPrometheusMetricsHandler(req=<unavailable>, resp=<unavailable>) at pgsql_webserver_wrapper.cc:324:5
    frame #4: 0x00007fbb7e87d450 libserver_process.so`yb::Webserver::Impl::RunPathHandler(yb::Webserver::Impl::PathHandler const&, sq_connection*, sq_request_info*) [inlined] std::__1::__function::__value_func<void (yb::WebCallbackRegistry::WebRequest const&, yb::WebCallbackRegistry::WebResponse*)>::operator(this=0x0000564588018720, __args=0x00007fbb71b08b90, __args=0x00007fbb71b08a60)[abi:v15007](yb::WebCallbackRegistry::WebRequest const&, yb::WebCallbackRegistry::WebResponse*&&) const at function.h:512:16
    frame #5: 0x00007fbb7e87d437 libserver_process.so`yb::Webserver::Impl::RunPathHandler(yb::Webserver::Impl::PathHandler const&, sq_connection*, sq_request_info*) [inlined] std::__1::function<void (yb::WebCallbackRegistry::WebRequest const&, yb::WebCallbackRegistry::WebResponse*)>::operator(this= Function = yb::PgPrometheusMetricsHandler(yb::WebCallbackRegistry::WebRequest const&, yb::WebCallbackRegistry::WebResponse*) , __arg=0x00007fbb71b08b90, __arg=0x00007fbb71b06a60)(yb::WebCallbackRegistry::WebRequest const&, yb::WebCallbackRegistry::WebResponse*) const at function.h:1197:12
    frame #6: 0x00007fbb7e87d437 libserver_process.so`yb::Webserver::Impl::RunPathHandler(this=0x0000564587f64600, handler=<unavailable>, connection=0x00005645d2416000, request_info=<unavailable>) at webserver.cc:599:5
    frame #7: 0x00007fbb7e87ceed libserver_process.so`yb::Webserver::Impl::BeginRequestCallback(this=0x0000564587f64600, connection=0x00005645d2416000, request_info=0x00005645d2416000) at webserver.cc:531:10
    frame #8: 0x00007fbb7e8899a8 libserver_process.so`process_new_connection + 4184
    frame #9: 0x00007fbb7e888658 libserver_process.so`worker_thread + 232
    frame #10: 0x00007fbb817a7694 libpthread.so.0`start_thread(arg=0x00007fbb71b18700) at pthread_create.c:333
    frame #11: 0x00007fbb80ee441d libc.so.6`__clone at clone.S:109
  thread #2, stop reason = signal 0
    frame #0: 0x00007fbb82d43a26 ld.so`strcmp at strcmp.S:173
    frame #1: 0x00007fbb82d3adf5 ld.so`_dl_name_match_p(name="libc.so.6", map=0x00007fbb82e38650) at dl-misc.c:289
    frame #2: 0x00007fbb82d33e70 ld.so`do_lookup_x(undef_name="__cxa_finalize", new_hash=<unavailable>, old_hash=0x00007fff6c47f5e0, ref=0x00007fbb81e96280, result=0x00007fff6c47f5f0, scope=<unavailable>, i=<unavailable>, version=0x00007fbb7c6e5840, flags=5, skip=0x0000000000000000, type_class=1, undef_map=0x00007fbb82e38650) at dl-lookup.c:554
    frame #3: 0x00007fbb82d34a7c ld.so`_dl_lookup_symbol_x(undef_name="__cxa_finalize", undef_map=0x00007fbb82e38650, ref=0x00007fff6c47f6a8, symbol_scope=0x00007fbb82e389a8, version=0x00007fbb7c6e5840, type_class=1, flags=5, skip_map=0x0000000000000000) at dl-lookup.c:829
    frame #4: 0x00007fbb82d3918f ld.so`_dl_fixup(l=<unavailable>, reloc_arg=<unavailable>) at dl-runtime.c:111
    frame #5: 0x00007fbb82d3f887 ld.so`_dl_runtime_resolve_avx512 at dl-trampoline.h:112
  thread #3, stop reason = signal 0
    frame #0: 0x00007fbb80edb5cd libc.so.6`poll at syscall-template.S:84
    frame #1: 0x00007fbb7e8881fc libserver_process.so`master_thread + 572
    frame #2: 0x00007fbb817a7694 libpthread.so.0`start_thread(arg=0x00007fbb72319700) at pthread_create.c:333
    frame #3: 0x00007fbb80ee441d libc.so.6`__clone at clone.S:109

Warning: Please confirm that this issue does not contain any sensitive information

sushantrmishra commented 1 year ago

Possible duplicate of https://github.com/yugabyte/yugabyte-db/issues/18948 . We should reverify once fix for https://github.com/yugabyte/yugabyte-db/issues/18948 has landed.