search

yugabyte / yugabyte-db

YugabyteDB - the cloud native distributed SQL database for mission-critical applications.
https://www.yugabyte.com
Other
8.68k stars 1.04k forks source link

[CDCSDK] CDC + CQL hit CppOomPolicy issues. Need to verify memtracker #19370

Open shamanthchandra-yb opened 9 months ago

shamanthchandra-yb commented 9 months ago

Jira Link: DB-8169

Description

We were able to 2 times (20% of time with original workload of CQL)

As per discussion, memtracker for CDC records needs to be taken a look. As discussed here https://yugabyte.slack.com/archives/C03H4D4EVC6/p1695806406849879

Discussed with @suranjan, that the universe shared already had memtracker cleaned up. So, we need to analyze fresh run. To track this, we can track this with ticket.

thread #1, name = 'yb-tserver', stop reason = signal SIGABRT
  * frame #0: 0x00007f3104d460a7 libc.so.6`__GI_raise(sig=6) at raise.c:54
    frame #1: 0x00007f3104d474aa libc.so.6`__GI_abort at abort.c:89
    frame #2: 0x000055b726a410c8 yb-tserver`tcmalloc::tcmalloc_internal::Crash(tcmalloc::tcmalloc_internal::CrashMode, char const*, int, tcmalloc::tcmalloc_internal::LogItem, tcmalloc::tcmalloc_internal::LogItem, tcmalloc::tcmalloc_internal::LogItem, tcmalloc::tcmalloc_internal::LogItem, tcmalloc::tcmalloc_internal::LogItem, tcmalloc::tcmalloc_internal::LogItem) + 312
    frame #3: 0x000055b726a4703b yb-tserver`tcmalloc::tcmalloc_internal::CppOomPolicy::handle_oom(unsigned long) + 123
    frame #4: 0x000055b726a068aa yb-tserver`void* slow_alloc<tcmalloc::tcmalloc_internal::TCMallocPolicy<tcmalloc::tcmalloc_internal::CppOomPolicy, tcmalloc::tcmalloc_internal::DefaultAlignPolicy, tcmalloc::tcmalloc_internal::AllocationAccessHotPolicy, tcmalloc::tcmalloc_internal::InvokeHooksPolicy, tcmalloc::tcmalloc_internal::LocalNumaPartitionPolicy>, std::nullptr_t>(tcmalloc::tcmalloc_internal::TCMallocPolicy<tcmalloc::tcmalloc_internal::CppOomPolicy, tcmalloc::tcmalloc_internal::DefaultAlignPolicy, tcmalloc::tcmalloc_internal::AllocationAccessHotPolicy, tcmalloc::tcmalloc_internal::InvokeHooksPolicy, tcmalloc::tcmalloc_internal::LocalNumaPartitionPolicy>, unsigned long, std::nullptr_t) + 794
    frame #5: 0x000055b726a0639e yb-tserver`TCMallocInternalNew + 414
    frame #6: 0x000055b724984d65 yb-tserver`std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>::push_back(char) + 165
    frame #7: 0x000055b72496f5dd yb-tserver`std::__1::basic_stringbuf<char, std::__1::char_traits<char>, std::__1::allocator<char>>::overflow(int) + 109
    frame #8: 0x000055b72498afcf yb-tserver`std::__1::basic_ostream<char, std::__1::char_traits<char>>::put(char) + 111
    frame #9: 0x000055b72615572e yb-tserver`rapidjson::Writer<yb::UTF8StringStreamBuffer, rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::CrtAllocator, 0u>::WriteString(this=0x000006adb4704670, str="[e1a49ff3-001d-4bfb-8897-8893eeb75df7:182854:S4O7aTT60334R07PTP0??a!0??a6SSP!8RT!T5P8bUU71445S18QUQ1aab?1aab7TTQ?9SU?U6Q9cVV82556T29RVR2bbca2bbc8UURa!TVa:seed, e1a49ff3-001d-4bfb-8897-8893eeb75df7:val:182854:seed:S4O7aTT60334R07PTP0??a!0??a6SSP!8RT!T5P8bUU71445S18QUQ1aab?1aab7TTQ?9SU?U6Q9cVV82556T29RVR2bbca2bbc8UURa!TVa, e1a49ff3-001d-4bfb-8897-8893eeb75df7:val:182854:seed:S4O7aTT60334R07PTP0??a!0??a6SSP!8RT!T5P8bUU71445S18QUQ1aab?1aab7TTQ?9SU?U6Q9cVV82556T29RVR2bbca2bbc8UURa!TVa, e1a49ff3-001d-4bfb-8897-8893eeb75df7:val:182854:seed:S4O7aTT60334R07PTP0??a!0??a6SSP!8RT!T5P8bUU71445S18QUQ1aab?1aab7TTQ?9SU?U6Q9cVV82556T29RVR2bbca2bbc8UURa!TVa, e1a49ff3-001d-4bfb-8897-8893eeb75df7:val:182854:seed:S4O7aTT60334R07PTP0??a!0??a6SSP!8RT!T5P8bUU71445S18QUQ1aab?1aab7TTQ?9SU?U6Q9cVV82556T29RVR2bbca2bbc8UURa!TVa, e1a49ff3-001d-4bfb-8897-8893eeb75df7:val:182854:seed:S4O7aTT60334R07PTP0??a!0??a6SSP!8RT!T5P8bUU71445S18QUQ1aab?1aab7TTQ?9SU?U6Q9cVV82556T29RVR2bbca2bbc8UURa!TVa, e1a49ff3-001d-4bfb-8897-8893eeb75df7:val:182854:s"..., length=4096) at writer.h:0
    frame #10: 0x000055b726156dbe yb-tserver`yb::JsonWriter::Protobuf(google::protobuf::Message const&) [inlined] yb::JsonWriter::String(this=0x00007f3031dcc910, str="[e1a49ff3-001d-4bfb-8897-8893eeb75df7:182854:S4O7aTT60334R07PTP0??a!0??a6SSP!8RT!T5P8bUU71445S18QUQ1aab?1aab7TTQ?9SU?U6Q9cVV82556T29RVR2bbca2bbc8UURa!TVa:seed, e1a49ff3-001d-4bfb-8897-8893eeb75df7:val:182854:seed:S4O7aTT60334R07PTP0??a!0??a6SSP!8RT!T5P8bUU71445S18QUQ1aab?1aab7TTQ?9SU?U6Q9cVV82556T29RVR2bbca2bbc8UURa!TVa, e1a49ff3-001d-4bfb-8897-8893eeb75df7:val:182854:seed:S4O7aTT60334R07PTP0??a!0??a6SSP!8RT!T5P8bUU71445S18QUQ1aab?1aab7TTQ?9SU?U6Q9cVV82556T29RVR2bbca2bbc8UURa!TVa, e1a49ff3-001d-4bfb-8897-8893eeb75df7:val:182854:seed:S4O7aTT60334R07PTP0??a!0??a6SSP!8RT!T5P8bUU71445S18QUQ1aab?1aab7TTQ?9SU?U6Q9cVV82556T29RVR2bbca2bbc8UURa!TVa, e1a49ff3-001d-4bfb-8897-8893eeb75df7:val:182854:seed:S4O7aTT60334R07PTP0??a!0??a6SSP!8RT!T5P8bUU71445S18QUQ1aab?1aab7TTQ?9SU?U6Q9cVV82556T29RVR2bbca2bbc8UURa!TVa, e1a49ff3-001d-4bfb-8897-8893eeb75df7:val:182854:seed:S4O7aTT60334R07PTP0??a!0??a6SSP!8RT!T5P8bUU71445S18QUQ1aab?1aab7TTQ?9SU?U6Q9cVV82556T29RVR2bbca2bbc8UURa!TVa, e1a49ff3-001d-4bfb-8897-8893eeb75df7:val:182854:s"...) at jsonwriter.cc:200:53
    frame #11: 0x000055b726156db0 yb-tserver`yb::JsonWriter::Protobuf(google::protobuf::Message const&) [inlined] yb::JsonWriter::ProtobufField(this=0x00007f3031dcc910, pb=0x000006acbcbc4d00, field=0x000006acbe473b70) at jsonwriter.cc:285:7
    frame #12: 0x000055b726156bcc yb-tserver`yb::JsonWriter::Protobuf(this=0x00007f3031dcc910, pb=0x000006acbcbc4d00) at jsonwriter.cc:251:7
    frame #13: 0x000055b726156a20 yb-tserver`yb::JsonWriter::ProtobufRepeatedField(this=0x00007f3031dcc910, pb=0x000006ad4c93f1c0, field=0x000006acb4daa098, index=149) at jsonwriter.cc:326:7
    frame #14: 0x000055b726156bc2 yb-tserver`yb::JsonWriter::Protobuf(this=0x00007f3031dcc910, pb=0x000006ad4c93f1c0) at jsonwriter.cc:247:9
    frame #15: 0x000055b726156df6 yb-tserver`yb::JsonWriter::Protobuf(google::protobuf::Message const&) [inlined] yb::JsonWriter::ProtobufField(this=0x00007f3031dcc910, pb=0x000006ad96913310, field=0x000006aca79fcbf8) at jsonwriter.cc:288:7
    frame #16: 0x000055b726156bcc yb-tserver`yb::JsonWriter::Protobuf(this=0x00007f3031dcc910, pb=0x000006ad96913310) at jsonwriter.cc:251:7
    frame #17: 0x000055b726156a20 yb-tserver`yb::JsonWriter::ProtobufRepeatedField(this=0x00007f3031dcc910, pb=0x000006ad41c3c300, field=0x000006acbf72e2f8, index=4) at jsonwriter.cc:326:7
    frame #18: 0x000055b726156bc2 yb-tserver`yb::JsonWriter::Protobuf(this=0x00007f3031dcc910, pb=0x000006ad41c3c300) at jsonwriter.cc:247:9
    frame #19: 0x000055b726156a20 yb-tserver`yb::JsonWriter::ProtobufRepeatedField(this=0x00007f3031dcc910, pb=0x00007f3031dcc860, field=0x000006acb81b6be0, index=11) at jsonwriter.cc:326:7
    frame #20: 0x000055b726156bc2 yb-tserver`yb::JsonWriter::Protobuf(this=0x00007f3031dcc910, pb=0x00007f3031dcc860) at jsonwriter.cc:247:9
    frame #21: 0x000055b725abc20b yb-tserver`yb::(anonymous namespace)::RpczPathHandler(messenger=<unavailable>, req=<unavailable>, resp=0x00007f3031dcc960) at rpcz-path-handler.cc:77:10
    frame #22: 0x000055b725add408 yb-tserver`yb::Webserver::Impl::BeginRequestCallbackStatic(sq_connection*) [inlined] std::__1::__function::__value_func<void (yb::WebCallbackRegistry::WebRequest const&, yb::WebCallbackRegistry::WebResponse*)>::operator(this=0x000006acbe8d0380, __args=0x00007f3031dcea88, __args=0x00007f3031dce970)[abi:v160006](yb::WebCallbackRegistry::WebRequest const&, yb::WebCallbackRegistry::WebResponse*&&) const at function.h:510:16
    frame #23: 0x000055b725add3ee yb-tserver`yb::Webserver::Impl::BeginRequestCallbackStatic(sq_connection*) [inlined] std::__1::function<void (yb::WebCallbackRegistry::WebRequest const&, yb::WebCallbackRegistry::WebResponse*)>::operator(this=0x000006acbe8d0380, __arg=0x00007f3031dcea88, __arg=0x00007f3031dcc960)(yb::WebCallbackRegistry::WebRequest const&, yb::WebCallbackRegistry::WebResponse*) const at function.h:1156:12
    frame #24: 0x000055b725add3ee yb-tserver`yb::Webserver::Impl::BeginRequestCallbackStatic(sq_connection*) at webserver.cc:625:5
    frame #25: 0x000055b725add0d7 yb-tserver`yb::Webserver::Impl::BeginRequestCallbackStatic(sq_connection*) [inlined] yb::Webserver::Impl::BeginRequestCallback(this=0x000006acbefe5400, connection=0x000000cb7827a000, request_info=0x000000cb7827a000) at webserver.cc:557:10
    frame #26: 0x000055b725adc67b yb-tserver`yb::Webserver::Impl::BeginRequestCallbackStatic(connection=0x000000cb7827a000) at webserver.cc:532:20
    frame #27: 0x000055b725ae3a87 yb-tserver`process_new_connection + 6119
    frame #28: 0x000055b725ae213a yb-tserver`worker_thread + 906
    frame #29: 0x00007f3104afc694 libpthread.so.0`start_thread(arg=0x00007f3031dde700) at pthread_create.c:333
    frame #30: 0x00007f3104df941d libc.so.6`__clone at clone.S:109

Source connector version

NA

Connector configuration

NA

YugabyteDB version

2.19.3.0-b136

Warning: Please confirm that this issue does not contain any sensitive information

shamanthchandra-yb commented 4 months ago

Duplicate of https://github.com/yugabyte/yugabyte-db/issues/20268