search

yugabyte / yugabyte-db

YugabyteDB - the cloud native distributed SQL database for mission-critical applications.
https://www.yugabyte.com
Other
8.96k stars 1.07k forks source link

[DocDB] TestPgColumnReadEfficiency FATALs on TSAN build #15931

Open deeps1991 opened 1 year ago

deeps1991 commented 1 year ago

Jira Link: DB-5344

From the trends report, it doesn't look like the tsan runs are failing anymore. From the mac failures, it seems like the failure is due to performance of the query being within an SLA. Not sure if the perf tests should be validated on mac builds.

Old issue (No longer reproducible):

Description

ts1|pid29665|:14653 F20230105 17:33:30 ../../src/yb/rocksdb/db/db_impl.cc:6056] T {t_v1_v2_v3_t1_t2_t3_t4_t5_idx_tablet_id1} P {ts1_peer_id} [I]: [default] for non-empty MemTable
ts1|pid29665|:14653 ${BUILD_ROOT}/../../src/yb/util/logging.cc:458:     @     0x7f2620613b5e  yb::LogFatalHandlerSink::send(int, char const*, char const*, int, tm const*, char const*, unsigned long)
ts1|pid29665|:14653     @     0x7f261f3c4bf1
ts1|pid29665|:14653     @     0x7f261f3c5af5
ts1|pid29665|:14653     @     0x7f261f3c57d1
ts1|pid29665|:14653 ${BUILD_ROOT}/../../src/yb/rocksdb/db/db_impl.cc:6056:     @     0x7f26240f77f9  rocksdb::DBImpl::GetMutableMemTableFrontier(rocksdb::UpdateUserValueType)
ts1|pid29665|:14653 ${BUILD_ROOT}/../../src/yb/tablet/tablet.cc:348:     @     0x7f2624a524eb  yb::tablet::(anonymous namespace)::MemTableFrontierFromDb(rocksdb::DB*, rocksdb::UpdateUserValueType)kSmallest frontier is not initialized 
ts1|pid29665|:14653 ${BUILD_ROOT}/../../src/yb/tablet/tablet.cc:3184:     @     0x7f2624a524eb  yb::tablet::Tablet::OldestMutableMemtableWriteHybridTime() const
ts1|pid29665|:14653 ${BUILD_ROOT}/../../src/yb/tserver/tablet_memory_manager.cc:323:     @     0x7f2624de6507  yb::tserver::TabletMemoryManager::TabletToFlush()
ts1|pid29665|:14653 ${BUILD_ROOT}/../../src/yb/tserver/tablet_memory_manager.cc:292:     @     0x7f2624de5d27  yb::tserver::TabletMemoryManager::FlushTabletIfLimitExceeded()
ts1|pid29665|:14653 ${BUILD_ROOT}/../../src/yb/tserver/tablet_memory_manager.cc:232:     @     0x7f2624de90e3  yb::tserver::TabletMemoryManager::ConfigureBackgroundTask(yb::tablet::TabletOptions*)::$_1::operator()() const
ts1|pid29665|:14653 /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221115221900-32d6b89b02-centos7-x86_64-clang15/installed/tsan/libcxx/include/c++/v1/__functional/invoke.h:394:     @     0x7f2624de90e3  decltype(std::declval<yb::tserver::TabletMemoryManager::ConfigureBackgroundTask(yb::tablet::TabletOptions*)::$_1&>()()) std::__invoke[abi:v15003]<yb::tserver::TabletMemoryManager::ConfigureBackgroundTask(yb::tablet::TabletOptions*)::$_1&>(yb::tserver::TabletMemoryManager::ConfigureBackgroundTask(yb::tablet::TabletOptions*)::$_1&)
ts1|pid29665|:14653 /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221115221900-32d6b89b02-centos7-x86_64-clang15/installed/tsan/libcxx/include/c++/v1/__functional/invoke.h:479:     @     0x7f2624de90e3  void std::__invoke_void_return_wrapper<void, true>::__call<yb::tserver::TabletMemoryManager::ConfigureBackgroundTask(yb::tablet::TabletOptions*)::$_1&>(yb::tserver::TabletMemoryManager::ConfigureBackgroundTask(yb::tablet::TabletOptions*)::$_1&)
ts1|pid29665|:14653 /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221115221900-32d6b89b02-centos7-x86_64-clang15/installed/tsan/libcxx/include/c++/v1/__functional/function.h:185:     @     0x7f2624de90e3  std::__function::__alloc_func<yb::tserver::TabletMemoryManager::ConfigureBackgroundTask(yb::tablet::TabletOptions*)::$_1, std::allocator<yb::tserver::TabletMemoryManager::ConfigureBackgroundTask(yb::tablet::TabletOptions*)::$_1>, void ()>::operator()[abi:v15003]()
ts1|pid29665|:14653 /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221115221900-32d6b89b02-centos7-x86_64-clang15/installed/tsan/libcxx/include/c++/v1/__functional/function.h:359:     @     0x7f2624de90e3  std::__function::__func<yb::tserver::TabletMemoryManager::ConfigureBackgroundTask(yb::tablet::TabletOptions*)::$_1, std::allocator<yb::tserver::TabletMemoryManager::ConfigureBackgroundTask(yb::tablet::TabletOptions*)::$_1>, void ()>::operator()()
ts1|pid29665|:14653 /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221115221900-32d6b89b02-centos7-x86_64-clang15/installed/tsan/libcxx/include/c++/v1/__functional/function.h:512:     @     0x7f262053c094  std::__function::__value_func<void ()>::operator()[abi:v15003]() const
ts1|pid29665|:14653 /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221115221900-32d6b89b02-centos7-x86_64-clang15/installed/tsan/libcxx/include/c++/v1/__functional/function.h:1197:     @     0x7f262053c094  std::function<void ()>::operator()() const

org.yb.pgsql.TestPgColumnReadEfficiency-output.txt

rthallamko3 commented 1 year ago

From the trends report, it doesn't look like the tsan runs are failing anymore. From the mac failures, it seems like the failure is due to performance of the query being within an SLA. Not sure if the perf tests should be validated on mac builds. cc @deeps1991 

15:45:59.696 (main) [ERROR - org.yb.BaseYBTest$1$2.logEventDetails(BaseYBTest.java:243)] YB Java test failed: class="org.yb.pgsql.TestPgColumnReadEfficiency", method="testScans"
java.lang.AssertionError: Expected median time 242.398000 of 'SELECT * FROM t WHERE v1 = 1' to be greater than median time 244.551000 of 'SELECT h FROM t WHERE v1 = 1'
deeps1991 commented 1 year ago

@rthallamko3 TSAN has not been enabled yet due to few other general issues that are still in code review. With all the TSAN fixes in place in https://phabricator.dev.yugabyte.com/D20730 I saw the test fail with the given FATAL above. As of now the only way to repro the above failure is to run the test manually with TSAN in a linux machine. Since it is a FATAL, I opened the issue early on so that investigation of this issue will not be delayed until all the TSAN fixes are code reviewed and checked in. Thus you will not see the failure in the trends report.

rthallamko3 commented 1 year ago

@jmeehan16 , Is this a known issue with frontiers? This is not urgent, but just checking in case you had seen this before.

deeps1991 commented 1 year ago

@jmeehan16 JFYI if you want to try to repro the issue locally on your dev-server (as we still don't have Detective runs for YSQL tests in TSAN yet) you can remove the YB_DISABLE_TEST_TSAN marker in the test file for this test and just run the test as usual.

bmatican commented 1 year ago

To clarify, this is a DFATAL, so it is strictly a debug issue, but still worth investigating if there's some invariants broken to get to this state.

            YB_LOG_EVERY_N_SECS(DFATAL, 5)
                << db_options_.log_prefix << "[" << cfd->GetName()
                << "] " << ToString(type) << " frontier is not initialized for non-empty MemTable";
jasonyb commented 1 year ago

Saw also in TestPgRegressSecondaryIndexScan (twice) https://detective-gcp.dev.yugabyte.com/job/yugabyte-db-phabricator%2F167188%2Fartifact%2Fjava%2Fyb-pgsql%2Ftarget%2Fsurefire-reports_org.yb.pgsql.TestPgRegressSecondaryIndexScan__testPgRegressSecondaryIndexScan%2Forg.yb.pgsql.TestPgRegressSecondaryIndexScan-output.txt.gz?class=org.yb.pgsql.TestPgRegressSecondaryIndexScan&name=testPgRegressSecondaryIndexScan

ts1|pid8160|:20050 Fatal failure details written to ${YB_SRC_ROOT}/java/yb-pgsql/target/surefire-reports_org.yb.pgsql.TestPgRegressSecondaryIndexScan__testPgRegressSecondaryIndexScan/org.yb.pgsql.TestPgRegressSecondaryIndexScan.testPgRegressSecondaryIndexScan.fatal_failure_details.ts-1.127.177.199.153-port20050.2023-07-20T05_43_06.pid8160.txt
ts1|pid8160|:20050 F20230720 05:43:06 ../../src/yb/rocksdb/db/db_impl.cc:6085] T {airport_type_region_idx_tablet_id1} P {ts1_peer_id} [R]: [default] kSmallest frontier is not initialized for non-empty MemTable
ts1|pid8160|:20050 ${BUILD_ROOT}/../../src/yb/util/logging.cc:465:     @     0x7ff6602da1de  yb::LogFatalHandlerSink::send(int, char const*, char const*, int, tm const*, char const*, unsigned long)
ts1|pid8160|:20050     @     0x7ff65f06f031
ts1|pid8160|:20050     @     0x7ff65f06ff35
ts1|pid8160|:20050     @     0x7ff65f06fc11
ts1|pid8160|:20050 ${BUILD_ROOT}/../../src/yb/rocksdb/db/db_impl.cc:6085:     @     0x7ff6642a9039  rocksdb::DBImpl::GetMutableMemTableFrontier(rocksdb::UpdateUserValueType)
ts1|pid8160|:20050 ${BUILD_ROOT}/../../src/yb/tablet/tablet.cc:368:     @     0x7ff664d79c9b  yb::tablet::(anonymous namespace)::MemTableFrontierFromDb(rocksdb::DB*, rocksdb::UpdateUserValueType)
ts1|pid8160|:20050 ${BUILD_ROOT}/../../src/yb/tablet/tablet.cc:3278:     @     0x7ff664d79c9b  yb::tablet::Tablet::OldestMutableMemtableWriteHybridTime() const
ts1|pid8160|:20050 ${BUILD_ROOT}/../../src/yb/tserver/tablet_memory_manager.cc:331:     @     0x7ff6651995b7  yb::tserver::TabletMemoryManager::TabletToFlush()
ts1|pid8160|:20050 ${BUILD_ROOT}/../../src/yb/tserver/tablet_memory_manager.cc:300:     @     0x7ff665198dd7  yb::tserver::TabletMemoryManager::FlushTabletIfLimitExceeded()
ts1|pid8160|:20050 ${BUILD_ROOT}/../../src/yb/tserver/tablet_memory_manager.cc:236:     @     0x7ff66519c393  yb::tserver::TabletMemoryManager::ConfigureBackgroundTask(yb::tablet::TabletOptions*)::$_1::operator()() const
ts1|pid8160|:20050 /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230627183716-a7f6643a59-centos7-x86_64-clang15/installed/tsan/libcxx/include/c++/v1/__functional/invoke.h:394:     @     0x7ff66519c393  decltype(std::declval<yb::tserver::TabletMemoryManager::ConfigureBackgroundTask(yb::tablet::TabletOptions*)::$_1&>()()) std::__invoke[abi:v15007]<yb::tserver::TabletMemoryManager::ConfigureBackgroundTask(yb::tablet::TabletOptions*)::$_1&>(yb::tserver::TabletMemoryManager::ConfigureBackgroundTask(yb::tablet::TabletOptions*)::$_1&)
ts1|pid8160|:20050 /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230627183716-a7f6643a59-centos7-x86_64-clang15/installed/tsan/libcxx/include/c++/v1/__functional/invoke.h:479:     @     0x7ff66519c393  void std::__invoke_void_return_wrapper<void, true>::__call<yb::tserver::TabletMemoryManager::ConfigureBackgroundTask(yb::tablet::TabletOptions*)::$_1&>(yb::tserver::TabletMemoryManager::ConfigureBackgroundTask(yb::tablet::TabletOptions*)::$_1&)
ts1|pid8160|:20050 /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230627183716-a7f6643a59-centos7-x86_64-clang15/installed/tsan/libcxx/include/c++/v1/__functional/function.h:185:     @     0x7ff66519c393  std::__function::__alloc_func<yb::tserver::TabletMemoryManager::ConfigureBackgroundTask(yb::tablet::TabletOptions*)::$_1, std::allocator<yb::tserver::TabletMemoryManager::ConfigureBackgroundTask(yb::tablet::TabletOptions*)::$_1>, void ()>::operator()[abi:v15007]()
ts1|pid8160|:20050 /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230627183716-a7f6643a59-centos7-x86_64-clang15/installed/tsan/libcxx/include/c++/v1/__functional/function.h:359:     @     0x7ff66519c393  std::__function::__func<yb::tserver::TabletMemoryManager::ConfigureBackgroundTask(yb::tablet::TabletOptions*)::$_1, std::allocator<yb::tserver::TabletMemoryManager::ConfigureBackgroundTask(yb::tablet::TabletOptions*)::$_1>, void ()>::operator()()
ts1|pid8160|:20050 /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230627183716-a7f6643a59-centos7-x86_64-clang15/installed/tsan/libcxx/include/c++/v1/__functional/function.h:512:     @     0x7ff6601ffff4  std::__function::__value_func<void ()>::operator()[abi:v15007]() const
ts1|pid8160|:20050 /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230627183716-a7f6643a59-centos7-x86_64-clang15/installed/tsan/libcxx/include/c++/v1/__functional/function.h:1197:     @     0x7ff6601ffff4  std::function<void ()>::operator()() const
ts1|pid8160|:20050 ${BUILD_ROOT}/../../src/yb/util/background_task.cc:66:     @     0x7ff6601ffff4  yb::BackgroundTask::Run()
ts1|pid8160|:20050 /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230627183716-a7f6643a59-centos7-x86_64-clang15/installed/tsan/libcxx/include/c++/v1/__functional/invoke.h:359:     @     0x7ff66020085b  decltype(*std::declval<yb::BackgroundTask*&>().*std::declval<void (yb::BackgroundTask::*&)()>()()) std::__invoke[abi:v15007]<void (yb::BackgroundTask::*&)(), yb::BackgroundTask*&, void>(void (yb::BackgroundTask::*&)(), yb::BackgroundTask*&)
ts1|pid8160|:20050 /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230627183716-a7f6643a59-centos7-x86_64-clang15/installed/tsan/libcxx/include/c++/v1/__functional/bind.h:263:     @     0x7ff66020085b  std::__bind_return<void (yb::BackgroundTask::*)(), tuple<yb::BackgroundTask*>, tuple<>, __is_valid_bind_return<void (yb::BackgroundTask::*)(), tuple<yb::BackgroundTask*>, tuple<>>::value>::type std::__apply_functor[abi:v15007]<void (yb::BackgroundTask::*)(), tuple<yb::BackgroundTask*>, 0ul, tuple<>>(void (yb::BackgroundTask::*&)(), tuple<yb::BackgroundTask*>&, std::__tuple_indices<0ul>, tuple<>&&)
ts1|pid8160|:20050 /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230627183716-a7f6643a59-centos7-x86_64-clang15/installed/tsan/libcxx/include/c++/v1/__functional/bind.h:295:     @     0x7ff66020085b  std::__bind_return<void (yb::BackgroundTask::*)(), tuple<yb::BackgroundTask*>, tuple<>, __is_valid_bind_return<void (yb::BackgroundTask::*)(), tuple<yb::BackgroundTask*>, tuple<>>::value>::type std::__bind<void (yb::BackgroundTask::* const&)(), yb::BackgroundTask* const&>::operator()[abi:v15007]<>()
ts1|pid8160|:20050 /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230627183716-a7f6643a59-centos7-x86_64-clang15/installed/tsan/libcxx/include/c++/v1/__functional/invoke.h:394:     @     0x7ff66020085b  decltype(std::declval<std::__bind<void (yb::BackgroundTask::* const&)(), yb::BackgroundTask* const&>&>()()) std::__invoke[abi:v15007]<std::__bind<void (yb::BackgroundTask::* const&)(), yb::BackgroundTask* const&>&>(std::__bind<void (yb::BackgroundTask::* const&)(), yb::BackgroundTask* const&>&)
ts1|pid8160|:20050 /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230627183716-a7f6643a59-centos7-x86_64-clang15/installed/tsan/libcxx/include/c++/v1/__functional/invoke.h:479:     @     0x7ff66020085b  void std::__invoke_void_return_wrapper<void, true>::__call<std::__bind<void (yb::BackgroundTask::* const&)(), yb::BackgroundTask* const&>&>(std::__bind<void (yb::BackgroundTask::* const&)(), yb::BackgroundTask* const&>&)
ts1|pid8160|:20050 /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230627183716-a7f6643a59-centos7-x86_64-clang15/installed/tsan/libcxx/include/c++/v1/__functional/function.h:185:     @     0x7ff66020085b  std::__function::__alloc_func<std::__bind<void (yb::BackgroundTask::* const&)(), yb::BackgroundTask* const&>, std::allocator<std::__bind<void (yb::BackgroundTask::* const&)(), yb::BackgroundTask* const&>>, void ()>::operator()[abi:v15007]()
ts1|pid8160|:20050 /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230627183716-a7f6643a59-centos7-x86_64-clang15/installed/tsan/libcxx/include/c++/v1/__functional/function.h:359:     @     0x7ff66020085b  std::__function::__func<std::__bind<void (yb::BackgroundTask::* const&)(), yb::BackgroundTask* const&>, std::allocator<std::__bind<void (yb::BackgroundTask::* const&)(), yb::BackgroundTask* const&>>, void ()>::operator()()
ts1|pid8160|:20050 /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230627183716-a7f6643a59-centos7-x86_64-clang15/installed/tsan/libcxx/include/c++/v1/__functional/function.h:512:     @     0x7ff6603d319a  std::__function::__value_func<void ()>::operator()[abi:v15007]() const
ts1|pid8160|:20050 /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230627183716-a7f6643a59-centos7-x86_64-clang15/installed/tsan/libcxx/include/c++/v1/__functional/function.h:1197:     @     0x7ff6603d319a  std::function<void ()>::operator()() const
ts1|pid8160|:20050 ${BUILD_ROOT}/../../src/yb/util/thread.cc:842:     @     0x7ff6603d319a  yb::Thread::SuperviseThread(void*)
ts1|pid8160|:20050 /opt/yb-build/llvm/yb-llvm-v15.0.7-yb-1-1680596282-6b9d30d8-centos7-x86_64-build/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1012:     @     0x55630f3a2036  __tsan_thread_start_func
ts1|pid8160|:20050     @     0x7ff65e57fea4
ts1|pid8160|:20050     @     0x7ff65e08db0c
ts1|pid8160|:20050
ts1|pid8160|:20050 *** Check failure stack trace: ***