Open deeps1991 opened 1 year ago
From the trends report, it doesn't look like the tsan runs are failing anymore. From the mac failures, it seems like the failure is due to performance of the query being within an SLA. Not sure if the perf tests should be validated on mac builds. cc @deeps1991
15:45:59.696 (main) [ERROR - org.yb.BaseYBTest$1$2.logEventDetails(BaseYBTest.java:243)] YB Java test failed: class="org.yb.pgsql.TestPgColumnReadEfficiency", method="testScans"
java.lang.AssertionError: Expected median time 242.398000 of 'SELECT * FROM t WHERE v1 = 1' to be greater than median time 244.551000 of 'SELECT h FROM t WHERE v1 = 1'
@rthallamko3 TSAN has not been enabled yet due to few other general issues that are still in code review. With all the TSAN fixes in place in https://phabricator.dev.yugabyte.com/D20730 I saw the test fail with the given FATAL above. As of now the only way to repro the above failure is to run the test manually with TSAN in a linux machine. Since it is a FATAL, I opened the issue early on so that investigation of this issue will not be delayed until all the TSAN fixes are code reviewed and checked in. Thus you will not see the failure in the trends report.
@jmeehan16 , Is this a known issue with frontiers? This is not urgent, but just checking in case you had seen this before.
@jmeehan16 JFYI if you want to try to repro the issue locally on your dev-server (as we still don't have Detective runs for YSQL tests in TSAN yet) you can remove the YB_DISABLE_TEST_TSAN marker in the test file for this test and just run the test as usual.
To clarify, this is a DFATAL, so it is strictly a debug issue, but still worth investigating if there's some invariants broken to get to this state.
YB_LOG_EVERY_N_SECS(DFATAL, 5)
<< db_options_.log_prefix << "[" << cfd->GetName()
<< "] " << ToString(type) << " frontier is not initialized for non-empty MemTable";
Saw also in TestPgRegressSecondaryIndexScan (twice) https://detective-gcp.dev.yugabyte.com/job/yugabyte-db-phabricator%2F167188%2Fartifact%2Fjava%2Fyb-pgsql%2Ftarget%2Fsurefire-reports_org.yb.pgsql.TestPgRegressSecondaryIndexScan__testPgRegressSecondaryIndexScan%2Forg.yb.pgsql.TestPgRegressSecondaryIndexScan-output.txt.gz?class=org.yb.pgsql.TestPgRegressSecondaryIndexScan&name=testPgRegressSecondaryIndexScan
ts1|pid8160|:20050 Fatal failure details written to ${YB_SRC_ROOT}/java/yb-pgsql/target/surefire-reports_org.yb.pgsql.TestPgRegressSecondaryIndexScan__testPgRegressSecondaryIndexScan/org.yb.pgsql.TestPgRegressSecondaryIndexScan.testPgRegressSecondaryIndexScan.fatal_failure_details.ts-1.127.177.199.153-port20050.2023-07-20T05_43_06.pid8160.txt
ts1|pid8160|:20050 F20230720 05:43:06 ../../src/yb/rocksdb/db/db_impl.cc:6085] T {airport_type_region_idx_tablet_id1} P {ts1_peer_id} [R]: [default] kSmallest frontier is not initialized for non-empty MemTable
ts1|pid8160|:20050 ${BUILD_ROOT}/../../src/yb/util/logging.cc:465: @ 0x7ff6602da1de yb::LogFatalHandlerSink::send(int, char const*, char const*, int, tm const*, char const*, unsigned long)
ts1|pid8160|:20050 @ 0x7ff65f06f031
ts1|pid8160|:20050 @ 0x7ff65f06ff35
ts1|pid8160|:20050 @ 0x7ff65f06fc11
ts1|pid8160|:20050 ${BUILD_ROOT}/../../src/yb/rocksdb/db/db_impl.cc:6085: @ 0x7ff6642a9039 rocksdb::DBImpl::GetMutableMemTableFrontier(rocksdb::UpdateUserValueType)
ts1|pid8160|:20050 ${BUILD_ROOT}/../../src/yb/tablet/tablet.cc:368: @ 0x7ff664d79c9b yb::tablet::(anonymous namespace)::MemTableFrontierFromDb(rocksdb::DB*, rocksdb::UpdateUserValueType)
ts1|pid8160|:20050 ${BUILD_ROOT}/../../src/yb/tablet/tablet.cc:3278: @ 0x7ff664d79c9b yb::tablet::Tablet::OldestMutableMemtableWriteHybridTime() const
ts1|pid8160|:20050 ${BUILD_ROOT}/../../src/yb/tserver/tablet_memory_manager.cc:331: @ 0x7ff6651995b7 yb::tserver::TabletMemoryManager::TabletToFlush()
ts1|pid8160|:20050 ${BUILD_ROOT}/../../src/yb/tserver/tablet_memory_manager.cc:300: @ 0x7ff665198dd7 yb::tserver::TabletMemoryManager::FlushTabletIfLimitExceeded()
ts1|pid8160|:20050 ${BUILD_ROOT}/../../src/yb/tserver/tablet_memory_manager.cc:236: @ 0x7ff66519c393 yb::tserver::TabletMemoryManager::ConfigureBackgroundTask(yb::tablet::TabletOptions*)::$_1::operator()() const
ts1|pid8160|:20050 /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230627183716-a7f6643a59-centos7-x86_64-clang15/installed/tsan/libcxx/include/c++/v1/__functional/invoke.h:394: @ 0x7ff66519c393 decltype(std::declval<yb::tserver::TabletMemoryManager::ConfigureBackgroundTask(yb::tablet::TabletOptions*)::$_1&>()()) std::__invoke[abi:v15007]<yb::tserver::TabletMemoryManager::ConfigureBackgroundTask(yb::tablet::TabletOptions*)::$_1&>(yb::tserver::TabletMemoryManager::ConfigureBackgroundTask(yb::tablet::TabletOptions*)::$_1&)
ts1|pid8160|:20050 /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230627183716-a7f6643a59-centos7-x86_64-clang15/installed/tsan/libcxx/include/c++/v1/__functional/invoke.h:479: @ 0x7ff66519c393 void std::__invoke_void_return_wrapper<void, true>::__call<yb::tserver::TabletMemoryManager::ConfigureBackgroundTask(yb::tablet::TabletOptions*)::$_1&>(yb::tserver::TabletMemoryManager::ConfigureBackgroundTask(yb::tablet::TabletOptions*)::$_1&)
ts1|pid8160|:20050 /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230627183716-a7f6643a59-centos7-x86_64-clang15/installed/tsan/libcxx/include/c++/v1/__functional/function.h:185: @ 0x7ff66519c393 std::__function::__alloc_func<yb::tserver::TabletMemoryManager::ConfigureBackgroundTask(yb::tablet::TabletOptions*)::$_1, std::allocator<yb::tserver::TabletMemoryManager::ConfigureBackgroundTask(yb::tablet::TabletOptions*)::$_1>, void ()>::operator()[abi:v15007]()
ts1|pid8160|:20050 /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230627183716-a7f6643a59-centos7-x86_64-clang15/installed/tsan/libcxx/include/c++/v1/__functional/function.h:359: @ 0x7ff66519c393 std::__function::__func<yb::tserver::TabletMemoryManager::ConfigureBackgroundTask(yb::tablet::TabletOptions*)::$_1, std::allocator<yb::tserver::TabletMemoryManager::ConfigureBackgroundTask(yb::tablet::TabletOptions*)::$_1>, void ()>::operator()()
ts1|pid8160|:20050 /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230627183716-a7f6643a59-centos7-x86_64-clang15/installed/tsan/libcxx/include/c++/v1/__functional/function.h:512: @ 0x7ff6601ffff4 std::__function::__value_func<void ()>::operator()[abi:v15007]() const
ts1|pid8160|:20050 /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230627183716-a7f6643a59-centos7-x86_64-clang15/installed/tsan/libcxx/include/c++/v1/__functional/function.h:1197: @ 0x7ff6601ffff4 std::function<void ()>::operator()() const
ts1|pid8160|:20050 ${BUILD_ROOT}/../../src/yb/util/background_task.cc:66: @ 0x7ff6601ffff4 yb::BackgroundTask::Run()
ts1|pid8160|:20050 /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230627183716-a7f6643a59-centos7-x86_64-clang15/installed/tsan/libcxx/include/c++/v1/__functional/invoke.h:359: @ 0x7ff66020085b decltype(*std::declval<yb::BackgroundTask*&>().*std::declval<void (yb::BackgroundTask::*&)()>()()) std::__invoke[abi:v15007]<void (yb::BackgroundTask::*&)(), yb::BackgroundTask*&, void>(void (yb::BackgroundTask::*&)(), yb::BackgroundTask*&)
ts1|pid8160|:20050 /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230627183716-a7f6643a59-centos7-x86_64-clang15/installed/tsan/libcxx/include/c++/v1/__functional/bind.h:263: @ 0x7ff66020085b std::__bind_return<void (yb::BackgroundTask::*)(), tuple<yb::BackgroundTask*>, tuple<>, __is_valid_bind_return<void (yb::BackgroundTask::*)(), tuple<yb::BackgroundTask*>, tuple<>>::value>::type std::__apply_functor[abi:v15007]<void (yb::BackgroundTask::*)(), tuple<yb::BackgroundTask*>, 0ul, tuple<>>(void (yb::BackgroundTask::*&)(), tuple<yb::BackgroundTask*>&, std::__tuple_indices<0ul>, tuple<>&&)
ts1|pid8160|:20050 /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230627183716-a7f6643a59-centos7-x86_64-clang15/installed/tsan/libcxx/include/c++/v1/__functional/bind.h:295: @ 0x7ff66020085b std::__bind_return<void (yb::BackgroundTask::*)(), tuple<yb::BackgroundTask*>, tuple<>, __is_valid_bind_return<void (yb::BackgroundTask::*)(), tuple<yb::BackgroundTask*>, tuple<>>::value>::type std::__bind<void (yb::BackgroundTask::* const&)(), yb::BackgroundTask* const&>::operator()[abi:v15007]<>()
ts1|pid8160|:20050 /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230627183716-a7f6643a59-centos7-x86_64-clang15/installed/tsan/libcxx/include/c++/v1/__functional/invoke.h:394: @ 0x7ff66020085b decltype(std::declval<std::__bind<void (yb::BackgroundTask::* const&)(), yb::BackgroundTask* const&>&>()()) std::__invoke[abi:v15007]<std::__bind<void (yb::BackgroundTask::* const&)(), yb::BackgroundTask* const&>&>(std::__bind<void (yb::BackgroundTask::* const&)(), yb::BackgroundTask* const&>&)
ts1|pid8160|:20050 /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230627183716-a7f6643a59-centos7-x86_64-clang15/installed/tsan/libcxx/include/c++/v1/__functional/invoke.h:479: @ 0x7ff66020085b void std::__invoke_void_return_wrapper<void, true>::__call<std::__bind<void (yb::BackgroundTask::* const&)(), yb::BackgroundTask* const&>&>(std::__bind<void (yb::BackgroundTask::* const&)(), yb::BackgroundTask* const&>&)
ts1|pid8160|:20050 /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230627183716-a7f6643a59-centos7-x86_64-clang15/installed/tsan/libcxx/include/c++/v1/__functional/function.h:185: @ 0x7ff66020085b std::__function::__alloc_func<std::__bind<void (yb::BackgroundTask::* const&)(), yb::BackgroundTask* const&>, std::allocator<std::__bind<void (yb::BackgroundTask::* const&)(), yb::BackgroundTask* const&>>, void ()>::operator()[abi:v15007]()
ts1|pid8160|:20050 /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230627183716-a7f6643a59-centos7-x86_64-clang15/installed/tsan/libcxx/include/c++/v1/__functional/function.h:359: @ 0x7ff66020085b std::__function::__func<std::__bind<void (yb::BackgroundTask::* const&)(), yb::BackgroundTask* const&>, std::allocator<std::__bind<void (yb::BackgroundTask::* const&)(), yb::BackgroundTask* const&>>, void ()>::operator()()
ts1|pid8160|:20050 /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230627183716-a7f6643a59-centos7-x86_64-clang15/installed/tsan/libcxx/include/c++/v1/__functional/function.h:512: @ 0x7ff6603d319a std::__function::__value_func<void ()>::operator()[abi:v15007]() const
ts1|pid8160|:20050 /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230627183716-a7f6643a59-centos7-x86_64-clang15/installed/tsan/libcxx/include/c++/v1/__functional/function.h:1197: @ 0x7ff6603d319a std::function<void ()>::operator()() const
ts1|pid8160|:20050 ${BUILD_ROOT}/../../src/yb/util/thread.cc:842: @ 0x7ff6603d319a yb::Thread::SuperviseThread(void*)
ts1|pid8160|:20050 /opt/yb-build/llvm/yb-llvm-v15.0.7-yb-1-1680596282-6b9d30d8-centos7-x86_64-build/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1012: @ 0x55630f3a2036 __tsan_thread_start_func
ts1|pid8160|:20050 @ 0x7ff65e57fea4
ts1|pid8160|:20050 @ 0x7ff65e08db0c
ts1|pid8160|:20050
ts1|pid8160|:20050 *** Check failure stack trace: ***
Jira Link: DB-5344
From the trends report, it doesn't look like the tsan runs are failing anymore. From the mac failures, it seems like the failure is due to performance of the query being within an SLA. Not sure if the perf tests should be validated on mac builds.
Old issue (No longer reproducible):
Description
org.yb.pgsql.TestPgColumnReadEfficiency-output.txt