search

yugabyte / yugabyte-db

YugabyteDB - the cloud native distributed SQL database for mission-critical applications.
https://www.yugabyte.com
Other
8.68k stars 1.04k forks source link

[YSQL] flaky test: PgDDLConcurrencyTest.IndexCreation #20517

Open bmatican opened 6 months ago

bmatican commented 6 months ago

Jira Link: DB-9523

Description

https://detective-gcp.dev.yugabyte.com/stability/test?analyze_trends=true&branch=master&build_type=all&class=PgDDLConcurrencyTest&fail_tag=all&name=IndexCreation&num_commits=500&platform=linux

Issue Type

kind/bug

Warning: Please confirm that this issue does not contain any sensitive information

d-uspenskiy commented 6 months ago

The test fails due to following ThreadSanitizer warning:

WARNING: ThreadSanitizer: data race (pid=38656) Read of size 8 at 0x7f6f2a44b038 by thread T21:

0 memcpy /opt/yb-build/llvm/yb-llvm-v17.0.2-yb-1-1696896765-6a83e4b2-almalinux8-x86_64-build/src/llvm-project/compiler-rt/lib/tsan/rtl/../../sanitizer_common/sanitizer_common_interceptors_memintrinsics.inc:115:5 (pg_ddl_concurrency-test+0x9e197)

#1 <null> <null> (libnss_sss.so.2+0x72ef) (BuildId: a17afeaa37369696ec2457ab7a311139707fca9b)
#2 pqGetpwuid ${YB_SRC_ROOT}/src/postgres/src/interfaces/libpq/thread.c:99:9 (libpq.so.5+0x4a8c9)
#3 pqGetHomeDirectory ${YB_SRC_ROOT}/src/postgres/src/interfaces/libpq/../../../../../../src/postgres/src/interfaces/libpq/fe-connect.c:6674:9 (libpq.so.5+0x2d3c7)
#4 connectOptions2 ${YB_SRC_ROOT}/src/postgres/src/interfaces/libpq/../../../../../../src/postgres/src/interfaces/libpq/fe-connect.c:1150:8 (libpq.so.5+0x2d3c7)
#5 PQconnectStart ${YB_SRC_ROOT}/src/postgres/src/interfaces/libpq/../../../../../../src/postgres/src/interfaces/libpq/fe-connect.c:791:7 (libpq.so.5+0x2c2fe)
#6 PQconnectdb ${YB_SRC_ROOT}/src/postgres/src/interfaces/libpq/../../../../../../src/postgres/src/interfaces/libpq/fe-connect.c:647:20 (libpq.so.5+0x2c279)
#7 yb::pgwrapper::PGConn::Connect(string const&, std::chrono::time_point<yb::CoarseMonoClock, std::chrono::duration<long long, std::ratio<1l, 1000000000l>>>, bool, string const&) ${BUILD_ROOT}/../../src/yb/yql/pgwrapper/libpq_utils.cc:278:24 (libpq_utils.so+0x11d6b)
#8 yb::pgwrapper::PGConn::Connect(string const&, bool, string const&) ${BUILD_ROOT}/../../src/yb/yql/pgwrapper/libpq_utils.h:234:12 (libpq_utils.so+0x17e2e)
#9 yb::pgwrapper::PGConnBuilder::Connect(bool) const ${BUILD_ROOT}/../../src/yb/yql/pgwrapper/libpq_utils.cc:657:10 (libpq_utils.so+0x17e2e)
#10 yb::pgwrapper::LibPqTestBase::ConnectToDBAsUser(string const&, string const&, bool) ${BUILD_ROOT}/../../src/yb/yql/pgwrapper/libpq_test_base.cc:54:6 (libpg_wrapper_test_base.so+0x23f94)

[ts-1] I0109 12:58:31.074257 39326 raft_consensus.cc:1745] T {transactions_tablet_id3} P {ts1_peer_id} [term 1 FOLLOWER]: Refusing update from remote peer {ts2_peer_id}: Log matching property violated. Preceding OpId in replica: 0.0. Preceding OpId from leader: 1.1. (index mismatch) [ts-1] #11 yb::pgwrapper::LibPqTestBase::ConnectToDB(string const&, bool) ${BUILD_ROOT}/../../src/yb/yql/pgwrapper/libpq_test_base.cc:44:10 (libpg_wrapper_test_base.so+0x23b7e) I0109 12:58:31.074357 39326 raft_consensus.cc:1961] T {transactions_tablet_id3} P {ts1_peer_id} [term 1 FOLLOWER]: Returning from UpdateConsensus because of error: error { code: PRECEDING_ENTRY_DIDNT_MATCH status { code: ILLEGAL_STATE message: "Log matching property violated. Preceding OpId in replica: 0.0. Preceding OpId from leader: 1.1. (index mismatch)" source_file: "../../src/yb/consensus/raft_consensus.cc" source_line: 1743 errors: "00" } }

12 yb::pgwrapper::LibPqTestBase::Connect(bool) ${BUILD_ROOT}/../../src/yb/yql/pgwrapper/libpq_test_base.cc:40:10 (libpg_wrapper_test_base.so+0x23b7e)

#13 yb::pgwrapper::PgDDLConcurrencyTest_IndexCreation_Test::TestBody()::$_0::operator()() const ${BUILD_ROOT}/../../src/yb/yql/pgwrapper/pg_ddl_concurrency-test.cc:66:23 (pg_ddl_concurrency-test+0x12c4e4)
#14 void yb::TestThreadHolder::AddThreadFunctor<yb::pgwrapper::PgDDLConcurrencyTest_IndexCreation_Test::TestBody()::$_0>(yb::pgwrapper::PgDDLConcurrencyTest_IndexCreation_Test::TestBody()::$_0 const&)::'lambda'()::operator()() const ${BUILD_ROOT}/../../src/yb/util/test_thread_holder.h:66:7 (pg_ddl_concurrency-test+0x12c4e4)
#15 decltype(std::declval<yb::pgwrapper::PgDDLConcurrencyTest_IndexCreation_Test::TestBody()::$_0>()()) std::__invoke[abi:v170002]<void yb::TestThreadHolder::AddThreadFunctor<yb::pgwrapper::PgDDLConcurrencyTest_IndexCreation_Test::TestBody()::$_0>(yb::pgwrapper::PgDDLConcurrencyTest_IndexCreation_Test::TestBody()::$_0 const&)::'lambda'()>(yb::pgwrapper::PgDDLConcurrencyTest_IndexCreation_Test::TestBody()::$_0&&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231120040932-e307bff3a7-almalinux8-x86_64-clang17/installed/tsan/libcxx/include/c++/v1/__type_traits/invoke.h:340:25 (pg_ddl_concurrency-test+0x12c4e4)
#16 void std::__thread_execute[abi:v170002]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void yb::TestThreadHolder::AddThreadFunctor<yb::pgwrapper::PgDDLConcurrencyTest_IndexCreation_Test::TestBody()::$_0>(yb::pgwrapper::PgDDLConcurrencyTest_IndexCreation_Test::TestBody()::$_0 const&)::'lambda'()>(std::tuple<yb::pgwrapper::PgDDLConcurrencyTest_IndexCreation_Test::TestBody()::$_0, void yb::TestThreadHolder::AddThreadFunctor<yb::pgwrapper::PgDDLConcurrencyTest_IndexCreation_Test::TestBody()::$_0>(yb::pgwrapper::PgDDLConcurrencyTest_IndexCreation_Test::TestBody()::$_0 const&)::'lambda'()>&, std::__tuple_indices<>) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231120040932-e307bff3a7-almalinux8-x86_64-clang17/installed/tsan/libcxx/include/c++/v1/__thread/thread.h:221:5 (pg_ddl_concurrency-test+0x12c4e4)
#17 void* std::__thread_proxy[abi:v170002]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void yb::TestThreadHolder::AddThreadFunctor<yb::pgwrapper::PgDDLConcurrencyTest_IndexCreation_Test::TestBody()::$_0>(yb::pgwrapper::PgDDLConcurrencyTest_IndexCreation_Test::TestBody()::$_0 const&)::'lambda'()>>(void*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231120040932-e307bff3a7-almalinux8-x86_64-clang17/installed/tsan/libcxx/include/c++/v1/__thread/thread.h:232:5 (pg_ddl_concurrency-test+0x12c4e4)

Previous write of size 8 at 0x7f6f2a44b038 by thread T20 (mutexes: write M0):

0 mmap64 /opt/yb-build/llvm/yb-llvm-v17.0.2-yb-1-1696896765-6a83e4b2-almalinux8-x86_64-build/src/llvm-project/compiler-rt/lib/tsan/rtl/../../sanitizer_common/sanitizer_common_interceptors.inc:7485:3 (pg_ddl_concurrency-test+0xda204)

#1 <null> <null> (libnss_sss.so.2+0x7169) (BuildId: a17afeaa37369696ec2457ab7a311139707fca9b)
#2 pqGetpwuid ${YB_SRC_ROOT}/src/postgres/src/interfaces/libpq/thread.c:99:9 (libpq.so.5+0x4a8c9)
#3 pqGetHomeDirectory ${YB_SRC_ROOT}/src/postgres/src/interfaces/libpq/../../../../../../src/postgres/src/interfaces/libpq/fe-connect.c:6674:9 (libpq.so.5+0x2d3c7)
#4 connectOptions2 ${YB_SRC_ROOT}/src/postgres/src/interfaces/libpq/../../../../../../src/postgres/src/interfaces/libpq/fe-connect.c:1150:8 (libpq.so.5+0x2d3c7)
#5 PQconnectStart ${YB_SRC_ROOT}/src/postgres/src/interfaces/libpq/../../../../../../src/postgres/src/interfaces/libpq/fe-connect.c:791:7 (libpq.so.5+0x2c2fe)
#6 PQconnectdb ${YB_SRC_ROOT}/src/postgres/src/interfaces/libpq/../../../../../../src/postgres/src/interfaces/libpq/fe-connect.c:647:20 (libpq.so.5+0x2c279)
#7 yb::pgwrapper::PGConn::Connect(string const&, std::chrono::time_point<yb::CoarseMonoClock, std::chrono::duration<long long, std::ratio<1l, 1000000000l>>>, bool, string const&) ${BUILD_ROOT}/../../src/yb/yql/pgwrapper/libpq_utils.cc:278:24 (libpq_utils.so+0x11d6b)
#8 yb::pgwrapper::PGConn::Connect(string const&, bool, string const&) ${BUILD_ROOT}/../../src/yb/yql/pgwrapper/libpq_utils.h:234:12 (libpq_utils.so+0x17e2e)
#9 yb::pgwrapper::PGConnBuilder::Connect(bool) const ${BUILD_ROOT}/../../src/yb/yql/pgwrapper/libpq_utils.cc:657:10 (libpq_utils.so+0x17e2e)
#10 yb::pgwrapper::LibPqTestBase::ConnectToDBAsUser(string const&, string const&, bool) ${BUILD_ROOT}/../../src/yb/yql/pgwrapper/libpq_test_base.cc:54:6 (libpg_wrapper_test_base.so+0x23f94)
#11 yb::pgwrapper::LibPqTestBase::ConnectToDB(string const&, bool) ${BUILD_ROOT}/../../src/yb/yql/pgwrapper/libpq_test_base.cc:44:10 (libpg_wrapper_test_base.so+0x23b7e)
#12 yb::pgwrapper::LibPqTestBase::Connect(bool) ${BUILD_ROOT}/../../src/yb/yql/pgwrapper/libpq_test_base.cc:40:10 (libpg_wrapper_test_base.so+0x23b7e)
#13 yb::pgwrapper::PgDDLConcurrencyTest_IndexCreation_Test::TestBody()::$_0::operator()() const ${BUILD_ROOT}/../../src/yb/yql/pgwrapper/pg_ddl_concurrency-test.cc:66:23 (pg_ddl_concurrency-test+0x12c4e4)
#14 void yb::TestThreadHolder::AddThreadFunctor<yb::pgwrapper::PgDDLConcurrencyTest_IndexCreation_Test::TestBody()::$_0>(yb::pgwrapper::PgDDLConcurrencyTest_IndexCreation_Test::TestBody()::$_0 const&)::'lambda'()::operator()() const ${BUILD_ROOT}/../../src/yb/util/test_thread_holder.h:66:7 (pg_ddl_concurrency-test+0x12c4e4)
#15 decltype(std::declval<yb::pgwrapper::PgDDLConcurrencyTest_IndexCreation_Test::TestBody()::$_0>()()) std::__invoke[abi:v170002]<void yb::TestThreadHolder::AddThreadFunctor<yb::pgwrapper::PgDDLConcurrencyTest_IndexCreation_Test::TestBody()::$_0>(yb::pgwrapper::PgDDLConcurrencyTest_IndexCreation_Test::TestBody()::$_0 const&)::'lambda'()>(yb::pgwrapper::PgDDLConcurrencyTest_IndexCreation_Test::TestBody()::$_0&&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231120040932-e307bff3a7-almalinux8-x86_64-clang17/installed/tsan/libcxx/include/c++/v1/__type_traits/invoke.h:340:25 (pg_ddl_concurrency-test+0x12c4e4)
#16 void std::__thread_execute[abi:v170002]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void yb::TestThreadHolder::AddThreadFunctor<yb::pgwrapper::PgDDLConcurrencyTest_IndexCreation_Test::TestBody()::$_0>(yb::pgwrapper::PgDDLConcurrencyTest_IndexCreation_Test::TestBody()::$_0 const&)::'lambda'()>(std::tuple<yb::pgwrapper::PgDDLConcurrencyTest_IndexCreation_Test::TestBody()::$_0, void yb::TestThreadHolder::AddThreadFunctor<yb::pgwrapper::PgDDLConcurrencyTest_IndexCreation_Test::TestBody()::$_0>(yb::pgwrapper::PgDDLConcurrencyTest_IndexCreation_Test::TestBody()::$_0 const&)::'lambda'()>&, std::__tuple_indices<>) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231120040932-e307bff3a7-almalinux8-x86_64-clang17/installed/tsan/libcxx/include/c++/v1/__thread/thread.h:221:5 (pg_ddl_concurrency-test+0x12c4e4)
#17 void* std::__thread_proxy[abi:v170002]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void yb::TestThreadHolder::AddThreadFunctor<yb::pgwrapper::PgDDLConcurrencyTest_IndexCreation_Test::TestBody()::$_0>(yb::pgwrapper::PgDDLConcurrencyTest_IndexCreation_Test::TestBody()::$_0 const&)::'lambda'()>>(void*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20231120040932-e307bff3a7-almalinux8-x86_64-clang17/installed/tsan/libcxx/include/c++/v1/__thread/thread.h:232:5 (pg_ddl_concurrency-test+0x12c4e4)

Location is global '??' at 0x7f6f2a44b000 (passwd+0x38)

Mutex M0 (0x7f6f2af29380) created at:

0 pthread_mutex_lock /opt/yb-build/llvm/yb-llvm-v17.0.2-yb-1-1696896765-6a83e4b2-almalinux8-x86_64-build/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1339:3 (pg_ddl_concurrency-test+0xa464b)

#1 <null> <null> (libnss_sss.so.2+0x70d6) (BuildId: a17afeaa37369696ec2457ab7a311139707fca9b)
#2 pqGetpwuid ${YB_SRC_ROOT}/src/postgres/src/interfaces/libpq/thread.c:99:9 (libpq.so.5+0x4a8c9)