[flaky test] CqlIndexTest.TxnCleanup

[bmatican]

Jira Link: DB-813

Description

https://detective-gcp.dev.yugabyte.com/stability/test?branch=master&build_type=all&class=CqlIndexTest&name=TxnCleanup&platform=centos

Bad status: Runtime error (yb/integration-tests/cql_test_util.cc:270): Timed out waiting for prepare child status, left to deadline: 53.998s

So likely related to you recent work on PrepareChildTxns

[bmatican]

Seems like this test also is failing with a similar status http://detective.dev.yugabyte.com/stability/test?branch=master&class=org.yb.cql.TestIndex&name=testClusterRestart

[arybochkin]

This test is now failing into asan_heap_use_after_free pattern:

==111989==ERROR: AddressSanitizer: heap-use-after-free on address 0x6040009b6168 at pc 0x7f407f8097f9 bp 0x7f3ef3cb4d10 sp 0x7f3ef3cb4d08
READ of size 8 at 0x6040009b6168 thread T320 (rpc_tp_TabletSe)
    #0 0x7f407f8097f8 in std::__hash_table<std::__hash_value_type<string, yb::IndexInfo>, std::__unordered_map_hasher<string, std::__hash_value_type<string, yb::IndexInfo>, std::hash<string>, std::equal_to<string>, true>, std::__unordered_map_equal<string, std::__hash_value_type<string, yb::IndexInfo>, std::equal_to<string>, std::hash<string>, true>, std::allocator<std::__hash_value_type<string, yb::IndexInfo>>>::size[abi:v15003]() const /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221115221821-32d6b89b02-almalinux8-x86_64-clang15/installed/asan/libcxx/include/c++/v1/__hash_table:935:53
    #1 0x7f407f8097f8 in std::unordered_map<string, yb::IndexInfo, std::hash<string>, std::equal_to<string>, std::allocator<std::pair<string const, yb::IndexInfo>>>::empty[abi:v15003]() const /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221115221821-32d6b89b02-almalinux8-x86_64-clang15/installed/asan/libcxx/include/c++/v1/unordered_map:1185:56
    #2 0x7f407f8097f8 in yb::docdb::QLWriteOperation::GetDocPaths(yb::docdb::GetDocPathsMode, boost::container::small_vector_base<yb::RefCntPrefix, void, void>*, yb::IsolationLevel*) const ${BUILD_ROOT}/../../src/yb/docdb/cql_operation.cc:364:89
    #3 0x7f407f7c676d in yb::docdb::(anonymous namespace)::TransactionConflictResolverContext::ReadConflicts(yb::docdb::(anonymous namespace)::ConflictResolver*) ${BUILD_ROOT}/../../src/yb/docdb/conflict_resolution.cc:966:7

==65163==ERROR: AddressSanitizer: heap-use-after-free on address 0x60400097cf28 at pc 0x7eff78b207f9 bp 0x7efde7deed10 sp 0x7efde7deed08
READ of size 8 at 0x60400097cf28 thread T327 (rpc_tp_TabletSe)
INSERT INTO t (key, value) VALUES (?, ?)
       ^^^^
    #0 0x7eff78b207f8 in std::__hash_table<std::__hash_value_type<string, yb::IndexInfo>, std::__unordered_map_hasher<string, std::__hash_value_type<string, yb::IndexInfo>, std::hash<string>, std::equal_to<string>, true>, std::__unordered_map_equal<string, std::__hash_value_type<string, yb::IndexInfo>, std::equal_to<string>, std::hash<string>, true>, std::allocator<std::__hash_value_type<string, yb::IndexInfo>>>::size[abi:v15003]() const /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221115221821-32d6b89b02-almalinux8-x86_64-clang15/installed/asan/libcxx/include/c++/v1/__hash_table:935:53
    #1 0x7eff78b207f8 in std::unordered_map<string, yb::IndexInfo, std::hash<string>, std::equal_to<string>, std::allocator<std::pair<string const, yb::IndexInfo>>>::empty[abi:v15003]() const /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221115221821-32d6b89b02-almalinux8-x86_64-clang15/installed/asan/libcxx/include/c++/v1/unordered_map:1185:56

==148358==ERROR: AddressSanitizer: heap-use-after-free on address 0x604000753a68 at pc 0x7f09d2daf7f9 bp 0x7f082cadad10 sp 0x7f082cadad08
READ of size 8 at 0x604000753a68 thread T342 (rpc_tp_TabletSe)
    10:key[int32 NOT NULL PARTITION KEY],
    11:value[int32 NULLABLE NOT A PARTITION KEY]
]
properties: contain_counters: false is_transactional: true consistency_level: STRONG use_mangled_column_name: false is_ysql_catalog_table: false retain_delete_markers: false partitioning_version: 1 version 3 to Schema [

Also similar pattern for https://detective-gcp.dev.yugabyte.com/stability/test?class=CqlIndexTest&name=TxnPollCleanup:

==149668==ERROR: AddressSanitizer: heap-use-after-free on address 0x604000763828 at pc 0x7ffbf3ea87f9 bp 0x7ffb9a5ded10 sp 0x7ffb9a5ded08
READ of size 8 at 0x604000763828 thread T70 (rpc_tp_TabletSe)
    10:key[int32 NOT NULL PARTITION KEY],
    11:value[int32 NULLABLE NOT A PARTITION KEY]
]
properties: contain_counters: false is_transactional: true consistency_level: STRONG use_mangled_column_name: false is_ysql_catalog_table: false retain_delete_markers: false partitioning_version: 1 version 3 to Schema [

It is looking like a bad signal, maybe worth prioritising @bmatican @lnguyen-yugabyte @tverona1

[bmatican]

Passing to Yusong as we assume it might be related to his ScopedOperation change on schema changes

[NatashaSerebryanaya]

CppCassandraDriverTest.TestTableCreateUniqueIndexCovered fails with the same stack trace https://detective.dev.yugabyte.com/?commits=8f596820290804084c85bd3ed39c4d049e67a7ab

[rthallamko3]

The more recent failure is different from the fixes that @yusong-yan made.

messenger.cc:221] Scheduled tasks is not empty after messenger shutdown: [{9, 0x000055c11b0d84f8 -> { id: 9 source: ../../src/yb/rpc/rpc.cc:170 }}] | Log: https://jenkins.dev.yugabyte.com/job/github-yugabyte-db-centos-master-clang15-debug/763/artifact/build/debug-clang15-dynamic-ninja/yb-test-logs/tests-integration-tests__cql-index-test/CqlIndexTest_TxnCleanup.log    messenger.cc:221] Scheduled tasks is not empty after messenger shutdown: [{9, 0x000055c11b0d84f8 -> { id: 9 source: ../../src/yb/rpc/rpc.cc:170 }}] @ 0x7fd7ae5762c9 @ 0x7fd7ae576d2f @ 0x7fd7ae576b51 @ 0x7fd7acaef554 @ 0x55c1082dced8 @ 0x7fd7ae57670f @ 0x7fd7ae576d2f @ 0x7fd7ae576b51 @ 0x7fd7acaef554 @ 0x55c1082dced8
-- | --

[rthallamko3]

The more recent failures are in cql

Bad status: Runtime error (yb/integration-tests/cql_test_util.cc:276): Timed out waiting for prepare child status, left to deadline: 53.944s