search

yugabyte / yugabyte-db

YugabyteDB - the cloud native distributed SQL database for mission-critical applications.
https://www.yugabyte.com
Other
8.85k stars 1.05k forks source link

[YSQL] PITR: Master metadata verified failed for restoration fatal in multiple tests #15927

Open NatashaSerebryanaya opened 1 year ago

NatashaSerebryanaya commented 1 year ago

Jira Link: DB-5340

Description

The fatal "PITR: Master metadata verified failed for restoration" is seen in multiple unit tests in debug builds.

[m-1] F0129 21:14:11.017827 20556 master_snapshot_coordinator.cc:883] PITR: Master metadata verified failed for restoration 432c73f8-6904-4bad-9a38-1b16df081a05, status: Illegal state (yb/master/catalog_manager.cc:1940): Not the leader. Local UUID: {m1_peer_id}, Consensus state: current_term: 2 config { opid_index: -1 peers { permanent_uuid: "{m1_peer_id}" member_type: VOTER last_known_private_addr { host: "127.0.0.1" port: 29644 } cloud_info { placement_cloud: "cloud1" placement_region: "datacenter1" placement_zone: "rack1" } } peers { permanent_uuid: "{m2_peer_id}" member_type: VOTER last_known_private_addr { host: "127.0.0.1" port: 29592 } cloud_info { placement_cloud: "cloud1" placement_region: "datacenter1" placement_zone: "rack1" } } peers { permanent_uuid: "{m3_peer_id}" member_type: VOTER last_known_private_addr { host: "127.0.0.1" port: 24494 } cloud_info { placement_cloud: "cloud1" placement_region: "datacenter1" placement_zone: "rack1" } } }
[m-1] Fatal failure details written to ${BUILD_ROOT}/yb-test-logs/tests-tools__yb-admin-snapshot-schedule-test/Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlAlterTableAddPK__0.fatal_failure_details.m-1.2023-01-29T21_14_11.pid19502.txt
F20230129 21:14:11 ../../src/yb/master/master_snapshot_coordinator.cc:883] PITR: Master metadata verified failed for restoration 432c73f8-6904-4bad-9a38-1b16df081a05, status: Illegal state (yb/master/catalog_manager.cc:1940): Not the leade
/nfusr/centos-gcp-cloud/jenkins-worker-k3440b/jenkins/jenkins-github-yugabyte-db-centos-master-clang15-debug-558/yugabyte-db/build/debug-clang15-dynamic-ninja/../../src/yb/util/logging.cc:458:     @     0x7f92a5dc8480  yb::LogFatalHandlerS
    @     0x7f92a5714599 
    @     0x7f92a5714d2f 
    @     0x7f92a5714b51 
/nfusr/centos-gcp-cloud/jenkins-worker-k3440b/jenkins/jenkins-github-yugabyte-db-centos-master-clang15-debug-558/yugabyte-db/build/debug-clang15-dynamic-ninja/../../src/yb/master/master_snapshot_coordinator.cc:883:     @     0x7f92ae0ce573
/nfusr/centos-gcp-cloud/jenkins-worker-k3440b/jenkins/jenkins-github-yugabyte-db-centos-master-clang15-debug-558/yugabyte-db/build/debug-clang15-dynamic-ninja/../../src/yb/master/master_snapshot_coordinator.cc:2087:     @     0x7f92ae0bd22
/nfusr/centos-gcp-cloud/jenkins-worker-k3440b/jenkins/jenkins-github-yugabyte-db-centos-master-clang15-debug-558/yugabyte-db/build/debug-clang15-dynamic-ninja/../../ent/src/yb/master/catalog_manager_ent.cc:7826:     @     0x7f92ae2c2ea4  y
/nfusr/centos-gcp-cloud/jenkins-worker-k3440b/jenkins/jenkins-github-yugabyte-db-centos-master-clang15-debug-558/yugabyte-db/build/debug-clang15-dynamic-ninja/../../src/yb/master/catalog_manager.cc:1088:     @     0x7f92add972a8  yb::maste
../../src/yb/gutil/bind_internal.h:148:                                                                 @     0x7f92adee2ca5  yb::internal::RunnableAdapter<void (yb::master::CatalogManager::*)()>::Run(yb::master::CatalogManager*)
../../src/yb/gutil/bind_internal.h:885:                                                                 @     0x7f92adee2c38  yb::internal::InvokeHelper<false, void, yb::internal::RunnableAdapter<void (yb::master::CatalogManager::*)()>, vo
../../src/yb/gutil/bind_internal.h:1077:                                                                @     0x7f92adee2c09  yb::internal::Invoker<1, yb::internal::BindState<yb::internal::RunnableAdapter<void (yb::master::CatalogManager::
../../src/yb/gutil/callback.h:411:                                                                      @     0x7f92ad1b62bd  yb::Callback<void ()>::Run() const
/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221115221900-32d6b89b02-centos7-x86_64-clang15/installed/uninstrumented/libcxx/include/c++/v1/__functional/invoke.h:343:     @     0x7f92a5f4a3a5  decltype(std::declval<yb::Callback<void (
/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221115221900-32d6b89b02-centos7-x86_64-clang15/installed/uninstrumented/libcxx/include/c++/v1/__functional/bind.h:263:     @     0x7f92a5f4a32c  std::__bind_return<void (yb::Callback<void
/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221115221900-32d6b89b02-centos7-x86_64-clang15/installed/uninstrumented/libcxx/include/c++/v1/__functional/bind.h:295:     @     0x7f92a5f516bf  std::__bind_return<void (yb::Callback<void
../../src/yb/util/threadpool.h:78:                                                                      @     0x7f92a5f51628  yb::RunnableImpl<std::__bind<void (yb::Callback<void ()>::*)() const, yb::Callback<void ()> const&>>::Run()
/nfusr/centos-gcp-cloud/jenkins-worker-k3440b/jenkins/jenkins-github-yugabyte-db-centos-master-clang15-debug-558/yugabyte-db/build/debug-clang15-dynamic-ninja/../../src/yb/util/threadpool.cc:612:     @     0x7f92a5f4342e  yb::ThreadPool::D
/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221115221900-32d6b89b02-centos7-x86_64-clang15/installed/uninstrumented/libcxx/include/c++/v1/__functional/invoke.h:359:     @     0x7f92a5f57b28  decltype(*std::declval<yb::ThreadPool*&>(
/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221115221900-32d6b89b02-centos7-x86_64-clang15/installed/uninstrumented/libcxx/include/c++/v1/__functional/bind.h:263:     @     0x7f92a5f57aa9  std::__bind_return<void (yb::ThreadPool::*)
/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221115221900-32d6b89b02-centos7-x86_64-clang15/installed/uninstrumented/libcxx/include/c++/v1/__functional/bind.h:295:     @     0x7f92a5f57a3f  std::__bind_return<void (yb::ThreadPool::*)
/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221115221900-32d6b89b02-centos7-x86_64-clang15/installed/uninstrumented/libcxx/include/c++/v1/__functional/invoke.h:394:     @     0x7f92a5f57a14  decltype(std::declval<std::__bind<void (y
/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221115221900-32d6b89b02-centos7-x86_64-clang15/installed/uninstrumented/libcxx/include/c++/v1/__functional/invoke.h:479:     @     0x7f92a5f579d4  void std::__invoke_void_return_wrapper<vo
/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221115221900-32d6b89b02-centos7-x86_64-clang15/installed/uninstrumented/libcxx/include/c++/v1/__functional/function.h:185:     @     0x7f92a5f579ac  std::__function::__alloc_func<std::__bi
/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221115221900-32d6b89b02-centos7-x86_64-clang15/installed/uninstrumented/libcxx/include/c++/v1/__functional/function.h:359:     @     0x7f92a5f56ee8  std::__function::__func<std::__bind<voi
/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221115221900-32d6b89b02-centos7-x86_64-clang15/installed/uninstrumented/libcxx/include/c++/v1/__functional/function.h:512:     @     0x7f92ae30d881  std::__function::__value_func<void ()>:
/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221115221900-32d6b89b02-centos7-x86_64-clang15/installed/uninstrumented/libcxx/include/c++/v1/__functional/function.h:1197:     @     0x7f92ae2e55a4  std::function<void ()>::operator()() c
/nfusr/centos-gcp-cloud/jenkins-worker-k3440b/jenkins/jenkins-github-yugabyte-db-centos-master-clang15-debug-558/yugabyte-db/build/debug-clang15-dynamic-ninja/../../src/yb/util/thread.cc:800:     @     0x7f92a5f24ca6  yb::Thread::Supervise
    @     0x7f92a3fc6ea4 
    @     0x7f92a3ceeb0c

grep on the the last 50 debug builds from http://jenkins.dev.yugabyte.com/job/github-yugabyte-db-centos-master-clang15-debug (521-572 builds) shows the following tests, most of them fail just once (test are sorted and the first number is the number of fails):

   2 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlAddCheckConstraint__0
   1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlAddColumn__1
   1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlAddColumn__2
   2 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlAlterTableAddFK__0
   1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlAlterTableAddFK__2
   1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlAlterTableAddPK__0
   1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlAlterTableAddPK__2
   1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlAlterTableSetOwner__0
   1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlAlterTableSetOwner__2
   1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlCreateTable__1
   1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlDeleteColumn__0
   1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlDropCheckConstraint__0
   1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlDropDefault__1
   1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlDropDefault__2
   1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlDropUniqueConstraint__1
   2 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlRenameTable__0
   2 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlSequencePartialCleanupAfterRestore__1
   1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlSequenceUndoCreateSequence__2
   1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlSequenceUndoDropSequence__0
   2 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlSequenceUndoDropSequence__2
   1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlSequenceUndoInsertedData__1
   1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlSequenceVerifyPartialRestore__1
   1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlSetNotNull__0
   1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlSetNotNull__1
   1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_Pgsql__1
   2 YbAdminSnapshotScheduleTest_CatalogLoadRace
   8 YbAdminSnapshotScheduleTest_ConsistentTxnRestore
   1 YbAdminSnapshotScheduleTest_TablegroupGCAfterRestore
   1 YbAdminSnapshotScheduleUpgradeTestWithYsql_PgsqlTestOldSysCatalogSnapshot
NatashaSerebryanaya commented 1 year ago

Some of Colocation_YbAdminSnapshotScheduleTestWithYsqlParam* tests fail with

../../src/yb/tools/yb-admin-snapshot-schedule-test.cc:1480: Failure
Failed
Bad status: Runtime error (yb/util/subprocess.cc:615): Subprocess '${BUILD_ROOT}/tests-tools/../bin/yb-admin' terminated with non-zero exit status 256: Error running create_snapshot_schedule: Invalid argument (yb/tools/yb-admin_client.cc:365): Namespace 'my_keyspace' of type 'ysql' not found
Usage: ${BUILD_ROOT}/tests-tools/../bin/yb-admin create_snapshot_schedule  <snapshot_interval_in_minutes> <snapshot_retention_in_minutes> <keyspace>

These are the tests out of last 50 builds: 

   1 Colocation_YbAdminSnapshotScheduleTestWithYsqlParam.FailAfterMigration_0
   1 Colocation_YbAdminSnapshotScheduleTestWithYsqlParam.PgsqlAddCheckConstraint_0
   1 Colocation_YbAdminSnapshotScheduleTestWithYsqlParam.PgsqlAddCheckConstraint_1
   1 Colocation_YbAdminSnapshotScheduleTestWithYsqlParam.PgsqlAddColumn_0
   1 Colocation_YbAdminSnapshotScheduleTestWithYsqlParam.PgsqlAddUniqueConstraint_0
   1 Colocation_YbAdminSnapshotScheduleTestWithYsqlParam.PgsqlAlterTableAddFK_1
   1 Colocation_YbAdminSnapshotScheduleTestWithYsqlParam.PgsqlCreateIndex_1
   1 Colocation_YbAdminSnapshotScheduleTestWithYsqlParam.PgsqlCreateTable_0
   1 Colocation_YbAdminSnapshotScheduleTestWithYsqlParam.PgsqlDropCheckConstraint_1
   1 Colocation_YbAdminSnapshotScheduleTestWithYsqlParam.PgsqlDropDatabaseAndSchedule_0
   1 Colocation_YbAdminSnapshotScheduleTestWithYsqlParam.PgsqlDropDefault_0
   1 Colocation_YbAdminSnapshotScheduleTestWithYsqlParam.PgsqlDropDefault_1
   1 Colocation_YbAdminSnapshotScheduleTestWithYsqlParam.PgsqlDropNotNull_1
   1 Colocation_YbAdminSnapshotScheduleTestWithYsqlParam.PgsqlDropTable_1
   1 Colocation_YbAdminSnapshotScheduleTestWithYsqlParam.PgsqlRenameColumn_0
   1 Colocation_YbAdminSnapshotScheduleTestWithYsqlParam.PgsqlRenameTable_0
   1 Colocation_YbAdminSnapshotScheduleTestWithYsqlParam.PgsqlSequenceUndoCreateSequence_0
   1 Colocation_YbAdminSnapshotScheduleTestWithYsqlParam.PgsqlSequenceUndoDeletedData_1
   1 Colocation_YbAdminSnapshotScheduleTestWithYsqlParam.PgsqlSequenceUndoDropSequence_1
   1 Colocation_YbAdminSnapshotScheduleTestWithYsqlParam.PgsqlSequenceUndoInsertedData_0
   1 Colocation_YbAdminSnapshotScheduleTestWithYsqlParam.PgsqlSequenceVerifyPartialRestore_1
   1 Colocation_YbAdminSnapshotScheduleTestWithYsqlParam.PgsqlSetDefault_1
   1 Colocation_YbAdminSnapshotScheduleTestWithYsqlParam.PgsqlSetNotNull_0
   1 Colocation_YbAdminSnapshotScheduleTestWithYsqlParam.Pgsql_0
   1 Colocation_YbAdminSnapshotScheduleTestWithYsqlParam.Pgsql_1
   1 YbAdminSnapshotScheduleTest.PgsqlRestoreDroppedTableWithTablespace
   1 YbAdminSnapshotScheduleTest.ReadsAfterRestoreWithSplit
   1 YbAdminSnapshotScheduleTest.TablegroupGCAfterRestore
   1 YbAdminSnapshotScheduleUpgradeTestWithYsql.PgsqlTestOldSysCatalogSnapshot

The same test can fail with both these errors. (Actually there is a third error too, will add it in a separate comment)

The list of tests failing with either of the errors looks even more intimidating:

   1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_FailAfterMigration__0
   3 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlAddCheckConstraint__0
   1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlAddCheckConstraint__1
   1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlAddColumn__0
   1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlAddColumn__1
   1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlAddColumn__2
   1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlAddUniqueConstraint__0
   2 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlAlterTableAddFK__0
   1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlAlterTableAddFK__1
   1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlAlterTableAddFK__2
   1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlAlterTableAddPK__0
   1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlAlterTableAddPK__2
   1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlAlterTableSetOwner__0
   1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlAlterTableSetOwner__2
   1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlCreateIndex__1
   1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlCreateTable__0
   1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlCreateTable__1
   1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlDeleteColumn__0
   1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlDropCheckConstraint__0
   1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlDropCheckConstraint__1
   1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlDropDatabaseAndSchedule__0
   1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlDropDefault__0
   2 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlDropDefault__1
   1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlDropDefault__2
   1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlDropNotNull__1
   1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlDropTable__1
   1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlDropUniqueConstraint__1
   1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlRenameColumn__0
   3 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlRenameTable__0
   2 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlSequencePartialCleanupAfterRestore__1
   1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlSequenceUndoCreateSequence__0
   1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlSequenceUndoCreateSequence__2
   1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlSequenceUndoDeletedData__1
   1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlSequenceUndoDropSequence__0
   1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlSequenceUndoDropSequence__1
   2 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlSequenceUndoDropSequence__2
   1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlSequenceUndoInsertedData__0
   1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlSequenceUndoInsertedData__1
   2 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlSequenceVerifyPartialRestore__1
   1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlSetDefault__1
   2 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlSetNotNull__0
   1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlSetNotNull__1
   1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_Pgsql__0
   2 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_Pgsql__1
   1 YbAdminSnapshotScheduleTest_CatalogLoadRace
   8 YbAdminSnapshotScheduleTest_ConsistentTxnRestore
   1 YbAdminSnapshotScheduleTest_PgsqlRestoreDroppedTableWithTablespace
   1 YbAdminSnapshotScheduleTest_ReadsAfterRestoreWithSplit
   1 YbAdminSnapshotScheduleTest_TablegroupGCAfterRestore
   1 YbAdminSnapshotScheduleTest__CatalogLoadRace
   1 YbAdminSnapshotScheduleTest__TablegroupGCAfterRestore
   2 YbAdminSnapshotScheduleUpgradeTestWithYsql_PgsqlTestOldSysCatalogSnapshot
NatashaSerebryanaya commented 1 year ago

The 3d error is

Bad status: Runtime error (yb/util/subprocess.cc:615): Subprocess '${BUILD_ROOT}/tests-tools/../bin/yb-admin' terminated with non-zero exit status 256: Timed out (yb/rpc/outbound_call.cc:488): Unable to establish connection to leader master at [127.0.0.1:23149,127.0.0.1:22220,127.0.0.1:16974]. Please verify the addresses and check if server is up, or if you're missing --certs_dir_name.

: Could not locate the leader master: GetMasterRegistration RPC (request call id 3) to 127.0.0.1:16974 timed out after 59.996s

The following tests fail with this error in the last 50 builds:

Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlAddCheckConstraint__2
Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlAddUniqueConstraint__1
Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlAddUniqueConstraint__2
Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlDropNotNull__1
Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlDropUniqueConstraint__1
Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlDropUniqueConstraint__2
Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlRenameColumn__1
YbAdminSnapshotScheduleTest_PgsqlRestoreDroppedTableWithTablespace
YbAdminSnapshotScheduleTest_SplitDisabledDuringRestore

If one runs any of the tests long enough all 3 errors could be hit.

For Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlAddColumn__0 in 1000 runs it was 6 - PITR: Master metadata verified failed for restoration 9 - Namespace 'my_keyspace' of type 'ysql' not found 6 - Unable to establish connection to leader master