Open NatashaSerebryanaya opened 1 year ago
Some of Colocation_YbAdminSnapshotScheduleTestWithYsqlParam* tests fail with
../../src/yb/tools/yb-admin-snapshot-schedule-test.cc:1480: Failure
Failed
Bad status: Runtime error (yb/util/subprocess.cc:615): Subprocess '${BUILD_ROOT}/tests-tools/../bin/yb-admin' terminated with non-zero exit status 256: Error running create_snapshot_schedule: Invalid argument (yb/tools/yb-admin_client.cc:365): Namespace 'my_keyspace' of type 'ysql' not found
Usage: ${BUILD_ROOT}/tests-tools/../bin/yb-admin create_snapshot_schedule <snapshot_interval_in_minutes> <snapshot_retention_in_minutes> <keyspace>
These are the tests out of last 50 builds:
1 Colocation_YbAdminSnapshotScheduleTestWithYsqlParam.FailAfterMigration_0
1 Colocation_YbAdminSnapshotScheduleTestWithYsqlParam.PgsqlAddCheckConstraint_0
1 Colocation_YbAdminSnapshotScheduleTestWithYsqlParam.PgsqlAddCheckConstraint_1
1 Colocation_YbAdminSnapshotScheduleTestWithYsqlParam.PgsqlAddColumn_0
1 Colocation_YbAdminSnapshotScheduleTestWithYsqlParam.PgsqlAddUniqueConstraint_0
1 Colocation_YbAdminSnapshotScheduleTestWithYsqlParam.PgsqlAlterTableAddFK_1
1 Colocation_YbAdminSnapshotScheduleTestWithYsqlParam.PgsqlCreateIndex_1
1 Colocation_YbAdminSnapshotScheduleTestWithYsqlParam.PgsqlCreateTable_0
1 Colocation_YbAdminSnapshotScheduleTestWithYsqlParam.PgsqlDropCheckConstraint_1
1 Colocation_YbAdminSnapshotScheduleTestWithYsqlParam.PgsqlDropDatabaseAndSchedule_0
1 Colocation_YbAdminSnapshotScheduleTestWithYsqlParam.PgsqlDropDefault_0
1 Colocation_YbAdminSnapshotScheduleTestWithYsqlParam.PgsqlDropDefault_1
1 Colocation_YbAdminSnapshotScheduleTestWithYsqlParam.PgsqlDropNotNull_1
1 Colocation_YbAdminSnapshotScheduleTestWithYsqlParam.PgsqlDropTable_1
1 Colocation_YbAdminSnapshotScheduleTestWithYsqlParam.PgsqlRenameColumn_0
1 Colocation_YbAdminSnapshotScheduleTestWithYsqlParam.PgsqlRenameTable_0
1 Colocation_YbAdminSnapshotScheduleTestWithYsqlParam.PgsqlSequenceUndoCreateSequence_0
1 Colocation_YbAdminSnapshotScheduleTestWithYsqlParam.PgsqlSequenceUndoDeletedData_1
1 Colocation_YbAdminSnapshotScheduleTestWithYsqlParam.PgsqlSequenceUndoDropSequence_1
1 Colocation_YbAdminSnapshotScheduleTestWithYsqlParam.PgsqlSequenceUndoInsertedData_0
1 Colocation_YbAdminSnapshotScheduleTestWithYsqlParam.PgsqlSequenceVerifyPartialRestore_1
1 Colocation_YbAdminSnapshotScheduleTestWithYsqlParam.PgsqlSetDefault_1
1 Colocation_YbAdminSnapshotScheduleTestWithYsqlParam.PgsqlSetNotNull_0
1 Colocation_YbAdminSnapshotScheduleTestWithYsqlParam.Pgsql_0
1 Colocation_YbAdminSnapshotScheduleTestWithYsqlParam.Pgsql_1
1 YbAdminSnapshotScheduleTest.PgsqlRestoreDroppedTableWithTablespace
1 YbAdminSnapshotScheduleTest.ReadsAfterRestoreWithSplit
1 YbAdminSnapshotScheduleTest.TablegroupGCAfterRestore
1 YbAdminSnapshotScheduleUpgradeTestWithYsql.PgsqlTestOldSysCatalogSnapshot
The same test can fail with both these errors. (Actually there is a third error too, will add it in a separate comment)
The list of tests failing with either of the errors looks even more intimidating:
1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_FailAfterMigration__0
3 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlAddCheckConstraint__0
1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlAddCheckConstraint__1
1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlAddColumn__0
1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlAddColumn__1
1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlAddColumn__2
1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlAddUniqueConstraint__0
2 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlAlterTableAddFK__0
1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlAlterTableAddFK__1
1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlAlterTableAddFK__2
1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlAlterTableAddPK__0
1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlAlterTableAddPK__2
1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlAlterTableSetOwner__0
1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlAlterTableSetOwner__2
1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlCreateIndex__1
1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlCreateTable__0
1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlCreateTable__1
1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlDeleteColumn__0
1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlDropCheckConstraint__0
1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlDropCheckConstraint__1
1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlDropDatabaseAndSchedule__0
1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlDropDefault__0
2 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlDropDefault__1
1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlDropDefault__2
1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlDropNotNull__1
1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlDropTable__1
1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlDropUniqueConstraint__1
1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlRenameColumn__0
3 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlRenameTable__0
2 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlSequencePartialCleanupAfterRestore__1
1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlSequenceUndoCreateSequence__0
1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlSequenceUndoCreateSequence__2
1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlSequenceUndoDeletedData__1
1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlSequenceUndoDropSequence__0
1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlSequenceUndoDropSequence__1
2 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlSequenceUndoDropSequence__2
1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlSequenceUndoInsertedData__0
1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlSequenceUndoInsertedData__1
2 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlSequenceVerifyPartialRestore__1
1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlSetDefault__1
2 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlSetNotNull__0
1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlSetNotNull__1
1 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_Pgsql__0
2 Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_Pgsql__1
1 YbAdminSnapshotScheduleTest_CatalogLoadRace
8 YbAdminSnapshotScheduleTest_ConsistentTxnRestore
1 YbAdminSnapshotScheduleTest_PgsqlRestoreDroppedTableWithTablespace
1 YbAdminSnapshotScheduleTest_ReadsAfterRestoreWithSplit
1 YbAdminSnapshotScheduleTest_TablegroupGCAfterRestore
1 YbAdminSnapshotScheduleTest__CatalogLoadRace
1 YbAdminSnapshotScheduleTest__TablegroupGCAfterRestore
2 YbAdminSnapshotScheduleUpgradeTestWithYsql_PgsqlTestOldSysCatalogSnapshot
The 3d error is
Bad status: Runtime error (yb/util/subprocess.cc:615): Subprocess '${BUILD_ROOT}/tests-tools/../bin/yb-admin' terminated with non-zero exit status 256: Timed out (yb/rpc/outbound_call.cc:488): Unable to establish connection to leader master at [127.0.0.1:23149,127.0.0.1:22220,127.0.0.1:16974]. Please verify the addresses and check if server is up, or if you're missing --certs_dir_name.
: Could not locate the leader master: GetMasterRegistration RPC (request call id 3) to 127.0.0.1:16974 timed out after 59.996s
The following tests fail with this error in the last 50 builds:
Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlAddCheckConstraint__2
Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlAddUniqueConstraint__1
Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlAddUniqueConstraint__2
Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlDropNotNull__1
Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlDropUniqueConstraint__1
Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlDropUniqueConstraint__2
Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlRenameColumn__1
YbAdminSnapshotScheduleTest_PgsqlRestoreDroppedTableWithTablespace
YbAdminSnapshotScheduleTest_SplitDisabledDuringRestore
If one runs any of the tests long enough all 3 errors could be hit.
For Colocation__YbAdminSnapshotScheduleTestWithYsqlParam_PgsqlAddColumn__0
in 1000 runs it was
6 - PITR: Master metadata verified failed for restoration
9 - Namespace 'my_keyspace' of type 'ysql' not found
6 - Unable to establish connection to leader master
Jira Link: DB-5340
Description
The fatal "PITR: Master metadata verified failed for restoration" is seen in multiple unit tests in debug builds.
grep on the the last 50 debug builds from http://jenkins.dev.yugabyte.com/job/github-yugabyte-db-centos-master-clang15-debug (521-572 builds) shows the following tests, most of them fail just once (test are sorted and the first number is the number of fails):