[xCluster][DB-Scope] When new database added to DB-Scope replication, Create objects timed out waiting for postgres backends to catch up

yugabyte / yugabyte-db

YugabyteDB - the cloud native distributed SQL database for mission-critical applications.

https://www.yugabyte.com

Other

8.99k stars 1.07k forks source link

[xCluster][DB-Scope] When new database added to DB-Scope replication, Create objects timed out waiting for postgres backends to catch up #23946

Open agsh-yb opened 1 month ago

agsh-yb commented 1 month ago

Jira Link: DB-12840

Description

Version: 2.23.1.0-b183

Deterministic Attached support bundles on Jira

Steps to repro:

Setup replication
perform some DDLs' DMLs'
Upgrade universe 
Alter xCluster checkpoint : Success
    (    21.723s) Add (db_dr_col_2, db_dr_2) to xCluster replication : Success
    (     2.457s) Create tables and indexes. : Success
Validate replication: Fail

Target

Source

Issue Type

kind/bug

Warning: Please confirm that this issue does not contain any sensitive information

[X] I confirm this issue does not contain any sensitive information.

agsh-yb commented 1 month ago

Goal of Test

To validate db_scope replication after upgrades

agsh-yb commented 1 month ago

It is observed that create objects on target it taking long and it's failing with

db_dr_col_2=# CREATE UNIQUE INDEX table_col_unique_index ON employees_col(id, age ) WITH (COLOCATION_ID = 16386);

ERROR:  timed out waiting for postgres backends to catch up
DETAIL:  1 backends on database 16641 are still behind catalog version 8.
HINT:  Run the following query on all tservers to find the lagging backends: SELECT * FROM pg_stat_activity WHERE catalog_version < 8 AND datid = 16641;
Time: 300139.170 ms (05:00.139)