yugabyte / yugabyte-db

YugabyteDB - the cloud native distributed SQL database for mission-critical applications.
https://www.yugabyte.com
Other
8.92k stars 1.06k forks source link

[xCluster] Target WAL is bigger than the source #22916

Open yugabyte-ci opened 3 months ago

yugabyte-ci commented 3 months ago

Jira Link: DB-11828

WAL size on xCluster target universe is 20% higher than the source universe

hari90 commented 3 months ago

Looks like the same external_hybrid_time is set on every write_pair and on the write_batch causing more space usage on target WAL.

hari90 commented 3 months ago

This only happens in YCQL direct write batches. Transactional writes use intents which do not set the external ht. YSQL uses transactional writes only.

Fix would be to skip the ht if the write batch had the same value. It won't handle all cases though

hari90 commented 3 months ago

The Target write batch has 98 kv pairs. Each 7 of these has one external HT. The batch has 14 HTs in total. Today we record the HT in each one of the 98 records. The fix would be to remove the HT on the kvs and break the batch per HT.