[DocDB] Clone sequences for a given database as part of Clone Database

[yugabyte-ci]

Jira Link: DB-10350

Currently, clone doesn't support sequences in YSQL database. That is, if we try to clone a database which has a sequence, cloning will fail with the following error: ERROR: Clone operation aborted: Not found (yb/tablet/tablet_metadata.cc:781): Failed to list snapshot: c0d01ed0-0344-49cb-9e69-386b8e75b558: Table <unknown_table_name> (0000ffff0000300080000000000004eb) not found in Raft group 00000000000000000000000000000000

[yamen-haddad]

The clone is failing in the Generate Snapshot Info as of time step. More specifically, clone repacks the SnapshotInfo of the suitable snapshot to be consumed in the restore operation. However, as the snapshot repacked by clone is part of a snapshot schedule, it contains the sequences_data table entry i.e: table: 0000ffff0000300080000000000004eb. However, the repacking step doesn't expect the sequences_table to be inside the snapshot schedule and thus throws the previous error. Cloning sequences is one of the situations where clone diverge from PITR. In clone, we use Backup/Restore flow to restore sequences while in PITR we use the snapshot of the sequnces_data table to get the correct value at restore time. Therefore, we should ignore the sequences_data table and related entries from the snapshotInfo while repacking.

[yamen-haddad]

After the first part of supporting sequences in clone, users should be able to clone a database that has sequences in it.

The only limitation we have today is that users cannot clone to a point in time before a drop sequences operation has been performed. For example, if the user drops a table that has a serial column, the user cannot clone to a time before the drop table.

The reason for the failure is that ysql_dump is unable to get the last_value of the dropped sequence. This is because the sequence is dropped and we cannot perform read as of time to the system_postgres.sequences_data table.

We are discussing two paths forward:

• Unblock the clone, by setting last_value to 1 or the mid of range value. This unblocks users who are using the clone to read deleted data. The downside is that they might get errors when trying to insert data using the cloned sequence as the generated next values might already have been used before. The error will disappear once the last_values reaches a unique value.
• Add support to read as of time from the system_postgres.sequences_data

[yamen-haddad]

Closing this as the two revisions landed in this thread should cover all the cases of sequences. @Arjun-yb To reopen in case any issue is observed.

[yamen-haddad]

Reopening as I forgot to backport to 2024.2