Can not restore schema from backup of cluster_1 to cluster_2, ScyllaDB 6.0 (tablets enabled)

[mikliapko]

Preconditions:

• 2-nodes cluster_1 and cluster_2;
• 1 keyspace + 1 table (20 rows) in each cluster;
• Manager configured for cluster_1.

Steps:

1. Backup the first cluster;
2. Corrupt the data in node_2 of cluster_2;
3. Configure Manager for cluster_2;
4. Restore the schema from the backup of the cluster_1 into the cluster_2; sctool restore -c 8ce76feb-d543-4d6d-8f6a-1e997726c4d1 --restore-schema --location s3:backup-bucket --snapshot-tag sm_20240621115957UTC
5. Check restore operation status

Actual result: The operation status is ERROR:

• Cause restore data: create ks (ks) with CREATE KEYSPACE ks WITH replication = {'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'datacenter1': '2'} AND durable_writes = true: Cannot add existing keyspace "ks"
• Details:

  Run:        be00bca2-2f31-11ef-bdda-16ffdc5b4c17
  Status:     ERROR (restoring backed-up data)
  Cause:      restore data: create ks (ks) with CREATE KEYSPACE ks WITH replication = {'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'datacenter1': '2'} AND durable_writes = true: Cannot add existing keyspace "ks"
  Start time: 20 Jun 24 18:20:15 UTC
  End time:   20 Jun 24 18:20:33 UTC
  Duration:   17s
  Progress:   0% | 0%
  Snapshot Tag:   sm_20240620181952UTC

Expected result: The operation status is DONE.

Environment:

• Scylla manager build: https://downloads.scylladb.com/manager/relocatable/unstable/branch-3.3/2024-06-21T12:54:49Z/scylla-manager_3.2.7-0.20240516.bca79c502-SNAPSHOT_linux_x86_64.tar.gz
• ScyllaDB: 6.0 with tablets enabled.

Additional info:

• The test passed for Scylla 2023.1;
• Logs attached;
• DTest - manager_restore_tests.TestScyllaMgmtRestore.test_post_restore_different_table_not_repaired.

[Michal-Leszczynski]

@mikliapko this looks like a test setup error. One of the requirements of restoring schema is to do so into an empty cluster - which is not the case here. This still worked fine in the past, because the existing schema and restored schema were the same, so restored sstables from backup didn't break anything.

This requirement will be less invasive when working with Scylla 6.0 (I have yet to prepare the docs describing it), but it will say that the restore destination cluster can't have any overlapping schema with the restored schema. That's because with Scylla 6.0 SM restores schema via CQL and if overlapping schema exists, it returns the CQL error mentioned above.

[mikliapko]

Since the setup the test does is not valid, it was decided to remove it from dtest. https://github.com/scylladb/scylla-dtest/pull/4526