ua: copy system metadata from default system tenant {1} to tenant {2}

[cthumuluru-crdb]

Clusters with a single tenant use the system tenant for both system metadata and user data. TenantOne is a special tenant that acts as a system tenant and uses no tenant prefix for the keys. New clusters can use multi-tenancy support, which separates system tables from user data. While new clusters can take advantage of multi-tenancy, migrating existing clusters to use Unified Architecture (UA) is challenging.

Migrating user data from the default system tenant to a user tenant can be challenging. Another option is to migrate system metadata into a new tenant (TenantTwo) and make it the default system tenant. TenantOne will be treated as a user tenant and continue to use prefix-less keys.

Explore options to copy system data from default tenant {1} to tenant {2} for UA migration experiment.

Epic: CRDB-42740

Jira issue: CRDB-42745

[shubhamdhama]

Jotting down notes that will primarily be useful for my future reference.

First goal here is to bring up the cluster with system tenant two with all the data from previous tenant copied. Most likely this won’t be the final approach but something that just works for prototyping. To get started we’re writing a small builtin that would just copy system table’s data from old system tenant to new system tenant. To achieve this we have two options, either do direct KV reads, add TenantTwo prefix, and write back, or use SQL layers to read and write. To achieve the latter one rough idea is that for current tenant we can read data using InternalExecutor, so we can also try building one for TenantTwo in the (old) system tenant. But I think building this could be a challenge.

Some leads to follow the KV approach is to use key_rewriter which has functions like KeyRewriter.RewriteTenant that would very useful.

Steven brainstormed another interesting idea of doing backup-restore for only system database. We are exploring that as well.