[FEATURE] Supports iceberg metadata migration through the iceberg rest server

[caican00]

Describe the feature

In our real-world scenario, a generic iceberg catalog name such as iceberg_{cluster_name} is used in a region, regardless of the specific iceberg backend metastore.

Users do not need to care about the underlying metadata storage service.

When we do the migration of the metadata service at the bottom of iceberg, users should not be aware of it, and they should not be asked to modify the iceberg catalog name in their sql, otherwise it will be very difficult to implement.

Therefore, we want to implicitly implement the switching of iceberg's underlying metadata storage service without requiring users to modify their sql in iceberg rest server.

My initial thought is: supporting a master-slave backend catalog in the iceberg rest server，as shown in the figure.

1. For new tables, access master metastore directly to create, read, or modify metadata

2. For the old table, if it exists in master metastore, the metadata is read or modified directly based on master metastore. If it does not exist in master metastore, copy the metadata from slave metastore to master metastore at first, and then read or modify the metadata based on master metastore.

Motivation

No response

Describe the solution

No response

Additional context

No response

[FANNG1]

Sounds like an op tools, I'm not sure whether should place the logic in Iceberg rest server module. Besides implementing a new Catalog, using event listener maybe another option.

[caican00]

Sounds like an op tools, I'm not sure whether should place the logic in Iceberg rest server module. Besides implementing a new Catalog, using event listener maybe another option.

If the event listener is used, the metadata is still updated in slave metastore, and it is difficult to find a suitable time to stop slave metastore.