Split Metadata Model for Tables

[c-thiel]

Currently we store TableMetadata as one blob in Postgres. We should investigate two options:

• [ ] Don't store TableMetadata in Postgres at all, but on a Object Store instead
• [ ] Split TableMetadata in Postgres into smaller tables, as requests often filter to only include snapshots for a certain ref. This would also align well with the current plans of the Iceberg REST v2 spec https://github.com/apache/iceberg/issues/10075

[corleyma]

Don't store TableMetadata in Postgres at all, but on a Object Store instead

I think supporting this option would be a big boon to adoption, because:

• that's how other SQL catalogs work (e.g., JVM's JDBC Catalog, PyIceberg SqlCatalog)
• the metadata json file is an integration point for query engines that don't (yet) support the REST API (e.g., BigQuery, but also the admittedly quite crappy duckdb iceberg extension, etc).

I would consider adopting this catalog implementation today in my work if this were supported.

[c-thiel]

@corleyma thanks for your Feedback! We are already today writing metadata to the location of the table (as metadata/<uuid>-.gz.metadata.json). We just store it additionally internally, which is our source of truth. This internal storage is what's bugging me mostly currently, because it's a big binary chunk in Postgres. The location of the external one is returned to the client as part of the TableMetadata.

The write happens here (conditionally, only if the client did not "stage" the table): https://github.com/hansetag/iceberg-catalog/blob/f5f4185927e6e9822cd815f1716b62ff9267de7f/crates/iceberg-catalog/src/catalog/tables.rs#L157

With the integration of other tools you are touching an interesting point: We are deliberately not writing the toplevel metadata.json in a file called like that, even if it would introduce higher compatibility with other engines. The reason is twofold:

1. It could lead to inconsistencies on write - only the catalog can guarantee transactions on some storage systems such as S3
2. We don't want clients to have credentials in the first place

Technically it wouldn't be a problem to enable it optionally. Is this what you are thinking about?

With our model we are following what tabular.io does. They also don't write a toplevel "metadata.json" and if you manually fiddle with metadata/<uuid>-.gz.metadata.json it has absolutely no effect because metadata stored somewhere internally is returned.