apache / gravitino

World's most powerful open data catalog for building a high-performance, geo-distributed and federated metadata lake.
https://gravitino.apache.org
Apache License 2.0
696 stars 216 forks source link

[FEATURE] Store metadata of databases and tables into Gravitino stotage #4026

Open caican00 opened 1 week ago

caican00 commented 1 week ago

Describe the feature

Store more metadata of dbs and tables, such as columns, properties, to gravitino storage.

Motivation

To provide more metadata of database and table for other systems to use more easily, we can dump the meta from gravitino storage directly instead of invoking the api of gravitino .

But now gravitino doesn't store the columns and properties of table.

Describe the solution

the extra metadata, such as columns, properties, should be written not only to the backend storage of the catalogs, but also to the gravitino storage.

Additional context

Since there are both write to rdbms and write to non-rdbms operations, how to ensure transactionality will be a problem.