search

apache / gravitino

World's most powerful open data catalog for building a high-performance, geo-distributed and federated metadata lake.
https://gravitino.apache.org
Apache License 2.0
1.09k stars 343 forks source link

[FEATURE] Why gravitino store schema and table propertites？ #1728

Open mygrsun opened 10 months ago

mygrsun commented 10 months ago

Describe the feature

I can understand why gravitino store metalake and catalogs，but i cant understand why gravitino store schema and table propertites。because I don‘t find any place useing it out of audit。Is this to prepare for other future needs?

image

Motivation

No response

Describe the solution

No response

Additional context

No response

jerryshao commented 10 months ago

This is our design of Gravitino. In Gravitino, we will have two parts of metadata, one part is system metadata that coming from underlying sources like HMS, JDBC, etc; another part is business metadata, like tags, audit info, etc. For the first part of the metadata, it can be retrieved from underlying sources, another part is stored in our storage. When Gravitino get a request from user, it will fetch both system metadata and business metadata, merge them together, give back to the users.

Because we will potentially have more business metadata in future, so that's why we also need a storage to store schema/table additional metadata.