apache / gravitino

World's most powerful open data catalog for building a high-performance, geo-distributed and federated metadata lake.
https://datastrato.ai/docs/
Apache License 2.0
679 stars 208 forks source link

[FEATURE] Introduce mature ORM framework for key-value storage #458

Open yuqi1129 opened 9 months ago

yuqi1129 commented 9 months ago

Describe the feature

As we plan to store user-related information such as user accounts and audit information in key-value storage, we need to save Java objects in the form of key-value pairs to RocksDB, then a generic ORM framework is required to convert object to key-value pair.

Motivation

No response

Describe the solution

No response

Additional context

No response

yuqi1129 commented 9 months ago

@xunliu cc @jerryshao I conducted a primary survey on the ORM framework for key-value storage, and it seems that it's not recommended to use ORM in key-value storage and there's no mature resolution available. I also agree with this point. Here are my thoughts on how to solve this problem.

So, we can handle user account information similarly to graviton entities and choose appropriate encode/decode methods if possible, for complex scenarios, we may need to copy them one by one. What's your opinion?

xunliu commented 8 months ago

I think the ORM framework is very important, we can create a database access layer so that we can also support a variety of heterogeneous databases. snowflake has also implemented an EntityLayer in the FDB on FDB.

image

otherwise, I'm very curious how Arangodb implements search on Rocksdb, can you research that? Can you share your research documents or web page?