[Feature] introduce key-value separation for big values

apache / paimon

Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.

https://paimon.apache.org/

Apache License 2.0

2.29k stars 916 forks source link

[Feature] introduce key-value separation for big values #1428

Open zhangjun0x01 opened 1 year ago

zhangjun0x01 commented 1 year ago

Search before asking

[X] I searched in the issues and found nothing similar.

Motivation

If the values is very big, we can separate key and value in the LSM tree，so that we avoid copying the values over and over again during compaction to improve the performance .

refer to https://rocksdb.org/blog/2021/05/26/integrated-blob-db.html

Solution

No response

Anything else?

No response

Are you willing to submit a PR?

[ ] I'm willing to submit a PR!

MonsterChenzhuo commented 1 year ago

@zhangjun0x01 please assign to me; thx。

JingsongLi commented 1 year ago

@zhangjun0x01 please assign to me; thx。

Please DO NOT just reply "please assign it to me", see https://paimon.apache.org/community/how-to-contribute/#code-contribution-guide