search

apache / paimon

Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.
https://paimon.apache.org/
Apache License 2.0
2.43k stars 957 forks source link

[Feature] Introduce secondary index for paimon #2925

Open leaves12138 opened 8 months ago

leaves12138 commented 8 months ago

Search before asking

Motivation

Up to now, Paimon use zorder & order sort compaction to speed up query. After sort compaction, files will be sorted by the order of specified columns. But in some situations, for example, we have tens of columns that should be added in the filter column, sometimes all of them come up together, sometimes, just a few of them. Zorder or order compaction can't handle this situation, because too many columns will reduce the effect of sorting. So if the column base number of these columns is small, we can use bloomfilter or other indexes to speed up queries. That's why this PIP comes up. I want to introduce a index framework to support paimon with flexible index system. 

Solution

No response

Anything else?

No response

Are you willing to submit a PR?

zyl891229 commented 8 months ago

The index should at the rowgroup(parquet) or stripe(orc) level for better or
it can be configured at the file or row group level ？

FangYongs commented 6 months ago

Hi @leaves12138 , what's the status of this feature?