[X] I had searched in the issues and found no similar feature requirement.
Description
This issue proposes the introduction of a skipping index in the new columned-based storage for Measure and Stream. The skipping index is intended to help filter the data more efficiently. The index will support two types: Min-Max and Bloom filter.
The proposed index will support two types:
Min-Max Index: This index will store the minimum and maximum values of each column in each data block. It will quickly eliminate the data blocks that do not contain the desired range of values. All numeric data will applied to this type.
Bloom Filter Index: This probabilistic data structure will help in determining whether an element may be in a set or not. It will reduce the I/O operations by avoiding unnecessary disk reads for data that are not present. All data will applied to this type.
The numeric data includes both types at the same time.
In the API, we propose to add a new index rule type: "skipping".
Use case
No response
Related issues
No response
Are you willing to submit a pull request to implement this on your own?
[ ] Yes I am willing to submit a pull request on my own!
zesiar0
commented
4 months ago
Search before asking
Description
This issue proposes the introduction of a skipping index in the new columned-based storage for Measure and Stream. The skipping index is intended to help filter the data more efficiently. The index will support two types: Min-Max and Bloom filter.
The proposed index will support two types:
Min-Max Index: This index will store the minimum and maximum values of each column in each data block. It will quickly eliminate the data blocks that do not contain the desired range of values. All numeric data will applied to this type.
Bloom Filter Index: This probabilistic data structure will help in determining whether an element may be in a set or not. It will reduce the I/O operations by avoiding unnecessary disk reads for data that are not present. All data will applied to this type.
The numeric data includes both types at the same time.
In the API, we propose to add a new index rule type: "skipping".
Use case
No response
Related issues
No response
Are you willing to submit a pull request to implement this on your own?
Code of Conduct