[core] Manage the bloomfilter buffer in cache manager

apache / paimon

Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.

https://paimon.apache.org/

Apache License 2.0

2.43k stars 957 forks source link

[core] Manage the bloomfilter buffer in cache manager #4403

Closed Aitozi closed 3 weeks ago

Aitozi commented 3 weeks ago

Purpose

The BloomFilter in SortLookupStoreReader is not managed by the CacheManager, which will lead to the excessive buffer usage. This PR do these two things:

Manage the BloomFilter's buffer with the CacheManager
Release the BloomFilter's buffer when the Reader is closed.

Linked issue: close #xxx

apache / paimon

[core] Manage the bloomfilter buffer in cache manager #4403

Purpose

Tests

API and Format

Documentation