search

hazelcast / hazelcast

Hazelcast is a unified real-time data platform combining stream processing with a fast data store, allowing customers to act instantly on data-in-motion for real-time insights.
https://www.hazelcast.com
Other
6.12k stars 1.83k forks source link

Limit result set growth of PagingPredicate for full scans #13510

Open taburet opened 6 years ago

taburet commented 6 years ago

For every full scanned partition, the result set produced by the scan is sorted and trimmed if PagingPredicate is used (QueryResult#orderAndLimit), this is done by returning a sublist of the original result set list (SortingUtil#getSortedSubList).

This is fine for per-partition result sets, but if the result set is reused for a multi-partition scan, as in QueryRunner#runUsingPartitionScanSafely using CallerRunsPartitionScanExecutor, the insertions done to the result set's sublist are in fact going to the original underlying list (see java.util.SubList) which accumulates all the entries from all scanned partitions and this causes OOMEs.

mmedenjak commented 4 years ago

Since we're heavily invested in the new query engine, I don't think we should spend too much time on this.