Is your feature request related to a problem? Please describe
Sort and aggregation of segment documents when building star tree takes up lots of resources if they are stored in in-memory structures like Lists. So this issue tracks the off heap implementation where we store and retrieve the star tree documents in off heap method using temporary files.
Describe the solution you'd like
We will store the offsets in array and swap the offsets when we go through the files. We offload the entire sort and aggregation of segment documents and construction of star tree documents to off-heap.
peternied
commented
1 month ago
Is your feature request related to a problem? Please describe
Sort and aggregation of segment documents when building star tree takes up lots of resources if they are stored in in-memory structures like Lists. So this issue tracks the off heap implementation where we store and retrieve the star tree documents in off heap method using temporary files.
Describe the solution you'd like
We will store the offsets in array and swap the offsets when we go through the files. We offload the entire sort and aggregation of segment documents and construction of star tree documents to off-heap.
Related component
Other
Describe alternatives you've considered
No response
Additional context
No response