search

apache / datafusion-comet

Apache DataFusion Comet Spark Accelerator
https://datafusion.apache.org/comet
Apache License 2.0
447 stars 100 forks source link

chore: Add allocation source to StreamReader #332

Closed viirya closed 3 weeks ago

viirya commented 3 weeks ago

Which issue does this PR close?

Closes #.

Rationale for this change

There is a memory leak reported by Java Arrow in #324. When looking at the error message for debugging:

Memory was leaked by query. Memory leaked: (49152)
Allocator(ROOT) 0/49152/180352/9223372036854775807 (res/actual/peak/limit)

I figured that it is hard to know where the allocation comes from. In this PR, I added the source of allocators.

The error message is updated to:

Memory was leaked by query. Memory leaked: (49152)                                                                                                                                                                                     
Allocator(StreamReader/shuffle reader) 0/49152/180352/9223372036854775807 (res/actual/peak/limit)

What changes are included in this PR?

How are these changes tested?

viirya commented 3 weeks ago

cc @andygrove @sunchao @comphead

viirya commented 3 weeks ago

Merged. Thanks.