search

apache / drill

Apache Drill is a distributed MPP query layer for self describing data
https://drill.apache.org/
Apache License 2.0
1.92k stars 985 forks source link

DRILL-8485: HashJoinPOP memory leak is caused by an oom exception whe… #2891

Closed shfshihuafeng closed 3 months ago

shfshihuafeng commented 3 months ago

DRILL-8485: HashJoinPOP memory leak is caused by an oom exception when read data from InputStream

Description

it is similar to DRILL-8484

exception info 

Caused by: org.apache.drill.exec.exception.OutOfMemoryException: Unable to allocate buffer of size 16384 (rounded from 15364) due to memory limit (41943040). Current allocation: 4337664
        at org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:241)
        at org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:216)
        at org.apache.drill.exec.memory.BaseAllocator.read(BaseAllocator.java:856)

leak info

  Allocator(frag:5:1) 5000000/1000000/27824128/40041943040 (res/actual/peak/limit)
      child allocators: 1
        Allocator(op:5:1:1:HashJoinPOP) 1000000/16384/22822912/41943040 (res/actual/peak/limit)
          child allocators: 0
          ledgers: 2
            ledger[442780] allocator: op:5:1:1:HashJoinPOP), isOwning: true, size: 8192, references: 2, life: 4486836603491..0, allocatorManager: [390894, life: 4486836601180..0] holds 4 buffers. 
                DrillBuf[458469], udle: [390895 1024..8192]
         event log for: DrillBuf[458469]

Documentation

(Please describe user-visible changes similar to what should appear in the Drill documentation.)

Testing

The testing method for drill-8485 is the similar as for DRILL-8484. we can throw exception in the method readVectors

shfshihuafeng commented 3 months ago

LGTM +1 Thanks @shfshihuafeng for all these memory leak fixes.

Thank for your reviewed. I will do my best to continuously contribute to the community