Before submitting a bug report, please verify that you are using the most current version of Drill.
Describe the bug
During the process of reading data from disk to building hash tables in memory, if an exception is thrown, it will result in a memory SpilledRecordBatch leak
To Reproduce
prepare data for tpch 1s
20 concurrent for tpch sql8
set direct memory 5g
when it had OutOfMemoryException , stopped all sql.
finding memory leak
Expected behavior
when all sql sop , we find direct memory is not 0 AND could find leak log like following.
Allocator(op:5:0:1:HashJoinPOP) 1000000/98304/22822912/41943040 (res/actual/peak/limit)
Error detail, log output or screenshots
Caused by: org.apache.drill.exec.exception.OutOfMemoryException: Unable to allocate buffer of size 8192 due to memory limit (41943040). Current allocation: 3684352
at org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:241)
at org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:216)
at org.apache.drill.exec.vector.VarCharVector.allocateNew(VarCharVector.java:411)
at org.apache.drill.exec.vector.NullableVarCharVector.allocateNew(NullableVarCharVector.java:270)
at org.apache.drill.exec.physical.impl.common.HashPartition.allocateNewVectorContainer(HashPartition.java:215)
at org.apache.drill.exec.physical.impl.common.HashPartition.allocateNewCurrentBatchAndHV(HashPartition.java:238)
at org.apache.drill.exec.physical.impl.common.HashPartition.<init>(HashPartition.java:165)
Drill version
The version of Drill you encountered the issue in.
Additional contextselect o_year, sum(case when nation = 'CHINA' then volume else 0 end) / sum(volume) as mkt_share from ( select extract(year from o_orderdate) as o_year, l_extendedprice * 1.0 as volume, n2.n_name as nation from hive.tpch1s.part, hive.tpch1s.supplier, hive.tpch1s.lineitem, hive.tpch1s.orders, hive.tpch1s.customer, hive.tpch1s.nation n1, hive.tpch1s.nation n2, hive.tpch1s.region where p_partkey = l_partkey and s_suppkey = l_suppkey and l_orderkey = o_orderkey and o_custkey = c_custkey and c_nationkey = n1.n_nationkey and n1.n_regionkey = r_regionkey and r_name = 'ASIA' and s_nationkey = n2.n_nationkey and o_orderdate between date '1995-01-01' and date '1996-12-31' and p_type = 'LARGE BRUSHED BRASS') as all_nations group by o_year order by o_year
Before submitting a bug report, please verify that you are using the most current version of Drill.
Describe the bug During the process of reading data from disk to building hash tables in memory, if an exception is thrown, it will result in a memory SpilledRecordBatch leak
To Reproduce prepare data for tpch 1s
Expected behavior when all sql sop , we find direct memory is not 0 AND could find leak log like following.
Allocator(op:5:0:1:HashJoinPOP) 1000000/98304/22822912/41943040 (res/actual/peak/limit)Error detail, log output or screenshots
Drill version The version of Drill you encountered the issue in.
Additional context
select o_year, sum(case when nation = 'CHINA' then volume else 0 end) / sum(volume) as mkt_share from ( select extract(year from o_orderdate) as o_year, l_extendedprice * 1.0 as volume, n2.n_name as nation from hive.tpch1s.part, hive.tpch1s.supplier, hive.tpch1s.lineitem, hive.tpch1s.orders, hive.tpch1s.customer, hive.tpch1s.nation n1, hive.tpch1s.nation n2, hive.tpch1s.region where p_partkey = l_partkey and s_suppkey = l_suppkey and l_orderkey = o_orderkey and o_custkey = c_custkey and c_nationkey = n1.n_nationkey and n1.n_regionkey = r_regionkey and r_name = 'ASIA' and s_nationkey = n2.n_nationkey and o_orderdate between date '1995-01-01' and date '1996-12-31' and p_type = 'LARGE BRUSHED BRASS') as all_nations group by o_year order by o_year