DRILL-8489: Sender memory leak when rpc encode exception
Description
When encode throw Exception, if encode msg instanceof ReferenceCounted, netty can release msg, but drill convert msg to OutboundRpcMessage, so netty can not release msg.
we can reproduce this scenario by break point and add debug log. Seeing Testing#Test1
Documentation
(Please describe user-visible changes similar to what should appear in the Drill documentation.)
Testing
Test 1
1. set -Ddrill.memory.debug.allocator=TRUE
2. we add debug log as following
DrillByteBufAllocator #DrillByteBufAllocator
public ByteBuf buffer() {
File file = new File("/data/shf/b.log");
if (file.exists()) {
throw new OutOfMemoryException("shf encode exception");
}
return buffer(DEFAULT_BUFFER_SIZE);
}
3. restart drillbit
4. Run tpch sql8
select
o_year,
sum(case when nation = 'CHINA' then volume else 0 end) / sum(volume) as mkt_share
from (
select
extract(year from o_orderdate) as o_year,
l_extendedprice * 1.0 as volume,
n2.n_name as nation
from hive.tpch1s.part, hive.tpch1s.supplier, hive.tpch1s.lineitem, hive.tpch1s.orders, hive.tpch1s.customer, hive.tpch1s.nation n1, hive.tpch1s.nation n2, hive.tpch1s.region
where
p_partkey = l_partkey
and s_suppkey = l_suppkey
and l_orderkey = o_orderkey
and o_custkey = c_custkey
and c_nationkey = n1.n_nationkey
and n1.n_regionkey = r_regionkey
and r_name = 'ASIA'
and s_nationkey = n2.n_nationkey
and o_orderdate between date '1995-01-01'
and date '1996-12-31'
and p_type = 'LARGE BRUSHED BRASS') as all_nations
group by o_year
order by o_year;
5.Break point: BroadcastSenderRootExec#innerNext#tunnels[i].sendRecordBatch(batch);
we resume program (F9, idea tool ) until there is memory had been allocated in the writableBatch object shown below
6.Break point: MessageToMessageEncoder#encode
we resume program (F9, idea tool ) until step 5 writableBatch encode
we mkdir "/data/shf/b.log" for debug on step 2
end break point
find memory leak
Check whether the leaked memory id is equal to that allocated by writableBatch
DRILL-8489: Sender memory leak when rpc encode exception
Description
When encode throw Exception, if encode msg instanceof ReferenceCounted, netty can release msg, but drill convert msg to OutboundRpcMessage, so netty can not release msg.
we can reproduce this scenario by break point and add debug log. Seeing Testing#Test1
Documentation
(Please describe user-visible changes similar to what should appear in the Drill documentation.)
Testing
Test 1
1. set -Ddrill.memory.debug.allocator=TRUE
2. we add debug log as following
DrillByteBufAllocator #DrillByteBufAllocator
3. restart drillbit
4. Run tpch sql8
5.Break point: BroadcastSenderRootExec#innerNext#tunnels[i].sendRecordBatch(batch); we resume program (F9, idea tool ) until there is memory had been allocated in the writableBatch object shown below
6.Break point: MessageToMessageEncoder#encode we resume program (F9, idea tool ) until step 5 writableBatch encode
we mkdir "/data/shf/b.log" for debug on step 2
end break point
find memory leak
Check whether the leaked memory id is equal to that allocated by writableBatch
Test 2
random_sql(){
for i in
seq 1 3while true do num=$((RANDOM%22+1)) if [ -f $fileName ]; then echo "$fileName" " is exit" exit 0 else $drill_home/sqlline -u \"jdbc:drill:zk=jupiter-2:2181/drill_shf/jupiterbits_shf1\" -f tpch_sql8.sql >> sql8.log 2>&1 fi done } main(){ unset HADOOP_CLASSPATH
TPCH power test
for i in
seq 1 25do random_sql & done}