taiyang-li
commented
1 month ago Compare expression execution performance without or with JIT
ClickHouse is compiled with patches from https://github.com/ClickHouse/ClickHouse/pull/70598 gluten is compiled with branch in pr: https://github.com/apache/incubator-gluten/pull/7536
Query
subquery extracted from d_5231_0.sql
select day, avg(if(totaltime = 60 and frame_sent>0 and frame_sent/60<60,frame_sent/60,
if(totaltime <> 60 and frame_sent>0 and frame_sent/totaltime< 60,frame_sent/totaltime,null))) as avg_enc_frame_rate
from
(
select day, uid
,cast(other_unfixed_para['1018'] as bigint)-cast(other_unfixed_para['1019'] as bigint)-cast(other_unfixed_para['1020'] as bigint) as frame_sent
,if((other_unfixed_para['65'] = 4294967295 OR other_unfixed_para['65'] is null OR other_unfixed_para['65']%60=0),60,other_unfixed_para['65']%60) as totaltime
from bigolive.live_sdk_video_stats_event_all
where day='${day}'
and cast ((cast(other_unfixed_para['1036'] as bigint) & 2) as bigint) =2
and (other_unfixed_para['65']>0 or other_unfixed_para['65'] is null)
and if(((cast(other_unfixed_para['1036'] as bigint) & 524288) /524288)=1 and other_unfixed_para['65'] = 4294967295,0,1)=1
and cast(other_unfixed_para['92'] as bigint)>0
and cast(other_unfixed_para['1018'] as bigint)>= 0 and cast(other_unfixed_para['1019'] as bigint)>= 0 and cast(other_unfixed_para['1020'] as bigint)>= 0
) tt
group by day
Description
In the old version,
min_count_to_compile_expressionwas 3, which meant that each executor had to encounter the same CompileDAG more than three times before it could be compiled, which is difficult to meet in spark execution. As a result, the JIT in the old version was almost ineffectiveUsers can control whether JIT is available through spark conf. By default
compile_expressionsis true, andmin_count_to_compile_expressionis 0