jerryleooo
commented
3 years ago The result of "explain":
explain select sum(col1) from <table name> where __time between timestamp'2021-06-01' and timestamp'2021-06-30';
Fragment 0 [SINGLE]
Output layout: [sum]
Output partitioning: SINGLE []
Stage Execution Strategy: UNGROUPED_EXECUTION
Output[_col0]
│ Layout: [sum:bigint]
│ Estimates: {rows: ? (?), cpu: ?, memory: ?, network: ?}
│ _col0 := sum
└─ Aggregate(FINAL)
│ Layout: [sum:bigint]
│ Estimates: {rows: ? (?), cpu: ?, memory: ?, network: ?}
│ sum := sum("sum_0")
└─ LocalExchange[SINGLE] ()
│ Layout: [sum_0:row(bigint, boolean, bigint, boolean)]
│ Estimates: {rows: ? (?), cpu: ?, memory: ?, network: ?}
└─ RemoteSource[1]
Layout: [sum_0:row(bigint, boolean, bigint, boolean)]
Fragment 1 [SOURCE]
Output layout: [sum_0]
Output partitioning: SINGLE []
Stage Execution Strategy: UNGROUPED_EXECUTION
Aggregate(PARTIAL)
│ Layout: [sum_0:row(bigint, boolean, bigint, boolean)]
│ sum_0 := sum("col1")
└─ ScanFilterProject[table = druid:druid.<table name> druid.druid.<table name> columns=[__time:timestamp(3):TIMESTAMP, col1:bigint:BIGINT], grouped = false, filterPredicate = ("__time" BETWEEN TIMESTAMP '2021-06-01 00:00:00.000' AND TIMESTAMP '2021-06-30 00:00:00.000')]
Layout: [col1:bigint]
Estimates: {rows: ? (?), cpu: ?, memory: 0B, network: 0B}/{rows: ? (?), cpu: ?, memory: 0B, network: 0B}/{rows: ? (?), cpu: ?, memory: 0B, network: 0B}
__time := __time:timestamp(3):TIMESTAMP
col1 := col1:bigint:BIGINT
seems the filter is pushed down, but from what we observed, the scan rows are still full table, not sure if there is issue in buildSql
sumannewton
hashhar
findepi
jibinpt
maudrid
Praveen2112
In Druid queries,
__timeis a very important filter since applying it narrows down the scan range, otherwise, Druid will do a full table scan, which is very slow and resource-consuming.Currently, I found the
__timeis not pushed down, not sure if I use it wrongly or the pushdown is not supported yet:Trino SQL:
select sum(col1) from <table name> where __time between date'2021-06-01' and date'2021-06-30';SQLs sent to Druid:
SELECT "__time", "col1" FROM "druid"."<table name>"I noticed there are ongoing works (#4109 or #4313) of implementing Druid aggregation pushdown, not sure if this is part work of it