Open lmatz opened 2 years ago
Good point!
This can also be done by query optimization:
Join -> Agg
(->
means before) with the same group key and join key can be optimized into Agg -> Join
(see also Eager Aggregation and Lazy AggregationAgg -> Join
with the same group key and join key can be optimized into a single special physical operator HashGroupJoin
to reuse the hash table between HashAgg and HashJoin.In this way, Agg and Join will be fused into one operator.
Seems like group join optimization.
Very interesting. I wonder what kind of runtime support is needed to support "state prefetching". 🤔
Very interesting. I wonder what kind of runtime support is needed to support "state prefetching". 🤔
If we target this the same key
scenario, then query optimization is the way to go. We fuse two operators into one, essentially eliminating the need for prefetching
as this one big fused operator would do fetching
for the state of both operators before operator fusion. No additional runtime support is needed.
If we target some other scenarios that operator fusion cannot help and prefetching must be done at runtime., we can enable each executor not only to deal with StreamChunk
and Barrier
, but also a PrefetchHint
. Then downstream executors both locally and remotely can be asked to prefetch its state in the same way. This may be a good solution engineering-wise but not performance-wise.
In terms of the storage, we may need to pin the prefetched state in the cache until the arrival of the StreamChunk
to be processed, or some scan-resistant
caching policy may also do the trick.
Good point!
This can also be done by query optimization:
1. `Join -> Agg` (`->` means before) with the same group key and join key can be optimized into `Agg -> Join` (see also [Eager Aggregation and Lazy Aggregation](https://www.vldb.org/conf/1995/P345.PDF) 2. `Agg -> Join` with the same group key and join key can be optimized into a single special physical operator `HashGroupJoin` to reuse the hash table between HashAgg and HashJoin.
In this way, Agg and Join will be fused into one operator.
I'm wondering if anyone is working on this at the moment? If not I'm interested in implementing this.
Good point! This can also be done by query optimization:
1. `Join -> Agg` (`->` means before) with the same group key and join key can be optimized into `Agg -> Join` (see also [Eager Aggregation and Lazy Aggregation](https://www.vldb.org/conf/1995/P345.PDF) 2. `Agg -> Join` with the same group key and join key can be optimized into a single special physical operator `HashGroupJoin` to reuse the hash table between HashAgg and HashJoin.
In this way, Agg and Join will be fused into one operator.
I'm wondering if anyone is working on this at the moment? If not I'm interested in implementing this.
This might need some work and I would recommend to defer it until some real cases appear.
In this case,
Agg
is downstream ofHashJoin
. The group-by-key ofAgg
happens to be the same as the join key ofHashJoin
. When a new input record comes from the left input ofHashJoin
, we fetch/probe the state of the right join side. At the same time, we can prefetch the state ofAgg
with respect to the same key:HashJoin
produces nothing, the prefetch does not help.HashJoin
does produce output, then ideally, the fetching ofAgg
's state is overlapped with the fetching ofHashJoin
's right side state.Considering the latency of
get
from S3 is likely to dominate the overall e2e latency, the improvement could be substantial.In this case, we have an
Agg
as the right child ofHashJoin
. (TheExchange
between them should be optimized away as the right child has already been partitioned bykey=[0]
once before.)Unlike the case above, the
Agg
in this case is upstream ofHashJoin
and must produce some outputs. Therefore, prefetching the state forHashJoin
while fetching the state forAgg
is very likely to be beneficial.Just a thought. The scenarios may be quite limited in real workload though. Sort of like operator fusion but it only fuses the state access but not computation.