Open BugenZhao opened 1 year ago
This seems hard to do this under current architecture as we always use static type for arrays, so introducing a wrapper requires a lot of changes.
Can you elaborate this? How is "static type" a problem and how dynamic is ConstantArray
/RunArrary
?
This seems hard to do this under current architecture as we always use static type for arrays, so introducing a wrapper requires a lot of changes.
Can you elaborate this? How is "static type" a problem and how dynamic is
ConstantArray
/RunArrary
?
For example, arrays in arrow
and arrow2
are all trait objects, so it can introduce a RunArray
wrapper easily without exposing it to any callers. However in our type system, we need to write a lot of stuff like MaybeRun<Utf8Array>
or MaybeRun<ArrayImpl>
. š¤
How is the situation now after #9049?
How is the situation now after #9049?
I guess the ultimate solution should be allowing Value::Scalar
to directly be passed among different executors and even remote actors, as described in https://github.com/risingwavelabs/risingwave/pull/9733#issuecomment-1543658669. But yes, It appears that #9049 has accomplished everything we can do without introducing a significant refactor. š
FWIW this looks similar š https://github.com/apache/arrow-rs/issues/1047
Wonder if we can further generalize this into some compact encoding for multiple repeated datums. It could potentially optimize join performance, since the datums in the join key don't need to be expanded inline.
Specifically for high amplification join, when building the new chunk, the probe side's record, just needs to convert its scalar values into constant array, then we can just concat that with the build side to form the new stream chunk.
Wonder if we can further generalize this into some compact encoding for multiple repeated datums.
Yes. Are you referring to...
- Run-Length Encoding (arrow-array)
Just FYI: eval_v2
, introduced in #9049, is not adopted by all proc-macro-generated function impl any more.
For example, there's an
EXTRACT(HOUR FROM col)
in Nexmark Q14, where theHOUR
is compiled to a literalVARCHAR
expression. When evaluating theEXTRACT
, we need to first repeat the same scalar"HOUR"
1024 times into an array, then evaluate the outerEXTRACT
function. This is not efficient. https://github.com/risingwavelabs/risingwave/issues/8503#issuecomment-1465808412Possible solutions:
Introduce
ConstantArray
than only stores the scalar and the time it appears, which is essentially a special case of Run-Length Encoding (arrow-array). This seems hard to do this under current architecture as we always use static type for arrays, so introducing a wrapper requires a lot of changes.Check whether an argument of the expression is constant (literal) during the
build_from_proto
with macro (introduced in #8499). In this case, we're not able to handle the structure where aliteral
is nested under another expression, though in most cases this should be folded by the optimizer.Allow the expression to directly return a scalar, and expands it into an array by repeating only if necessary. This sounds much simpler and the refactoring can be progressive.