Open alamb opened 1 week ago
take
I am hoping we can figure out a straightforward pattern that can generate code for any string function that works well for StringArray as well as LargeStringArry and StringViewArray
Agree, I have a prototype StringArrayType
(https://github.com/apache/arrow-rs/pull/5931) for LIKE
operations (like
, ilike
, contains
, begins_with
etc). A similar pattern could be applied here.
Maybe we can directly use the StringArrayType
trait here.
Is your feature request related to a problem or challenge?
Part of https://github.com/apache/datafusion/issues/10918 where we are integrating
StringView
into DataFusion, initially targeting making ClickBench queries fasterIn the ClickBench queries there is a
REGEXP_REPLACE
function on String columns such ashttps://github.com/apache/datafusion/blob/5bfc11ba4ac4f11eaf9793c668e4a064fb697e6e/benchmarks/queries/clickbench/queries.sql#L29
Describe the solution you'd like
I would like to be able to run
REGEXP_REPLACE
on string view columnsGiven this table:
I would like to be able to run this function
It works fine if you cast the column first to a string:
Describe alternatives you've considered
We could add a coercion rule to automatically cast
Utf8View
to Utf8 which we probably should do in general to make it easy to work with Utf8ViewHowever that is inefficient as it will involve copying all the strings. It would be much better to actually implement
REGEXP_REPLACE
forUtf8View
arrays directlyI am hoping we can figure out a straightforward pattern that can generate code for any string function that works well for StringArray as well as LargeStringArry and StringViewArray
Here is how the regexp replace function is implemented now:
https://github.com/apache/datafusion/blob/5bfc11ba4ac4f11eaf9793c668e4a064fb697e6e/datafusion/functions/src/regex/regexpreplace.rs#L142-L182
Additional context
Please remember to target the
string-view
branch in DataFusion, rather thanmain
with your PR