search

kwai / blaze

Blazing-fast query execution engine speaks Apache Spark language and has Arrow-DataFusion at its core.
Apache License 2.0
1.19k stars 112 forks source link

introduce StringView #436

Open richox opened 6 months ago

richox commented 6 months ago

Is your feature request related to a problem? Please describe. arrow-rs is supporting new StringViewArray type (https://github.com/apache/arrow-rs/issues/5374), which is likely to achieve better performance by reducing large memcpys.

Describe the solution you'd like

  1. supports StringView basics like hashing, serialization.
  2. supports StringView in methods inside datafusion-ext-functions.
  3. test/benchmark.

Describe alternatives you've considered

Additional context

alamb commented 6 months ago

FWIW I think the main usecases where StringView will Significantly help are:

  1. Low selectivity filters (where most values are not filtered) on string columns
  2. String manipulation functions (like substr) where the output can be calculated by changing the views rather than the underlying string values.

If you are interested and willing to help, getting StringViewArray support into the DataFusion expression evaluator (especially for filtering) would be a huge help