apache / datafusion

Apache DataFusion SQL Query Engine
https://datafusion.apache.org/
Apache License 2.0
6.3k stars 1.19k forks source link

Convert `BuiltInWindowFunction::{NthValue}` to a user defined functions #12649

Closed alamb closed 2 days ago

alamb commented 1 month ago

Is your feature request related to a problem or challenge?

Part of https://github.com/apache/datafusion/issues/8709

There is now no difference between "built in" / "prepackaged" scalar and aggregate functions in DataFusion, however there are still some "built in" window functions -- see the current source for BuiltInWindowFunction for the up to date list of what remains

The problem with having two different kinds of window functions is

  1. There are some features that may not be available to User Defined Window Functions that rely on built in
  2. Users can not easily choose which window functions to include or override the behavior if they need something different

Describe the solution you'd like

I would like to remove the "built in" version of this function and convert it to a user defined function

Describe alternatives you've considered

At a high level the process is:

  1. Add a new WindowUDFImpl in the functions-window crate
  2. Port the code from the relevant existing implementation of the the built in functions in datafusion/physical-expr/src/window
  3. Remove the BuiltInWindowFunction variant and then get everything to compile (the compiler will show you where the existing implementations are)

Additional context

Here are some good examples:

alamb commented 1 month ago

I think this one is also relatively straightforward and would make a good first project for someone skilled in Rust who wanted to get experience with the Datafusion codebase and tests

dharanad commented 1 month ago

take

Omega359 commented 1 month ago

Convert BuiltInScalarFunction::{NthValue} -> Convert BuiltInWindowFunction::{NthValue}

alamb commented 1 month ago

Convert BuiltInScalarFunction::{NthValue} -> Convert BuiltInWindowFunction::{NthValue}

🤦 -- fixed

jatin510 commented 3 weeks ago

Hi, @dharanad . If you are busy can I look into this !

dharanad commented 3 weeks ago

Hi, @dharanad . If you are busy can I look into this !

Sure

buraksenn commented 2 weeks ago

take