search

DataSQRL / sqrl

Flexible development framework for building streaming data applications in SQL with Kafka, Flink, Postgres, GraphQL, and more.
https://www.datasqrl.com/
97 stars 14 forks source link

Add full support for flink functions #926

Closed henneberger closed 2 weeks ago

henneberger commented 1 month ago

This is a fairly involved task. Many functions do not work as intended in the current Flink library.

Some functions extend the Convertlet table, which would need to be replicated:

We currently only map the BuiltInFunctionDefinitions, but the FlinkSqlOperatorTable has other functions that get mapped at different phases in flink:

All flink functions should be available by default (if the streaming engine is flink) rather than requiring an import.

mbroecheler commented 1 month ago

What does it mean that many functions 'do not work as intended'? Do they produce runtime errors or they cannot be resolved by the compiler?

What exactly is involved in this task? Thank you!

henneberger commented 1 month ago

What does it mean that many functions 'do not work as intended'? Do they produce runtime errors or they cannot be resolved by the compiler?

What exactly is involved in this task? Thank you!

The two points in the ticket describe the functions that have problematic support: those that extend the convertlet table and those that are custom created in the FlinkSqlOperatorTable but not in the BuiltInFunctionDefinitions (e.g. try_cast). We working on the test coverage to verify the extent of flink function coverage to derisk this item.

henneberger commented 2 weeks ago

Resolved in #952 by moving to the flink function catalog.