search

apache / kyuubi

Apache Kyuubi is a distributed and multi-tenant gateway to provide serverless SQL on data warehouses and lakehouses.
https://kyuubi.apache.org/
Apache License 2.0
2.07k stars 902 forks source link

[SUB-TASK][KPIP-2] Support Flink SQL Client startup options. #2153

Open a49a opened 2 years ago

a49a commented 2 years ago

Code of Conduct

Search before asking

What would you like to be improved?

The following issue only achieves FlinkSQLEngine process, but users have no way to set --jar and --library options in Kyuubi. We should achieve this in KyuubiServer process too.

https://github.com/apache/incubator-kyuubi/issues/1685 https://github.com/SteNicholas/incubator-kyuubi/commit/dd5a704342334feb24eae981d184943f618dd228

We can put --jar and --library options in kyuuubi-default.conf, and KyuubiServer process load these options.

How should we improve?

KyuubiServer process loads the Flink SQL Client startup options in kyuuubi-default.conf. And then KyuubiServer transforms the options to standard Flink SQL Client startup options and passes them to flink-sql-engine.sh when the flink-sql-engine.sh process will be started.

The kyuuubi-default.confoptions like the following.

flink.client.option.jar foo.jar
flink.client.option.library /foo/bar

Now flink-sql-engine.sh startup options are nothing, so users can't set any options.

Are you willing to submit PR?

a49a commented 2 years ago

The FlinkSQLEngine process life is inconsistent with Session. One FlinkSQLEngine process corresponds to multiple sessions. So flink-sql-engine.sh can not use configuration of Session or ExecuteStatement.

pan3793 commented 2 years ago

The FlinkSQLEngine process life is inconsistent with Session. One FlinkSQLEngine process corresponds to multiple sessions. So flink-sql-engine.sh can not use configuration of Session or ExecuteStatement.

From Flink document, Flink support ADD JAR and REMOVE JAR commands, what's the difference between use ADD JAR and inject jars on launching Flink engine?

pan3793 commented 2 years ago

The kyuuubi-default.confoptions like the following.

flink.client.option.jar foo.jar
flink.client.option.library /foo/bar

Give some comments based on my little understand of Flink, please correct me if I'm wrong.

There is no equality configuration in Flink of --jar and --library options defined by Flink CLI, if yes we'd better to introduce Kyuubi's style configurations instead of Flink's style to avoid potential conflict, e.g.

kyuubi.session.engine.flink.jar
kyuubi.session.engine.flink.library
pan3793 commented 2 years ago

https://github.com/apache/incubator-kyuubi/blob/3eb1fa9e48b4c533f05b3719486df198c6e29751/externals/kyuubi-flink-sql-engine/src/main/scala/org/apache/kyuubi/engine/flink/FlinkSQLEngine.scala#L79

BTW, I think we can modify this line to support overwriting Flink configuration by flink-sql-engine.sh, then we can achieve same experience like Spark engine, which means we can define Flink confs in JDBC URL, kyuubi-defaults.conf, flink-conf.yaml, and the former has higher priority.

a49a commented 2 years ago

https://github.com/apache/incubator-kyuubi/blob/3eb1fa9e48b4c533f05b3719486df198c6e29751/externals/kyuubi-flink-sql-engine/src/main/scala/org/apache/kyuubi/engine/flink/FlinkSQLEngine.scala#L79

BTW, I think we can modify this line to support overwriting Flink configuration by flink-sql-engine.sh, then we can achieve same experience like Spark engine, which means we can define Flink confs in JDBC URL, kyuubi-defaults.conf, flink-conf.yaml, and the former has higher priority.

I spilled a new issue, I will achieve this first.

a49a commented 2 years ago

The kyuuubi-default.confoptions like the following.

flink.client.option.jar foo.jar
flink.client.option.library /foo/bar

Give some comments based on my little understand of Flink, please correct me if I'm wrong.

There is no equality configuration in Flink of --jar and --library options defined by Flink CLI, if yes we'd better to introduce Kyuubi's style configurations instead of Flink's style to avoid potential conflict, e.g.

kyuubi.session.engine.flink.jar
kyuubi.session.engine.flink.library

I second with introducing Kyuubi's style configurations, but this has nothing to do with session.

Could we design prefix is kyuubi.engine.flink.?

kyuubi.engine.flink.jar
kyuubi.engine.flink.library
a49a commented 2 years ago

https://issues.apache.org/jira/browse/FLINK-27010