Contribute Clickhouse Connector To Apache Flink

[czy006]

After consultation with the original author @itinycheng of the warehouse, and working together for the better development of this project, we decided to donate it to Apache Flink and become an official Connector. We have already started relevant solution design and discussion. If you are interested, you are welcome to join.

While completing this step, we need to complete the following tasks to ensure donations and maintain project integrity

Apache Flink DEV Discuss E-Mail: FLIP-202

• ASF compliance check, Listen file and dependency check before initiating donation apache

At the same time, we are also carrying out some improvement work related to:

• Modify the client to be compatible with the latest version
• Support boolean type
• Supplement related functional unit tests to better identify problems
• Support Flink latest version
• Add CI and Unit Tests With Connector

....

If you are interested in any of these jobs, feel free to comment below

Here is a to-do list summarizing all our future work

• [x] #103
• [x] #105
• [x] #122
• [x] #130

[itinycheng]

Thanks for the proposal.

• First, we should upgrade the Flink version to 1.17.
• The latest version of ClickHouse JDBC driver is 0.5.0, it has several breaking changes, we can rely on this version?

[czy006]

Thanks for the proposal.

I agree with you. I will create some tasks later At the same time, the documentation may also need to be supplemented and improved.

cc @itinycheng

[czy006]

Design Google Flink Clickhouse Connector doc

cc @itinycheng

[itinycheng]

Design Google Flink Clickhouse Connector doc

cc @itinycheng

@czy006

I read the document and have two questions:

1. Maybe more protocols will be supported in the future, is it better to let users specify their own protocols? Just like descriptions from clickhouse.com:

   
   URL Syntax: jdbc:(ch|clickhouse)[:<protocol>]://endpoint1[,endpoint2,...][/<database>][?param1=value1&param2=value2][#tag1,tag2,...], for example:

jdbc:ch://localhost is same as jdbc:clickhouse:http://localhost:8123 jdbc:ch:https://localhost is same as jdbc:clickhouse:http://localhost:8443?ssl=true&sslmode=STRICT jdbc:ch:grpc://localhost is same as jdbc:clickhouse:grpc://localhost:9100

2. Sometimes it‘s more efficient to use a local table to execute select SQL(such as: read large amounts of table data),  should we add an option(such as: `scan.shard-read-local`) to identify this behavior?

[czy006]

1、If the first type can be supported by the JDBC protocol, I think it is no problem. I suggest that the link format should be consistent with the official one (by the way, GRPC seems to be an unstable and deprecated protocol) 2、scan.shard-read-local is Read Good Idea,I will revise the document later and review it again

[Ryado]

👋 I'm Ryadh from ClickHouse, stumbled this thread which represent an exciting milestone for the connector. Just wanted to let you know that we'll be happy to help if anything is needed or if you have questions about ClickHouse and its roadmap

[itinycheng]

Hi @Ryado Thanks, any new developments about the project will be mentioned here. We will ask for help from you/clickhouse when needed, thanks for your kindness.

[czy006]

👋 I'm Ryadh from ClickHouse, stumbled this thread which represent an exciting milestone for the connector. Just wanted to let you know that we'll be happy to help if anything is needed or if you have questions about ClickHouse and its roadmap

Yeah, we're working on it a while ago because my personal work switch put some things on hold, and now I'm back to keep moving forward. Thank you for your attention. The main issues that are difficult to push forward now are as follows

• Lack of staff with knowledge of both flink and clickhouse projects
• Some parameters of the connector and some api design need to be redesigned
• Basic unit tests are lacking

It would be best if these issues were supported by the clickhouse community

[Ryado]

Thanks for the insights @czy006, the ClickHouse integrations team will look into what we can do to help and we'll propose a plan

[czy006]

Thanks for the insights @czy006, the ClickHouse integrations team will look into what we can do to help and we'll propose a plan

Thank you for your help. Timely communication is welcome. If you have any questions, please contact me. We can continue to communicate on this thread. I will open the design document and discussion document later