[Task Scheduling] Support scheduling database SQL

apache / incubator-streampark

Make stream processing easier! Easy-to-use streaming application development framework and operation platform.

https://streampark.apache.org/

Apache License 2.0

3.91k stars 1.01k forks source link

[Task Scheduling] Support scheduling database SQL #1627

Closed ChenpiDog closed 1 year ago

ChenpiDog commented 2 years ago

Search before asking

[X] I had searched in the feature and found no similar feature requirement.

Description

At present, the landing scenarios of Flink are limited. Batch processing still requires the help of database SQL and other technologies. It is expected that the scheduled scheduling function of database SQL can be added.

Usage Scenario

No response

Related issues

No response

Are you willing to submit a PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

1996fanrui commented 2 years ago

Hi @ChenpiDog , I'm sorry, I don't understand your mean:Batch processing still requires the help of database SQL and other technologies..

Do you want to schedule your Flink Batch job with other scheduling frameworks? If yes, you can follow this issue. #640

ChenpiDog commented 2 years ago

@1996fanrui For example, MySQL and Greenplum SQL statements.

ChenpiDog commented 2 years ago

@1996fanrui It is hoped that streampark can schedule database sql to achieve unified scheduling and management of batch tasks and stream processing tasks, otherwise, additional tools such as airflow will have to be used.

ChenpiDog commented 2 years ago

Hi @ChenpiDog , I'm sorry, I don't understand your mean:Batch processing still requires the help of database SQL and other technologies..

Do you want to schedule your Flink Batch job with other scheduling frameworks? If yes, you can follow this issue. #640

I mean that the batch processing capability of flink is limited, and the database sql used in production does batch processing.