search

apache / inlong

Apache InLong - a one-stop, full-scenario integration framework for massive data
https://inlong.apache.org/
Apache License 2.0
1.4k stars 530 forks source link

[Feature][Manager] Support Dolphinscheduler schedule engine #11401

Closed aloyszhang closed 1 week ago

aloyszhang commented 1 month ago

Description

Currently, InLong supports both real-time and offline data synchronization. The offline data sync is scheduled repeatedly by the schedule engine.

10412 introduced a built-in schedule engine and #10515 supports the built-in engine to trigger the Flink batch job.

So, InLong has the basic capability for offline data synchronization.

There are some mature and stable scheduling engines like Apache Airflow and Apache Dolphinscheduler. It will be helpful for Inlong to support these scheduling engines in offline data synchronization.

This issue is part of this which aims to support Dolphinscheduler Airflow.

Ref: InLong offline synchronization Offline data synchronization Example Offline Sync Connector Extension

Use case

No response

Are you willing to submit PR?

Code of Conduct

emptyOVO commented 1 month ago

I'm interested in related implementation, could you plz assign this to me?

aloyszhang commented 1 month ago

I'm interested in related implementation, could you plz assign this to me?

Sure, I've assigned it to you.