[Feature] Flink Kubernetes Module V2

[Al-assad]

Search before asking

• [X] I had searched in the feature and found no similar feature requirement.

Description

Due to many issues with the current version of StreamPark Flink on Kubernetes in terms of submitting and tracking tasks, we hope to refactor the entire module based on the Flink Kubernetes Operator to better address the following problems:

• Inaccurate and confusing tracking of Flink on Kubernetes status;
• Unstable and lengthy Flink K8s Application Job submission process;
• Scattered Event Watcher thread pool code;

The following is the design document for the StreamPark Flink Kubernetes V2 module: https://www.craft.do/s/Pu5GI6vr4KIrj5

At the same time, I have quickly developed a runnable prototype for StreamPark Flink Kubernetes V2: https://github.com/Al-assad/streampark-flink-kubernetes-v2

We welcome your ideas and feedback on this proposed change.

This is a fairly big change, and after the relevant designs are approved, I will break down the task into multiple small, risk-controlled tasks for tracking in this issue. In the design document, I have detailed the plan for advancing v2 and specific code modifications. Any developer interested in StreamPark, Flink on Kubernetes, Scala, or ZIO can claim corresponding tasks to participate.

Are you willing to submit a PR?

• [X] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[wolfboys]

✌️ good job. Looks good 😂

[GOODBOY008]

Especially looking forward to this feature ！