[Feature-3960] Add task distribution for streampark

HxpSerein commented 2 months ago

What changes were proposed in this pull request

Issue Number: close https://github.com/apache/incubator-streampark/issues/3960

Brief change log

Add task distribution for streampark

Verifying this change

This change added tests and can be verified as follows:

ConsistentHashTest
FlinkTaskServiceTest

Does this pull request potentially affect one of the following parts

Dependencies (does it add or upgrade a dependency): no

Mrart commented 2 months ago

Thanks for contributing, I'm leaving a few comments, 1) We only support flink yarn jobs? we need supported k8s and spark jobs. 2) We can have more test cases to protect the stability of the program. So far we have only tested consistent hashes,.

HxpSerein commented 2 months ago

Thank you, @Mrart, for your review.

I fully agree with your suggestion to add more test cases to enhance the program's stability. So far, we've only tested consistent hashing, and additional tests for other scenarios will be helpful moving forward.

We can have more test cases to protect the stability of the program. So far we have only tested consistent hashes.

This PR modifies the task distribution logic in ApplicationActionServiceImpl. For single-node scenarios, the current logic will still be executed locally. In multi-node HA scenarios, since Streampark currently doesn’t support multi-node integration testing (IT), and adding IT tests in the short term is challenging, I plan to add unit tests (UT) for logic correctness. Once the resource center is integrated, we can proceed with multi-node cluster deployment testing.

We only support flink yarn jobs? we need supported k8s and spark jobs.

Since the resource center is not yet developed and HA is incomplete, the current testing support is somewhat limited. To ensure code stability, I propose to temporarily support only Flink tasks, with support for Kubernetes and Spark jobs added once the resource center is completed. Do you have any suggestions on this approach? @Mrart @wolfboys

Mrart commented 2 months ago

Since the resource center is not yet developed and HA is incomplete, the current testing support is somewhat limited. To ensure code stability, I propose to temporarily support only Flink tasks, with support for Kubernetes and Spark jobs added once the resource center is completed. Do you have any suggestions on this approach? @Mrart @wolfboys

I agree it.

Mrart commented 2 months ago

Another suggestion is that we merge our commits before we merge them and change the commit to match what we changed