search

apache / dolphinscheduler

Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code
https://dolphinscheduler.apache.org/
Apache License 2.0
12.88k stars 4.63k forks source link

Apache DolphinScheduler deployment for Amazon EKS #16433

Closed SEZ9 closed 3 months ago

SEZ9 commented 3 months ago

Deploy a distributed and highly reliable architecture of Apache DolphinScheduler on Amazon Elastic Kubernetes Service (EKS) using Terraform.

Purpose of the pull request

Brief change log

Verify this pull request

This pull request is code cleanup without any test coverage.

(or)

This pull request is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(or)

Pull Request Notice

Pull Request Notice

If your pull request contain incompatible change, you should also add it to docs/docs/en/guide/upgrede/incompatible.md

boring-cyborg[bot] commented 3 months ago

Thanks for opening this pull request! Please check out our contributing guidelines. (https://github.com/apache/dolphinscheduler/blob/dev/docs/docs/en/contribute/join/pull-request.md)

SbloodyS commented 3 months ago

DS is an open source component, like Flink and Doris. We only want to support open source k8s. Each public cloud has its own customized implementation of k8s, and we don't provide deployment solutions for all of them. These should be implemented by the public cloud itself. I will close this PR, if you insist on your point of view, welcome to open an issue for discussion first. @SEZ9

SEZ9 commented 3 months ago

DS is an open source component, like Flink and Doris. We only want to support open source k8s. Each public cloud has its own customized implementation of k8s, and we don't provide deployment solutions for all of them. These should be implemented by the public cloud itself. I will close this PR, if you insist on your point of view, welcome to open an issue for discussion first. @SEZ9

Thank you for the suggestion, but I still have a different perspective and hope you can reconsider. I have raised an issue here: https://github.com/apache/dolphinscheduler/issues/16434. @SbloodyS

EricGao888 commented 3 months ago

Looks like features this PR introduces include two parts:

  1. Support setting up EKS related stuff with terraform.
  2. Deploy dolphin to EKS with helm chart.

For the first part, is there any way you could keep those terraform related stuff tested against each future release? Since few maintainers in this community are familiar with terraform and have free access to AWS resources at the same time, it is not possible for the community to maintain the code.

For the second part, this PR introduces too many repeated code. As said in this comment https://github.com/apache/dolphinscheduler/issues/16434#issuecomment-2280273926 , if EKS fully compatible with K8S, there is no need to repeat this code. In the future, with other cloud providers, it could be a disaster to maintain these repeated and probably inconsistent code.

Above all, if you could offer a mechanism to keep the terraform related code well-tested against each future release, I'm ok with the first part. For the second part, I don't think it is appropriate to generate repeated code.

SEZ9 commented 3 months ago

@EricGao888 Thank you for your response. Your analysis is correct. This feature is composed of two parts: one is Terraform code, and the other is a Helm chart. Given the unique requirements of cloud service deployment, users need to use Terraform to provision resources suitable for DolphinScheduler and replace resource-related configurations in the Helm scripts.

Part One: Although most of the Helm chart is repetitive, variables have been replaced, and some critical configurations have been exposed for public cloud settings. This part will need to be maintained and updated according to the specific version of the DolphinScheduler Helm chart.

Part Two: As you mentioned, the Terraform script needs to be tested on the cloud. On the AWS side, we will provide dedicated one-time AWS accounts for DolphinScheduler users to conduct testing and releases. I can take responsibility for maintaining these test versions and promptly updating the DolphinScheduler documentation and code.

For this feature, I can also add a DolphinScheduler document explaining how to obtain a test account, perform the deployment, and update it with major releases. Would this be sufficient to proceed with merging the PR?