[Feature][Resource Center] Simultaneous support multi resource components

apache / dolphinscheduler

Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code

https://dolphinscheduler.apache.org/

Apache License 2.0

12.66k stars 4.57k forks source link

[Feature][Resource Center] Simultaneous support multi resource components #14120

Open qingwli opened 1 year ago

qingwli commented 1 year ago

Search before asking

[X] I had searched in the issues and found no similar feature requirement.

Description

We have a scene about the need to support multi-resource components simultaneously.

Use case

Users can upload files to s3 or hdfs. We need to change the resource center to support multiple sources and the download files logic when the job running.

Related issues

No response

Are you willing to submit a PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

github-actions[bot] commented 1 year ago

Thank you for your feedback, we have received your issue, Please wait patiently for a reply.

In order for us to understand your request as soon as possible, please provide detailed information, version or pictures.
If you haven't received a reply for a long time, you can join our slack and send your question to channel #troubleshooting

zhongjiajie commented 1 year ago

hi @liqingwang thanks for your issue, but does it a part of the function in https://github.com/apache/dolphinscheduler/issues/10283? we could add this as subtask if it is a part of 10283

qingwli commented 1 year ago

Hi @zhongjiajie , I have saw #10283 , It talks about the data center, but this feature we would like to change in the storage center

zhongjiajie commented 1 year ago

Hi @zhongjiajie , I have saw #10283 , It talks about the data center, but this feature we would like to change in the storage center

I think its scope is not only the data center, it should contain all external connections, such as databases, file systems, machine hosts, and also the resource center. it is just a connector between dolphinscheduler and external system. As you can see we will support zeppelin in that connection center which currently only support config in common.properties and only for one single zeppelin cluster

github-actions[bot] commented 1 month ago

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.