apache / dolphinscheduler

Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code
https://dolphinscheduler.apache.org/
Apache License 2.0
12.67k stars 4.57k forks source link

[Feature][resource center] Can the resource center support Tencent Cloud COS #16396

Open kiwi-field opened 1 month ago

kiwi-field commented 1 month ago

Search before asking

Description

I am currently using HDFS in the resource center and I want to migrate to Tencent Cloud COS uniformly. What changes do I need to make? Tencent Cloud's bucket is public read

Use case

After switching to Tencent Cloud COS, you can perform upload, download, and preview operations in the resource center

Related issues

No response

Are you willing to submit a PR?

Code of Conduct

SbloodyS commented 1 month ago

Currently there are no plans to support it.

kiwi-field commented 1 month ago

Can I connect to Tencent Cloud COS through the dependency of S3Utils and AWS Java SDK S3

SbloodyS commented 1 month ago

You can try it, but cos doesn't work well with S3.

kiwi-field commented 1 month ago

image The above picture shows our program configuration, and when downloading from the resource center, the following error occurred。 image

kiwi-field commented 1 month ago

It is obvious that AWS Java SDK S3 is designed for Amazon S3 cloud services, not the S3 protocol

kiwi-field commented 1 month ago

image The optional regions are shown in the above figure

kiwi-field commented 1 month ago

@SbloodyS

fuchanghai commented 2 weeks ago

for me ,it work

Mighten commented 1 week ago

Hi, I am currently working on this feature (#16396) to support Tencent Cloud COS as a storage plugin in DS.

Could you please assign this issue to me?

This work is due by September 30, 2024.

fuchanghai commented 1 week ago

Hi, I am currently working on this feature (#16396) to support Tencent Cloud COS as a storage plugin in DS.

Could you please assign this issue to me?

This work is due by September 30, 2024.

done