[Feature][resource center] Can the resource center support Tencent Cloud COS

apache / dolphinscheduler

Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code

https://dolphinscheduler.apache.org/

Apache License 2.0

12.88k stars 4.63k forks source link

[Feature][resource center] Can the resource center support Tencent Cloud COS #16396

Closed kiwi-field closed 1 month ago

kiwi-field commented 3 months ago

Search before asking

[X] I had searched in the issues and found no similar feature requirement.

Description

I am currently using HDFS in the resource center and I want to migrate to Tencent Cloud COS uniformly. What changes do I need to make? Tencent Cloud's bucket is public read

Use case

After switching to Tencent Cloud COS, you can perform upload, download, and preview operations in the resource center

Related issues

No response

Are you willing to submit a PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

SbloodyS commented 3 months ago

Currently there are no plans to support it.

kiwi-field commented 3 months ago

Can I connect to Tencent Cloud COS through the dependency of S3Utils and AWS Java SDK S3

SbloodyS commented 3 months ago

You can try it, but cos doesn't work well with S3.

kiwi-field commented 3 months ago

The above picture shows our program configuration, and when downloading from the resource center, the following error occurred。

kiwi-field commented 3 months ago

It is obvious that AWS Java SDK S3 is designed for Amazon S3 cloud services, not the S3 protocol

kiwi-field commented 3 months ago

The optional regions are shown in the above figure

kiwi-field commented 3 months ago

@SbloodyS

fuchanghai commented 2 months ago

for me ,it work

Mighten commented 2 months ago

Hi, I am currently working on this feature (#16396) to support Tencent Cloud COS as a storage plugin in DS.

Could you please assign this issue to me?

This work is due by September 30, 2024.

fuchanghai commented 2 months ago

Hi, I am currently working on this feature (#16396) to support Tencent Cloud COS as a storage plugin in DS.

Could you please assign this issue to me?

This work is due by September 30, 2024.

done