Support more storage vendors

[lancemao]

Self Checks

• [X] I have searched for existing issues search for existing issues, including closed ones.
• [X] I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [X] Pleas do not modify this template :) and fill in all the required fields.

1. Is this request related to a challenge you're experiencing?

In our case, for security reason, dataset cannot be stored on Amazon or Azure

2. Describe the feature you'd like to see

Could you please add more storage like Huawei Cloud or Alicloud?

3. How will this feature improve your workflow or experience?

For domestic users, it is crucial to have data stored in a secure place, this is very important for large enterprises. Without this feature, we cannot go production

4. Additional context or comments

No response

5. Can you help us with this feature?

• [ ] I am interested in contributing to this feature.

[takatost]

If these vendors are compatible with the S3 protocol, Dify can directly integrate with S3. If they are not compatible, our team has limited resources, so we would greatly welcome contributions.

[fatelei]

assign to me, i can add more extend storage

[LeoQuote]

I personally think there is no need to write additional code to support multiple storage engines. The current Dify supports the S3 protocol, and both Huawei Cloud and Alibaba Cloud you mentioned also support the S3 protocol. For Alibaba Cloud, you can clearly see in the documentation that you can use the AWS S3 SDK to call the relevant interfaces. Although there is no explicit documentation for Huawei Cloud, it also supports the S3 protocol. You can confirm this by raising a ticket with their customer service.

https://help.aliyun.com/zh/oss/developer-reference/migrate-data-from-amazon-s3-to-alibaba-cloud-oss-1?spm=a2c4g.11186623.0.0.3dea4639WitA4X

The only potential issue that may arise is that some vendors may not support the auto S3_ADDRESS_STYLE, only the virtual one. If this issue occurs, you just need to set the environment variables accordingly. This has already been implemented in #2934.

Personally, I suggest that unless you are a developer for a cloud vendor, there is no need to do this kind of development. S3 has already become the de facto standard, and if there are any incompatibilities, you can push the vendors to provide support.

[dosubot[bot]]

Hi, @lancemao

I'm helping the Dify team manage their backlog and am marking this issue as stale. From what I understand, the issue you opened requests support for additional storage vendors such as Huawei Cloud or Alicloud due to security reasons preventing the use of Amazon or Azure. There have been responses from takatost welcoming contributions, fatelei offering to work on the feature, and LeoQuote suggesting that the current Dify supports the S3 protocol, which both Huawei Cloud and Alibaba Cloud also support. LeoQuote also provides guidance on potential issues and suggests pushing the vendors to provide support.

Could you please confirm if this issue is still relevant to the latest version of the Dify repository? If it is, please let the Dify team know by commenting on the issue. Otherwise, feel free to close the issue yourself, or the issue will be automatically closed in 7 days. Thank you!