apache / opendal

Apache OpenDAL: One Layer, All Storage.
https://opendal.apache.org
Apache License 2.0
3.44k stars 481 forks source link

new feature: Support more storage services and APIs for Go bindings. #5326

Open diqiu50 opened 2 hours ago

diqiu50 commented 2 hours ago

Feature Description

In Go, we are using OpenDAL to implement file and directory operations for storage systems such as S3, HDFS, GCS, OSS, and Azure Blob. We need to support the following features:

  1. Go service support for HDFS, GCS, OSS, and Azure Blob.
  2. Go service support for macOS.
  3. The Go binding API should support the RangeBounds parameter in the Reader, as in the Rust version.

Problem and Solution

The main issues:

  1. The backend storage systems does not support HDFS, GCS, OSS, and Azure Blob..
  2. The Go binding API does not support offset-based read operation.

Additional Context

No response

Are you willing to contribute to the development of this feature?

Xuanwo commented 2 hours ago

cc @yuchanns, any comments?

Xuanwo commented 2 hours ago

Enable services here should be enough: https://github.com/apache/opendal-go-services/blob/main/.github/scripts/matrix.yaml

yuchanns commented 45 minutes ago

Adding support for GCS, OSS, and Azblob should be straightforward, but HDFS requires some testing first.

Yes, for most of the features, it can be enabled by appending services to the matrix of the opendal-go-services.

Adding macOS support should be easy as well, I suppose.

It depends on https://github.com/jupiterrider/ffi. And yes, it requires little effort to support.

Read and write support should be implemented in the C binding first.

I'll take a look at it if nobody is interested.