apache / iceberg-go

Apache Iceberg - Go
https://iceberg.apache.org/
Apache License 2.0
142 stars 34 forks source link

IO Implementation using Go CDK #175

Closed loicalleyne closed 1 month ago

loicalleyne commented 1 month ago

This PR extends PR #111

Implements #92. The Go CDK has well-maintained implementations for accessing objects stores from S3, Azure, and GCS via a io/fs.Fs-like interface. However, their file interface doesn't support the io.ReaderAt interface or the Seek() function that Iceberg-Go requires for files. Furthermore, the File components are private. So we copied the wrappers and implement the remaining functions inside of Iceberg-Go directly.

In addition, we add support for S3 Read IO using the CDK, providing the option to choose between the existing and new implementation using an extra property. GCS connection options (authentication, endpoint) can also be passed in the map of properties.