search

tabular-io / iceberg-rest-image

Simple project to expose a catalog over REST using a Java catalog backend
Apache License 2.0
100 stars 40 forks source link

AWS S3 Support #31

Open Ralnoc opened 1 year ago

Ralnoc commented 1 year ago

Add support for AWS S3 as a Iceberg table location through s3a:// as a filesystem to store Iceberg data and metadata.

Included are updates to pre-package the core-site.xml in the correct location so that AWS IAM Role policies will work, regardless to if they are pulling from an EC2 instance IAM Profile, Environment Variables, a credential file or Kubernetes IRSA Web Token authentication.

danielcweeks commented 1 year ago

Thanks @Ralnoc , but I don't think we want to add the Hadoop dependencies here. If we want to add S3 support, it should be through iceberg-aws and the S3FileIO implementation to avoid dragging in Hadoop.