apache / iceberg-python

Apache PyIceberg
https://py.iceberg.apache.org/
Apache License 2.0
459 stars 166 forks source link

Support for Nessie Rest s3 signer #1028

Closed guitcastro closed 18 hours ago

guitcastro commented 3 months ago

Apache Iceberg version

main (development)

Please describe the bug 🐞

Nessie s3 signer use a different endpoint than /aws/s3/sign and also includes query parameters in the URI. Therefore, we need to honor their s3.signer.endpoint which is currently static in code.

Additionally, we need to parameterize the X-Iceberg-Access-Delegation, which currently has the value vended-credentials hardcoded. This is necessary because Nessie does not return the s3.signer.endpoint when the value is set to vended-credentials.

An example of Nessie config from get table endpoint:

    "config": {
        "s3.path-style-access": "true",
        "s3.signer.uri": "http://127.0.0.1:19120/iceberg/",
        "s3.endpoint": "http://minio:9000/",
        "io-impl": "org.apache.iceberg.aws.s3.S3FileIO",
        "s3.signer.endpoint": "v1/main/s3-sign/default.datasets?e=1723099768&b=s3%3A%2F%2Fwarehouse&k=f1379b71-f73a-4fcd-8765-4a1aba6a3585&s=22f3f6b52d586c3b9b310eb2df0e2e51d0c8d0b03030f5bb25f8b2548ef406da&w=s3%3A%2F%2Fwarehouse%2Fdefault%2Fdatasets_f244e57f-143a-4e19-b1c6-591cafbe96fe",
        "s3.remote-signing-enabled": "true",
        "client.region": "us-east-1"
    }
guitcastro commented 2 months ago

Create the PRs #1029 and #1033 to fix it.