STS needs to be statically linked

[osalloum]

This is needed to support loading roles on kubernetes pod using web identity federation, if STS is not there, the aws-sdk credentials chain will ignore this method and fallback to the instance metadata role which would not be the expected role for the application code

https://github.com/duckdb/duckdb_aws/issues/31

[samansmink]

Thanks for the PR! After https://github.com/duckdb/duckdb_aws/pull/46 is merged, then this will be available in v1.0.0 to install from the nightly repository using:

force install aws from core_nightly

edit: note that the windows failure is being fixed upstream so that should be fixed soonish

[osalloum]

Thanks @samansmink

This still somehow does not work with all docker images. If you try to use Alpine or Slim images, it wouldn't work but if you combine with an Amazon Linux based docker image it would work

As examples: amazoncorretto:21 --> works eclipse-temurin:21 --> does not work

public.ecr.aws/lambda/python:3.10 --> works

etc

[samansmink]

Thanks for checking @osalloum! oh thats interesting. Are you using python binaries on ARM based machines? That could be a clue here. For ARM we distribute both a manylinux_2_17 and a manylinux_2_24 wheel

[osalloum]

I tried on arm64v8/eclipse-temurin:21 and it works! where as the amd64 variant before does not This test was run with the CLI https://github.com/duckdb/duckdb/releases/download/v1.0.0/duckdb_cli-linux-aarch64.zip

Now to try the with the wheels I went back and tried with python:3.11-slim (ie amd64) and it did not work

Next test on arm64v8/python:3.11-slim using duckdb==1.0.0 and force install aws from core_nightly it works

Then i try installing both of the variants

manylinux_2_17_aarch64

pip3 install --force https://files.pythonhosted.org/packages/08/7b/d646a7483d9d00d3438d264773dc2c65e447d2456694934c2ef734626c92/duckdb-1.0.0-cp311-cp311-manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl

# run my tests
root@pyso-d5bdb6698-rlvm4:/py# python3 ba.py
Traceback (most recent call last):
  File "/py/ba.py", line 6, in <module>
    db.execute("install aws ")
duckdb.duckdb.HTTPException: HTTP Error: Failed to download extension "aws" at URL "http://extensions.duckdb.org/v1.0.0/linux_arm64_gcc4/aws.duckdb_extension.gz"
Extension "aws" is an existing extension.

root@pyso-d5bdb6698-rlvm4:/py# vim ba.py # remove the install call
root@pyso-d5bdb6698-rlvm4:/py# python3 ba.py
Traceback (most recent call last):
  File "/py/ba.py", line 7, in <module>
    print(db.execute("call load_aws_credentials(redact_secret=false)").fetchall())
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
duckdb.duckdb.Error: An error occurred while trying to automatically install the required extension 'aws':
Failed to download extension "aws" at URL "http://extensions.duckdb.org/v1.0.0/linux_arm64_gcc4/aws.duckdb_extension.gz"
Extension "aws" is an existing extension.

manylinux_2_24_aarch64 variant

pip3 install --force https://files.pythonhosted.org/packages/08/7b/d646a7483d9d00d3438d264773dc2c65e447d2456694934c2ef734626c92/duckdb-1.0.0-cp311-cp311-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl

# run my tests

and it works

Before each test i would delete any duckdb related files

 rm -Rf ~/.duckdb 
rm -Rf /usr/local/lib/python3.11/site-packages/duckdb

Is there anything which i can do to get the manylinux_2_17_aarch64 test working?

[osalloum]

Another good way to know if the credentials works properly on amd64: is using Fargate(serverless compute engine) on EKS because Fargate does not have any real NodeGroup behind it, just a virtual node from AWS which does not have a node role

When the code is not able to load credentials using STS (ie AWS_WEB_IDENTITY_TOKEN_FILE) then it would be give empty results, interestingly enough

Unfortunately EKS only supports Fargate for amd64 and not for arm64, so i can't do that test