Assuming IAM role from within a EKS Pod Identity-enabled container does not work using named profile

rkubik-hostersi commented 1 month ago

Describe the bug

When working on a pod in EKS with Pod Identity assigned, it is not possible to assume another role using ~/.aws/config and profiles.

When specifying role_arn in ~/.aws/config, it is required to provide source_profile or credential_source. Since we are in the pod, source_profile is not an option. Unfortunately credential_source is pretty limited:

Environment value does not work as there is no env variables
Ec2InstanceMetadata points to the IAM role attached to the EC2, Pod Identity is not being used
EcsContainer is for ECS

Expected Behavior

It should be possible to instruct aws-cli to use EKS Pod Identity as a credential_source.

Current Behavior

It is not possible to utilize aws-cli with Assume Role mechanism using named profiles within ~/.aws/config when working on a EKS Pod Identity-enabled pods.

Reproduction Steps

Create EKS with Pod Identity agent
Assign sts:assumeRole permission to the pod
Prepare IAM role to be assumed
Create the pod with the Pod Identity assigned, prepare ~/.aws/config
Try to assume a different IAM role using aws --profile

Possible Solution

No response

Additional Information/Context

No response

CLI version used

2.15.57

Environment details (OS name and version, etc.)

aws-cli/2.15.57 Python/3.12.6 Linux/6.8.0-41-generic source/x86_64.alpine.3

drunkensway commented 1 month ago

experiencing this as well using hashicorp/terraform:1.5.6.

after installing the aws cli and running aws configure set role_arn <role-arn> getting:

Error relocating /usr/lib/python3.11/lib-dynload/pyexpat.cpython-311-x86_64-linux-musl.so: XML_SetReparseDeferralEnabled: symbol not found

tim-finnigan commented 1 month ago

experiencing this as well using hashicorp/terraform:1.5.6.

after installing the aws cli and running aws configure set role_arn <role-arn> getting:

Error relocating /usr/lib/python3.11/lib-dynload/pyexpat.cpython-311-x86_64-linux-musl.so: XML_SetReparseDeferralEnabled: symbol not found

Same error as https://github.com/aws/aws-cli/issues/8913, replied there:

Looks like this is the same as hashicorp/terraform#35715, where a member of Terraform replied:

The Dockerfile for the build wasn't changed during that time, so any differences would be solely from the upstream image. Your above example works correctly if the package is updated, and I also confirmed that newer images have already updated the problematic packages. Closing since there's nothing the Terraform CLI can do to fix the old docker image.

Can you confirm that this is fixed in newer images?

tim-finnigan commented 1 month ago

But the original issue here looks related to https://github.com/aws/aws-cli/issues/3875 and https://github.com/aws/aws-sdk/issues/350.

jcary741 commented 1 month ago

I am encountering this as well, which is breaking our gitlab CI that uses apk add aws-cli.

Here is the relevant section from a working run from yesterday:

$ apk add --no-cache aws-cli
fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/APKINDEX.tar.gz
(1/59) Installing libbz2 (1.0.8-r5)
(2/59) Installing libffi (3.4.4-r2)
(3/59) Installing gdbm (1.23-r1)
(4/59) Installing xz-libs (5.4.3-r0)
(5/59) Installing libgcc (12.2.1_git20220924-r10)
(6/59) Installing libstdc++ (12.2.1_git20220924-r10)
(7/59) Installing mpdecimal (2.5.1-r2)
(8/59) Installing libpanelw (6.4_p20230506-r0)
(9/59) Installing readline (8.2.1-r1)
(10/59) Installing sqlite-libs (3.41.2-r3)
(11/59) Installing python3 (3.11.8-r1)
(12/59) Installing python3-pycache-pyc0 (3.11.8-r1)
(13/59) Installing pyc (0.1-r0)
(14/59) Installing py3-certifi (2024.2.2-r0)
(15/59) Installing py3-certifi-pyc (2024.2.2-r0)
(16/59) Installing py3-cparser (2.21-r2)
(17/59) Installing py3-cparser-pyc (2.21-r2)
(18/59) Installing py3-cffi (1.15.1-r3)
(19/59) Installing py3-cffi-pyc (1.15.1-r3)
(20/59) Installing py3-cryptography (41.0.3-r0)
(21/59) Installing py3-cryptography-pyc (41.0.3-r0)
(22/59) Installing py3-six (1.16.0-r6)
(23/59) Installing py3-six-pyc (1.16.0-r6)
(24/59) Installing py3-dateutil (2.8.2-r3)
(25/59) Installing py3-dateutil-pyc (2.8.2-r3)
(26/59) Installing py3-distro (1.8.0-r2)
(27/59) Installing py3-distro-pyc (1.8.0-r2)
(28/59) Installing py3-colorama (0.4.6-r3)
(29/59) Installing py3-colorama-pyc (0.4.6-r3)
(30/59) Installing py3-docutils (0.19-r4)
(31/59) Installing py3-docutils-pyc (0.19-r4)
(32/59) Installing py3-jmespath (1.0.1-r1)
(33/59) Installing py3-jmespath-pyc (1.0.1-r1)
(34/59) Installing py3-urllib3 (1.26.18-r0)
(35/59) Installing py3-urllib3-pyc (1.26.18-r0)
(36/59) Installing py3-wcwidth (0.2.6-r2)
(37/59) Installing py3-wcwidth-pyc (0.2.6-r2)
(38/59) Installing py3-prompt_toolkit (3.0.38-r1)
(39/59) Installing py3-prompt_toolkit-pyc (3.0.38-r1)
(40/59) Installing py3-ruamel.yaml.clib (0.2.7-r1)
(41/59) Installing py3-ruamel.yaml (0.17.28-r0)
(42/59) Installing py3-ruamel.yaml-pyc (0.17.28-r0)
(43/59) Installing aws-cli-pyc (2.15.14-r0)
(44/59) Installing py3-awscrt-pyc (0.20.2-r0)
(45/59) Installing python3-pyc (3.11.8-r1)
(46/59) Installing aws-c-common (0.9.12-r0)
(47/59) Installing aws-c-cal (0.6.9-r0)
(48/59) Installing aws-c-compression (0.2.17-r0)
(49/59) Installing s2n-tls (1.3.47-r0)
(50/59) Installing aws-c-io (0.14.2-r0)
(51/59) Installing aws-c-http (0.8.0-r0)
(52/59) Installing aws-c-sdkutils (0.1.14-r0)
(53/59) Installing aws-c-auth (0.7.14-r0)
(54/59) Installing aws-checksums (0.1.17-r0)
(55/59) Installing aws-c-event-stream (0.4.1-r0)
(56/59) Installing aws-c-mqtt (0.10.1-r0)
(57/59) Installing aws-c-s3 (0.4.10-r0)
(58/59) Installing py3-awscrt (0.20.2-r0)
(59/59) Installing aws-cli (2.15.14-r0)
Executing busybox-1.36.1-r2.trigger
OK: 200 MiB in 100 packages

The CI job then goes on to use the AWS CLI successfully.

And here is a broken one today:

$ apk add --no-cache aws-cli
fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/APKINDEX.tar.gz
(1/59) Installing libbz2 (1.0.8-r5)
(2/59) Installing libffi (3.4.4-r2)
(3/59) Installing gdbm (1.23-r1)
(4/59) Installing xz-libs (5.4.3-r0)
(5/59) Installing libgcc (12.2.1_git20220924-r10)
(6/59) Installing libstdc++ (12.2.1_git20220924-r10)
(7/59) Installing mpdecimal (2.5.1-r2)
(8/59) Installing libpanelw (6.4_p20230506-r0)
(9/59) Installing readline (8.2.1-r1)
(10/59) Installing sqlite-libs (3.41.2-r3)
(11/59) Installing python3 (3.11.10-r0)
(12/59) Installing python3-pycache-pyc0 (3.11.10-r0)
(13/59) Installing pyc (0.1-r0)
(14/59) Installing py3-certifi (2024.2.2-r0)
(15/59) Installing py3-certifi-pyc (2024.2.2-r0)
(16/59) Installing py3-cparser (2.21-r2)
(17/59) Installing py3-cparser-pyc (2.21-r2)
(18/59) Installing py3-cffi (1.15.1-r3)
(19/59) Installing py3-cffi-pyc (1.15.1-r3)
(20/59) Installing py3-cryptography (41.0.3-r0)
(21/59) Installing py3-cryptography-pyc (41.0.3-r0)
(22/59) Installing py3-six (1.16.0-r6)
(23/59) Installing py3-six-pyc (1.16.0-r6)
(24/59) Installing py3-dateutil (2.8.2-r3)
(25/59) Installing py3-dateutil-pyc (2.8.2-r3)
(26/59) Installing py3-distro (1.8.0-r2)
(27/59) Installing py3-distro-pyc (1.8.0-r2)
(28/59) Installing py3-colorama (0.4.6-r3)
(29/59) Installing py3-colorama-pyc (0.4.6-r3)
(30/59) Installing py3-docutils (0.19-r4)
(31/59) Installing py3-docutils-pyc (0.19-r4)
(32/59) Installing py3-jmespath (1.0.1-r1)
(33/59) Installing py3-jmespath-pyc (1.0.1-r1)
(34/59) Installing py3-urllib3 (1.26.18-r0)
(35/59) Installing py3-urllib3-pyc (1.26.18-r0)
(36/59) Installing py3-wcwidth (0.2.6-r2)
(37/59) Installing py3-wcwidth-pyc (0.2.6-r2)
(38/59) Installing py3-prompt_toolkit (3.0.38-r1)
(39/59) Installing py3-prompt_toolkit-pyc (3.0.38-r1)
(40/59) Installing py3-ruamel.yaml.clib (0.2.7-r1)
(41/59) Installing py3-ruamel.yaml (0.17.28-r0)
(42/59) Installing py3-ruamel.yaml-pyc (0.17.28-r0)
(43/59) Installing aws-cli-pyc (2.15.14-r0)
(44/59) Installing py3-awscrt-pyc (0.20.2-r0)
(45/59) Installing python3-pyc (3.11.10-r0)
(46/59) Installing aws-c-common (0.9.12-r0)
(47/59) Installing aws-c-cal (0.6.9-r0)
(48/59) Installing aws-c-compression (0.2.17-r0)
(49/59) Installing s2n-tls (1.3.47-r0)
(50/59) Installing aws-c-io (0.14.2-r0)
(51/59) Installing aws-c-http (0.8.0-r0)
(52/59) Installing aws-c-sdkutils (0.1.14-r0)
(53/59) Installing aws-c-auth (0.7.14-r0)
(54/59) Installing aws-checksums (0.1.17-r0)
(55/59) Installing aws-c-event-stream (0.4.1-r0)
(56/59) Installing aws-c-mqtt (0.10.1-r0)
(57/59) Installing aws-c-s3 (0.4.10-r0)
(58/59) Installing py3-awscrt (0.20.2-r0)
(59/59) Installing aws-cli (2.15.14-r0)
Executing busybox-1.36.1-r2.trigger
OK: 200 MiB in 100 packages

Which then fails with Error relocating /usr/lib/python3.11/lib-dynload/pyexpat.cpython-311-x86_64-linux-musl.so: XML_SetReparseDeferralEnabled: symbol not found

The difference I'm seeing is python 3.11.10-r0 is used now, instead of 3.11.8, so maybe this is a new issue there?

joerawr commented 1 month ago

We are seeing this across our CICD. All versions of 1.5.x are impacted. So far in our brief testing 1.6 through 1.9 are not impacted. We're scrambling to test newer versions and update our shared templates.

Likely that Python 3 version from Alpine is the issue. The timestamp is 9/11:

https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/ python3-3.11.10-r0.apk 11-Sep-2024 10:14 9M

joerawr commented 1 month ago

Here is a similar issue with Alpine 3.18 via Terraform 1.5.7: https://gitlab.alpinelinux.org/alpine/aports/-/issues/16441

tim-finnigan commented 1 month ago

For those using Terraform have you referred to: https://github.com/hashicorp/terraform/issues/35715?

rkubik-hostersi commented 1 month ago

Guys, this is not about terraform or any other library, or even python versions. This is about the missing configuration parameter for credential_source when running aws in EKS Pod Identity enabled container. AWS CLI version also does not matter as there is no "legit" parameter to be used in EKS on PI containers and credential_source.

The scenario has been described in the first post. We need to be able to use aws --profile from within a pod to assume some external role with Pod Identity. This is not possible for now officially. :)

jcary741 commented 1 month ago

My bad @rkubik-hostersi, the timing of when you submitted this issue and the environment you described, then followed by what drunkensway said made me think we were encountering different versions of the same problem. I see now that your submission is actually different. Just to update anyone who happens upon this issue who makes the same mistake, the issue we were encountering appears to have been resolved in Python build 3.11.10-r1.

rkubik-hostersi commented 1 month ago

@tim-finnigan I just don't understand this is being marked as feature request. IMO it's a bug as it does not allow to use EKS Pod Identity feature fully with aws-cli tool. The documentation says that Pod Identities are supported in various SDK versions, and AWS CLI, but they are not (fully).

rkubik-hostersi commented 1 month ago

The https://github.com/aws/aws-cli/issues/3875 is not exactly about the same behavior, it's more generic case.

gamma425 commented 6 days ago

100% agree with @rkubik-hostersi that this is not a feature request. It is a bug. Please label it accordingly and please prioritize it.

tim-finnigan commented 5 days ago

Checking in again — can you specify which documentation is not accurate? Here is the EKS User Guide on Pod Identities: https://docs.aws.amazon.com/eks/latest/userguide/pod-id-how-it-works.html , and the AWS CLI documentation on authentication and access credentials: https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-authentication.html

aws / aws-cli