iguyking commented 2 years ago

Description

Attempting to utilize AWS Powershell Netcore tooling with EKS using IRSA credentials which supply the AWS_WEB_IDENTITY_TOKEN_FILE environment variable and file for getting credentials for a kubernetes pod. When making calls such as Use-STSCallerIdentity, the AWS cmdlets do not find the Web Identity Token information. Instead it is defaulting to the node's instance profile.

Reproduction Steps

Create a basic Ubuntu Docker Image with .NET 5 (latest) and Powershell Core downloaded.
Install AWSPowershell.NetCore version 4.1.15.0. Setup EKS with the recommended IRSA configuration and assign this role to the deployment pod in question Run Get-STSCallerIdentity

This will return the node's IAM role, not the Pods.

AWS CLI will return correctly:

{
    "UserId": "AROAVHFETX5TATZNVXBVJ:botocore-session-1637098166",
    "Account": "123456789",
    "Arn": "arn:aws:sts::123456789:assumed-role/runner-service-account/botocore-session-1637098166"
}

Powershell call will return

{
  "Account": "123456789",
  "Arn": "arn:aws:sts::123456789:assumed-role/platform-2021102117024492730000000e/i-0c37ddbaf651488a1",
  "UserId": "AROAVHFETX5TH76JTXAAB:i-0c37ddbaf651488a1",
  "ResponseMetadata": {
    "RequestId": "b445fa1c-342a-477a-a2fb-c23b0e9a53d5",
    "Metadata": {}
  },
  "ContentLength": 493,
  "HttpStatusCode": 200,
  "LoggedAt": "2021-11-16T21:39:23.3531025+00:00"
}

Environment

Build Version: 4.1.15.0
OS Info: Ubuntu 20.04
Build Environment: Download from nuget registry
Targeted .NET Platform: 5

Resolution

[ ] :wave: I can/would-like-to implement a fix for this problem myself

This is a :bug: bug-report

ashishdhingra commented 2 years ago

Hi @iguyking,

Good morning.

Could you please confirm the following:

Value of environment variable AWS_ROLE_ARN.
Value of environment variable AWS_WEB_IDENTITY_TOKEN_FILE.
EKS version

Some issues are resolved after cluster upgrade as was the case in https://github.com/aws/aws-sdk-net/issues/1615.

Thanks, Ashish

iguyking commented 2 years ago

EKS version is 1.20

AWS_ROLE_ARN & AWS_WEB_IDENTITY_TOKEN_FILE are set and work properly for the python based aws cli commands.

I ran the aws cli & powershell commands in the same OS/same docker container session.

That ticket is around Windows based OSes. This is from an ubuntu 20.04 image.

iguyking commented 2 years ago

What can I help give you to see this work or not work?

iguyking commented 2 years ago

Update:

EKS version 1.21.5-20211117 Powershell 7.2.0 Using Ubuntu 20.04.3 LTS.

Still not working where the aws python works.

PS /> import-module AWSPowershell.NetCore
PS /> gci env:A*

Name                           Value
----                           -----
AWS_ROLE_ARN                   arn:aws:iam::123423341127:role/powershell-test-role
AWS_WEB_IDENTITY_TOKEN_FILE    /var/run/secrets/eks.amazonaws.com/serviceaccount/token
AWS_DEFAULT_REGION             us-west-2
AWS_REGION                     us-west-2

PS /> Get-STSCallerIdentity | select *

LoggedAt         : 11/23/2021 22:54:09
Account          : 123423341127
Arn              : arn:aws:sts::123423341127:assumed-role/team2021111722262155740000000e/i-00e0324c06a60e198
UserId           : AXOAQWCVGYJDSCMQZGHOP:i-00e0324c06a60e198
ResponseMetadata : Amazon.Runtime.ResponseMetadata
ContentLength    : 486
HttpStatusCode   : OK

PS /> aws sts get-caller-identity
{
    "UserId": "AXOAQWCVGYJDTJRFCNTOS:botocore-session-1637708010",
    "Account": "123423341127",
    "Arn": "arn:aws:sts::123423341127:assumed-role/powershell-test-role/botocore-session-1637708010"
}

PS /> $PSVersionTable

Name                           Value
----                           -----
PSVersion                      7.2.0
PSEdition                      Core
GitCommitId                    7.2.0
OS                             Linux 5.4.156-83.273.amzn2.x86_64 #1 SMP Sat Oct 30 12:59:07 UTC 2021
Platform                       Unix
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

PS /> cat /etc/issue
Ubuntu 20.04.3 LTS \n \l

Test info:

Create a new role called powershell-test-role in the account & update the trust to match your EKS cluster for IRSA powershell-test-role-trust.json.txt

Setup a standard EKS Cluster Use this set of YAML (update the role ARN in the ServiceAccount powershell-test.yaml.txt

Connect in over kubectl exec Run

apt-get update
apt-get -y install ca-certificates unzip curl git gnupg apt-transport-https pkg-config
curl https://packages.microsoft.com/config/ubuntu/20.04/packages-microsoft-prod.deb --output packages.deb
dpkg -i ./packages.db
 apt-get update
 apt-get install powershell
 pwsh -c Install-Module -Name AWSPowerShell.NetCore -Repository PSGallery -Force -ErrorAction Stop
 pwsh
 import-module AWSPowershell.NetCore
 Get-STSCallerIdentity

ashishdhingra commented 2 years ago

Reproducible with output of Get-STSCallerIdentity different from aws sts get-caller-identity.

STEPS:

Create cluster with Ubuntu 20.04 LTS and with IAM OIDC associated.
```
eksctl create cluster --name ubuntu-powershell-test --version 1.21 --node-ami-family=Ubuntu2004 --with-oidc
```
Namespace is kube-system. Use this in subsequent commands and deployment templates.

Create IAM Service Account:

eksctl create iamserviceaccount --name my-serviceaccount --namespace kube-system --cluster ubuntu-powershell-test --attach-policy-arn arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess --approve

Use the below template and execute command kubectl apply -f ubuntu-powershell-test.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
name: ubuntu
labels:
app: ubuntu
namespace: kube-system
spec:
selector:
matchLabels:
  app.kubernetes.io/name: ubuntu
replicas: 1
template:
metadata:
  name: ubuntu
  labels:
    app.kubernetes.io/name: ubuntu
spec:
  serviceAccountName: "my-serviceaccount"
  nodeSelector:
    beta.kubernetes.io/arch: amd64
  containers:
  - name: ubuntu
    image: "ubuntu:20.04"
    imagePullPolicy: Always
    command: ["/bin/sh"]
    args: ["-c", "while true; do echo hello; sleep 10;done"]

Watch for pod creation in the namespace using command  kubectl get pods --namespace kube-system -o wide --watch
Once the Ubuntu pod is created, open interactive session on Ubuntu node using command kubectl exec --stdin --tty ubuntu-56c5bf5cf6-68hdp --namespace kube-system -- /bin/bash
On the connected Ubuntu pod,

Execute the below commands in connected pod to install PowerShell:

apt-get update
apt-get -y install ca-certificates unzip curl git gnupg apt-transport-https pkg-config
curl https://packages.microsoft.com/config/ubuntu/20.04/packages-microsoft-prod.deb --output packages.deb
dpkg -i ./packages.deb
apt-get update
apt-get install powershell
pwsh -c Install-Module -Name AWSPowerShell.NetCore -Repository PSGallery -Force -ErrorAction Stop

Execute the below commands in connected Ubuntu pod to install AWS CLI:

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
./aws/install

Examine the AWS environment variables using command env | grep AWS:

AWS_ROLE_ARN=arn:aws:iam::<<ACCOUNT-ID>>:role/eksctl-ubuntu-powershell-test-addon-iamservi-Role1-1ENWEJZZEEZNE
AWS_DEFAULT_REGION=us-east-2
AWS_REGION=us-east-2
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token

Execute aws sts get-caller-identity in the connected pod and examine the output:

{
"UserId": "AROASA6NRDFTRA3E5WHQ2:botocore-session-1643830450",
"Account": "139480602983",
"Arn": "arn:aws:sts::<<ACCOUNT-ID>>:assumed-role/eksctl-ubuntu-powershell-test-addon-iamservi-Role1-1ENWEJZZEEZNE/botocore-session-1643830450"
}

Execute the below commands in the connected pod to launch PowerShell session and get the result of Get-STSCallerIdentity:

pwsh
Import-Module AWSPowershell.NetCore
Get-STSCallerIdentity | select *

It produces the below output:

LoggedAt         : 02/02/2022 11:34:44
Account          : 139480602983
Arn              : arn:aws:sts::<<ACCOUNT-ID>>:assumed-role/eksctl-ubuntu-powershell-test-nod-NodeInstance
           Role-3H8K5EYL8LDE/i-0357f9a442d1eeba4
UserId           : AROASA6NRDFTWYT5EEQAO:i-0357f9a442d1eeba4
ResponseMetadata : Amazon.Runtime.ResponseMetadata
ContentLength    : 508
HttpStatusCode   : OK

Notice that output of aws sts get-caller-identity is pointing to the value specified by environment variable AWS_ROLE_ARN. Whereas the output of Get-STSCallerIdentity is pointing to node instance role.

abstrask commented 2 years ago

I encountered the same issue and managed to find a workaround:

Import-Module AWSPowershell.NetCore
$AWS_CREDS = Use-STSWebIdentityRole -RoleArn $env:AWS_ROLE_ARN -RoleSessionName $env:HOSTNAME -WebIdentityToken $(gc $env:AWS_WEB_IDENTITY_TOKEN_FILE) -Select 'Credentials'
Set-AWSCredential -Credential $AWS_CREDS
Get-STSCallerIdentity

Haven't refactored my workload yet, but I guess I would need to implement logic to ensure the creds are periodically updated.

mark-hubers commented 2 years ago

I encountered the same issue and managed to find a workaround:
Import-Module AWSPowershell.NetCore
$AWS_CREDS = Use-STSWebIdentityRole -RoleArn $env:AWS_ROLE_ARN -RoleSessionName $env:HOSTNAME -WebIdentityToken $(gc $env:AWS_WEB_IDENTITY_TOKEN_FILE) -Select 'Credentials'
Set-AWSCredential -Credential $AWS_CREDS
Get-STSCallerIdentity
Haven't refactored my workload yet, but I guess I would need to implement logic to ensure the creds are periodically updated.

Thanks so much, abstrask! This worked, but still weird why we have to do this.

I end up adding this to my scripts so I can still run them both in pods or on EC2 instances.

## Fix some weird problem when running inside a AWS EKS pod IAM it not pick up pod role base
if ( $env:AWS_ROLE_ARN -ne $null -and $env:AWS_WEB_IDENTITY_TOKEN_FILE -ne $null ) {
   ## We are running in AWS EKS using POD base IAM so we have to help get the pod role base working
   $AWS_CREDS = Use-STSWebIdentityRole -RoleArn $env:AWS_ROLE_ARN -RoleSessionName $env:HOSTNAME -WebIdentityToken $(gc $env:AWS_WEB_IDENTITY_TOKEN_FILE) -Select 'Credentials'
   Set-AWSCredential -Credential $AWS_CREDS
}

aws / aws-tools-for-powershell

Powershell NetCore does not work with AWS_WEB_IDENTITY_TOKEN_FILE #243

Description

Reproduction Steps

Environment

Resolution

Test info: