Feature request: Add support for AWS EKS Iam Roles for ServiceAccounts

[lebenitza]

I already opened a ticket on Wall-E side: https://github.com/wal-e/wal-e/issues/425 I am curious if there are other things we would need to do on the operator side. The integration is just like with kube2iam or kiam but I think depends more on the client libraries each app uses.

Let me know how can I help. Thank you!

[Jan-M]

Can you do a bit more writing on how e.g. statefulset needs to change?

But I think this would need and outside PR if what is supported now is not enough. I mean it supports official AWS K8s annotations for IAM roles.

[lebenitza]

Not a thing needs to change on how we deploy the postgres clusters, is just about the applications that talk with AWS api, in this case S3, which need to support this. What happens:

1. The service account is annotated with which IAM role to use. This is supported by the current helm chart and operator by being allowed to specify your own ServiceAccounts for both the operator pod and Postgres pods.
2. A set of environment variables and a secret token are added to all pods using the annotated service account which can be used to make calls to s3.

Most of the official client libraries for AWS api already support detecting this new way of authenticating requests and using it without any change. Wall-e doesn't, it might need a boto3 upgrade, hence the opened ticket.

I am curious if there are other things maybe, on operator side or patroni, that talk with s3 and might need to support this. For example log_s3_bucket.

This will allow the operator and all the clusters it creates in EKS not to require and transfer around AWS credentials, it would be much cleaner and safer :smile:

[Jan-M]

So besides annotating the service account with the IAM role (vs current iam annotation on pod level) one also needs to spec the secrets in env variables?

[Jan-M]

Do you have an example of such a pod spec and service account? or link?

[lebenitza]

So besides annotating the service account with the IAM role (vs current iam annotation on pod level) one also needs to spec the secrets in env variables?

No :) The operator and the helm chart of the operator just need to allow custom annotations on the ServiceAccounts involved.

• custom values in the helm chart
• settings for the operator to add annotations to the ServiceAccounts of all the clusters it creates
• and maybe a setting in the Postgres CRD itself.

Currently this is not an issue, more of an improvement, because at the moment we can specify what ServiceAccounts to use (both operator and the clusters) that we created manually, already annotated.

What happens automatically in EKS when a POD uses a ServiceAccount annotated with:

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::AWS_ACCOUNT_ID:role/IAM_ROLE_NAME

is that a mutating webhook is injecting all the necessary volume (for the token) and environment variables to the POD itself, the operator does not need to do anything.

So what we only need to do, on operator side, is make sure that all the AWS SDKs we use or custom scripts are able to detect and use the injected token for AWS API requests. The whole mechanism behind IRSA will authenticate and authorize the request.

More about that here: https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html

I am sorry if I wasn't clear from the start.

[bkrugerp99]

Other gotchas you may run into:

(from external-dns that I had run into in the past)

      securityContext:
        fsGroup: 65534 # For ExternalDNS to be able to read Kubernetes and AWS token files

You may also need to set the AWS_SDK_LOAD_CONFIG env to true so it reads correctly. (Untested, but I've run into this with go and jenkins in the past) https://docs.aws.amazon.com/sdk-for-go/api/aws/session/

[bkrugerp99]

Looking at this more, granted, just installed the operator and sorry if I didn't RTFM and missed it when searching.

The service account created (postgres-pod) - looking at the complete-postgres-manifeset - is there a way I can append annotations to the service account? If I created databases in different namespaces, the static account name is fine, but I may wish to flip the annotation for eks-iam.

• kiam I know you add the annotation in your deploy/statefulset/pod/etc. -- eks-iam is a similar annotation is added to the service account instead and then service account used in your deploy/statefulset/pod/etc.

• wal-e vs wal-g -- likely need to use wal-g by default as wal-e is likely not going to support sts:AssumeRoleWithWebIdentity (This got added to aws sdk sometime in 2019 if I'm remembering correctly) - this is the secret sauce really for the eks-iam.

• the security fsgroup by default above, I'll play around with it on eks 1.18 to see if that's still relevant, but it ain't going to hurt. (mentioned previously)

• Golang by default I've run into in the past where AWS_SDK_LOAD_CONFIG needs to be set to true to pretty much follow the awscli/boto3 credential loading order. (mentioned previously)

PostgresPodRole - You'll need to add a trust policy for the eks cluster to do the AssumeRole with web identity. What isn't clear to me, is can we apply OperatorConfiguration per namespace? Or do we need to configmap it?

Here's what the steps look like as a horrible bash script with light error checking. (don't use this, use eksctl or something better):

script assumes you pre-created a role and have an existing policy (aws managed for example) that is unattached, but you'll get the idea.

## Janky service role creator
## example usage:
## ./service_role.sh EKS-CLUSTER-NAME external-dns-role default external-dns arn:aws:iam::YOUR_AWS_ACCOUNT_ID:policy/external-dns-k8s-policy
##                   ^- eks clustername    ^- role name  ^- namespace  ^- k8s account   ^- existing policy you have to limit aws resource access

CLUSTER_NAME=$1

if [ -z ${CLUSTER_NAME} ]; then
 echo "You need the eks cluster name"
 echo "Usage: service_role.sh <eks cluster name> <iam role name to use> <k8s-namespace> <service account name> <iam policy arn to apply>"
 exit 1
fi

ISSUER_URL=$(aws eks describe-cluster \
                       --name ${CLUSTER_NAME} \
                       --query cluster.identity.oidc.issuer \
                       --output text)

ISSUER_HOSTPATH=$(echo $ISSUER_URL | cut -f 3- -d'/')
ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
PROVIDER_ARN="arn:aws:iam::$ACCOUNT_ID:oidc-provider/$ISSUER_HOSTPATH"

ROLE_NAME=$2
NAMESPACE=$3
SERVICE_ACCOUNT_NAME=$4
POLICY_ARN=$5

if [ -z ${ROLE_NAME} ]; then
 echo "figure out a role name and pass that as second arg."
 exit 1
fi

if [ -z ${NAMESPACE} ]; then
 echo "name space required, argument 3."
 exit 1
fi;

if [ -z ${SERVICE_ACCOUNT_NAME} ]; then
 echo "k8s service account name is argument 4. you're missing it."
 exit 1
fi

if [ -z ${POLICY_ARN} ]; then
 echo "need iam policy you want to apply as arg 5.. way to forget it."
 exit 1
fi

cat > /tmp/${ROLE_NAME}-eks-trust-policy.json << EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "$PROVIDER_ARN"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "${ISSUER_HOSTPATH}:sub": "system:serviceaccount:${NAMESPACE}:${SERVICE_ACCOUNT_NAME}"
        }
      }
    }
  ]
}
EOF

aws iam create-role \
          --role-name $ROLE_NAME \
          --assume-role-policy-document file:///tmp/${ROLE_NAME}-eks-trust-policy.json

aws iam update-assume-role-policy \
          --role-name $ROLE_NAME \
          --policy-document file:///tmp/${ROLE_NAME}-eks-trust-policy.json

aws iam attach-role-policy \
          --role-name $ROLE_NAME \
          --policy-arn ${POLICY_ARN}

EKS_ROLE_ARN=$(aws iam get-role \
                        --role-name $ROLE_NAME \
                        --query Role.Arn --output text)

echo "If you see a kubectl error about account existing, that's fine, something else created it."
kubectl create sa -n ${NAMESPACE} ${SERVICE_ACCOUNT_NAME} || true
echo "We're going to annotate.  If this fails, we're not overwriting for right now."
kubectl annotate sa -n ${NAMESPACE} ${SERVICE_ACCOUNT_NAME} eks.amazonaws.com/role-arn=$EKS_ROLE_ARN

rm /tmp/${ROLE_NAME}-eks-trust-policy.json
echo "Now add service account to your deployment/pod/etc."
echo "here's a reference: https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/"

If someone knows the golang coding side of this and willing to write it, I am happy to help test/troubleshoot. I'm digging the operator so-far, but likely will have to do a little external mgmt at the moment to keep things tightened up and/or learn golang.

[james-callahan]

Is anyone working on this at the moment?

[michaeljguarino]

I've been trying to get the operator with eks IRSA-based IAM roles, and having issues as well. My guess is it's the AWS_SDK_LOAD_CONFIG env var (or deprecated golang sdk version) at fault, for whatever reason the go sdk does not handle credential fetching well. I'm not sure there's any means within the operator to add additional env vars to a provisioned pg pod though, my guess is the only recourse is to bake a new docker image with the env var manually set then?

[bkrugerp99]

Going to follow up on this. This could be partially mitigated if we can expand vars in part of the annotation creation for pod_service_account_definition.

  pod_service_account_definition: '{"apiVersion":"v1","kind":"ServiceAccount","metadata":{"annotations":{"eks.amazonaws.com/role-arn":"arn:aws:iam::AWS_ACCT:role/postgres-{cluster}-role"}}}'

Unfortunately, {cluster} doesn't expand out.

The docs do say that service accounts aren't monitored after, so one could add the annotations, but trying to keep this from having too many after-addons for provisioning. (Though if one is provisioning s3, and the IAM policies, probably don't matter)

@Jan-M @FxKu - I have zero experience with "go", but would it be possible to get some variable expansion for the pod s.a. definitions? I would figure that could probably close this ticket out in theory. I'm happy to beta test a build and write some docs against making it work.

[kahirokunn]

Is there any progress?

[martin31821]

We're running into the same problem, although IMO the operator side is perfectly fine. For better support it would be good to generate the eks.amazonaws.com/role-arn annotation on the ServiceAccounts, but the workaround suggested by @bkrugerp99 seems to work fine for now. (We're provisioning with terraform, so the {} cluster expansion isn't a problem for us)

However, going one step further, it seems that neither wal-e nor wal-g have support for using OIDC based auth flow, although wal-g got a merged PR for this: https://github.com/wal-g/wal-g/pull/1209, unfortunately it's not contained in any release yet.

[martin31821]

Got it working after a bit of fiddling.

First of all, when using IRSA / role-arn, I had to modify our trust relationship to include serviceaccounts across all kubernetes namespaces (since we're running and provisioning databases in several namespaces, more on that later).

Second, I built my own spilo image containing wal-g 2.0.0-rc1 (includes the PR I linked above), which works fine to archive the WAL segments to an S3 bucket.

Regarding the first point: It would be good to have support for the WAL archiving on a per-database region. I would like to provision a S3 bucket, IAM role, per database to follow good zero-trust policies. Currently, all database pods have read+write access to the entire WAL archive from all databases, which increases attack surface.

What do others think?

[Jasper-Ben]

:wave:, I am working with @martin31821.

We would understand that the maintainers would not want a RC build of wal-g in their upstream release spilo. I will therefore try to nudge the maintainers of wal-g towards a new stable release (it's been a while :wink:). In the meantime, we (that is to say @iris-GmbH) might be able to provide a working fork & container images for anyone interested in this feature. Will post updates on this into this thread.

[Kaelten]

Another +1 from another company that would prefer to not have to have IAM keys in play. :)

[Jasper-Ben]

@Kaelten Perfect timing. :wink:

We now have a freshly baked spilo image for amd64,arm64 hosted at ghcr.io/iris-gmbh/spilo:2daefaf-walg_v2.0.0-rc1. It is built from the current spilo master (commit 2daefaf) using wal-g v2.0.0-rc1 (I tried backporting this to the last two spilo releases, unfortunately there is some issue with building the container on these releases. Something, something a patch within the docker build does not apply, so master build will have to do for now. Reproducible builds would be nice @ zalando team :wink:).

Feel free to test this image as a drop-in replacement for the spilo image via the helm values. I believe it should just work (after applying step 1 from https://github.com/zalando/postgres-operator/issues/1124#issuecomment-1110788917), if not @martin31821 was working on this and should be able to provide you with any necessary details.

[davidspek]

I've noticed that the basebackups work with the latest spilo version, however writing the wal archives does not. It seems like https://github.com/zalando/spilo/pull/769 should probably fix this but I haven't tested this yet.

[davidspek]

With the new release of spilo available at registry.opensource.zalan.do/acid/spilo-14:2.1-p7 this issue can now be closed.

[ThomasK33]

With the new release of spilo available at registry.opensource.zalan.do/acid/spilo-14:2.1-p7 this issue can now be closed.

Not completely. The STS regional endpoints variable is not getting passed down to WAL-G, resulting in pods not being able to upload WAL logs if there is a dependency on regional endpoints (e.g., VPC Endpoints).