STS support by reading an AWS config in a secret

gallettilance commented 1 year ago

Is your feature request related to a problem?

Background

Operators (especially on OpenShift) that need to authenticate with the cloud provider, have started using the following flow to support authenticating using STS:

User applies a secret in a location specified by the operator with:

apiVersion: v1
kind: Secret
metadata: 
  namespace: <target-namespace> 
  name: <target-secret-name>
stringData:
  config: |- 
     [default]
     sts_regional_endpoints = regional
     role_name:<role ARN >
     web_identity_token_file: <path-to-token-projected-into-the-pod>

Operator is installed and can read the config from the secret (either by reading it directly from the apiserver or by having the secret mounted at a specified location) and can assume-role-with-web-identity and receive STS creds.

The issue

This flow does not seem to work with ACK controllers. Following the docs, here is how I tried to achieve this:

Create the above secret
Mount the secret to the pod at /some/location
Set the AWS_CONFIG_FILE env var to /some/location (I also tried setting the AWS_SHARED_CREDENTIALS_FILE to that same location)

Describe the solution you'd like

Option 1

Setting either the AWS_CONFIG_FILE or the AWS_SHARED_CREDENTIALS_FILE to the mount path location that contains the above info should be sufficient for the ACK controller to assume the role and receive STS creds.

Option 2

It is possible for the ACK controller to read the secret directly from the apiserver.

Some more info

I configured things properly on my cluster and have an OIDC provider and role created with the right permissions for the ack controller I'm trying to install (specifically was using the s3 one). I know things are set up right because I used the pod identity webhook approach by annotating the service account and that worked seamlessly. I also verified that the secret is properly mounted to the pod and the file is readable, contains the correct info, and the web identity token is valid.

Here is the CSV I used to install the controller:

apiVersion: operators.coreos.com/v1alpha1
kind: ClusterServiceVersion
metadata:
  annotations:
    alm-examples: |-
      [
        {
          "apiVersion": "s3.services.k8s.aws/v1alpha1",
          "kind": "Bucket",
          "metadata": {
            "name": "example"
          },
          "spec": {}
        }
      ]
    capabilities: Basic Install
    categories: Cloud Provider
    certified: "false"
    containerImage: public.ecr.aws/aws-controllers-k8s/s3-controller:v1.0.3
    createdAt: "2023-03-22 22:21:47"
    description: AWS S3 controller is a service controller for managing S3 resources
      in Kubernetes
    operatorframework.io/suggested-namespace: ack-system
    operators.operatorframework.io/builder: operator-sdk-v1.19.0+git
    operators.operatorframework.io/project_layout: unknown
    repository: https://github.com/aws-controllers-k8s
    support: Community
    cloudTokenEnabled: "true"
  name: ack-s3-controller.v1.0.3
  namespace: placeholder
spec:
  apiservicedefinitions: {}
  customresourcedefinitions:
    owned:
    - description: Bucket represents the state of an AWS s3 Bucket resource.
      displayName: Bucket
      kind: Bucket
      name: buckets.s3.services.k8s.aws
      version: v1alpha1
  description: |-
    Manage Amazon Simple Storage Service (S3) resources in AWS from within your Kubernetes cluster.
    **About Amazon S3**
    Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance. This means customers of all sizes and industries can use it to store and protect any amount of data for a range of use cases, such as data lakes, websites, mobile applications, backup and restore, archive, enterprise applications, IoT devices, and big data analytics. Amazon S3 provides easy-to-use management features so you can organize your data and configure finely-tuned access controls to meet your specific business, organizational, and compliance requirements. Amazon S3 is designed for 99.999999999% (11 9s) of durability, and stores data for millions of applications for companies all around the world.
    **About the AWS Controllers for Kubernetes**
    This controller is a component of the [AWS Controller for Kubernetes](https://github.com/aws/aws-controllers-k8s) project.
    **Pre-Installation Steps**
    Please follow the following link: [Red Hat OpenShift](https://aws-controllers-k8s.github.io/community/docs/user-docs/openshift/)
  displayName: AWS Controllers for Kubernetes - Amazon S3
  install:
    spec:
      clusterPermissions:
      - rules:
        - apiGroups:
          - ""
          resources:
          - configmaps
          verbs:
          - get
          - list
          - patch
          - watch
        - apiGroups:
          - ""
          resources:
          - namespaces
          verbs:
          - get
          - list
          - watch
        - apiGroups:
          - ""
          resources:
          - secrets
          verbs:
          - get
          - list
          - patch
          - watch
        - apiGroups:
          - ""
          resources:
          - serviceaccounts
          verbs:
          - get
          - patch
          - update
        - apiGroups:
          - s3.services.k8s.aws
          resources:
          - buckets
          verbs:
          - create
          - delete
          - get
          - list
          - patch
          - update
          - watch
        - apiGroups:
          - s3.services.k8s.aws
          resources:
          - buckets/status
          verbs:
          - get
          - patch
          - update
        - apiGroups:
          - services.k8s.aws
          resources:
          - adoptedresources
          verbs:
          - create
          - delete
          - get
          - list
          - patch
          - update
          - watch
        - apiGroups:
          - services.k8s.aws
          resources:
          - adoptedresources/status
          verbs:
          - get
          - patch
          - update
        - apiGroups:
          - services.k8s.aws
          resources:
          - fieldexports
          verbs:
          - create
          - delete
          - get
          - list
          - patch
          - update
          - watch
        - apiGroups:
          - services.k8s.aws
          resources:
          - fieldexports/status
          verbs:
          - get
          - patch
          - update
        - apiGroups:
          - "cloudcredential.openshift.io"
          resources:
          - credentialsrequests
          verbs:
          - create
          - delete
          - get
          - list
          - patch
          - update
          - watch
        serviceAccountName: ack-s3-controller
      deployments:
      - label:
          app.kubernetes.io/name: ack-s3-controller
          app.kubernetes.io/part-of: ack-system
        name: ack-s3-controller
        spec:
          replicas: 1
          selector:
            matchLabels:
              app.kubernetes.io/name: ack-s3-controller
          strategy: {}
          template:
            metadata:
              labels:
                app.kubernetes.io/name: ack-s3-controller
            spec:
              securityContext:
                runAsNonRoot: true
                seccompProfile:
                  type: RuntimeDefault
              containers:
              - image: quay.io/lgallett/s3:latest
                env:
                - name: AWS_REGION
                  value: us-east-1
                - name: AWS_SHARED_CREDENTIALS_FILE
                  value: /var/config
                - name: AWS_SDK_LOAD_CONFIG
                  value: "1"
                volumeMounts:
                - name: bound-sa-token
                  mountPath: /var/run/secrets/openshift/serviceaccount
                  readOnly: true
                - name: s3-controller
                  mountPath: /var
                  readOnly: true
                name: controller
                ports:
                - containerPort: 8080
                  name: http
                resources:
                  limits:
                    cpu: 100m
                    memory: 300Mi
                  requests:
                    cpu: 100m
                    memory: 200Mi
                securityContext:
                  allowPrivilegeEscalation: false
                  capabilities:
                    drop:
                    - ALL
                  privileged: false
                  runAsNonRoot: true
              serviceAccountName: ack-s3-controller
              terminationGracePeriodSeconds: 10
              volumes:
              # This service account token can be used to provide identity outside the cluster.
              # For example, this token can be used with AssumeRoleWithWebIdentity to authenticate with AWS using IAM OIDC provider and STS.
              - name: bound-sa-token
                projected:
                  sources:
                  - serviceAccountToken:
                      path: token
                      audience: openshift
              - name: s3-controller
                secret:
                  secretName: s3-controller
                  items:
                  - key: config
                    path: config
    strategy: deployment
  installModes:
  - supported: true
    type: AllNamespaces
  keywords:
  - s3
  - aws
  - amazon
  - ack
  links:
  - name: AWS Controllers for Kubernetes
    url: https://github.com/aws-controllers-k8s/community
  - name: Documentation
    url: https://aws-controllers-k8s.github.io/community/
  - name: Amazon S3 Developer Resources
    url: https://aws.amazon.com/s3/developer-resources/
  maintainers:
  - email: ack-maintainers@amazon.com
    name: s3 maintainer team
  maturity: alpha
  provider:
    name: Amazon, Inc.
    url: https://aws.amazon.com
  version: 1.0.3

quay.io/lgallett/s3:latest is built from the s3-controller repo.

acornett21 commented 1 year ago

@gallettilance A couple of questions:

What other operator(s) use this model today in OpenShift?
You mention a Secret, but in the example you are reading env variables, it's not clear which approach you are asking for, can you clarify?

Since this would potentially need changes in the helm flow and the bundle/olm generation flow (and corresponding runtime changes) , to add another authentication/authorization mechanism, I think this change needs larger discussion with the core maintainers.

gallettilance commented 1 year ago

OpenShift comes with the Cloud Credentials Operator (CCO). It provides a cloud agnostic experience for authenticating with cloud providers by adding a secret in the desired location with the credentials or config to get credentials for the operator to consume. Most if not all operators on OpenShift rely on CCO and hence rely on the secret created by it.

In the above example I attempted to set the env var to be the path to the config (at the mount path of the secret - /var/config in the above example. My hope was that this would be enough for the controller to assume the role with web identity and receive STS credentials but I kept running into the following error when I create a Bucket CR:

"error":"NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors"

gallettilance commented 1 year ago

STS_today

Here is a diagram for how I would like this to work

acornett21 commented 1 year ago

@gallettilance I'm familiar with CCO, I don't think most operators use CCO. Unless you are talking about the cluster operators that already come with OpenShift, then yes those are going to be backed by CCO. To my knowledge, no certified or community operator uses this approach, but it could be one that I am unaware of.

With that said, ACK supports IRSA in OpenShift (Technically ROSA, since that's the only place OIDC is fully supported to my knowledge). Why isn't the IRSA approach sufficient?

I'm still not clear if you want this as an env or as a secret reference in the deployment. In OpenShift the CSV contains both env references and secret references, which approach are you suggesting here? (sample spec below)

            spec:
              containers:
              - args:
                - --aws-region
                - $(AWS_REGION)
                - --aws-endpoint-url
                - $(AWS_ENDPOINT_URL)
                - --enable-development-logging
                - $(ACK_ENABLE_DEVELOPMENT_LOGGING)
                - --log-level
                - $(ACK_LOG_LEVEL)
                - --resource-tags
                - $(ACK_RESOURCE_TAGS)
                - --watch-namespace
                - $(ACK_WATCH_NAMESPACE)
                command:
                - ./bin/controller
                env:
                - name: ACK_SYSTEM_NAMESPACE
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.namespace
                envFrom:
                - configMapRef:
                    name: ack-s3-user-config
                    optional: false
                - secretRef:
                    name: ack-s3-user-secrets
                    optional: true

gallettilance commented 1 year ago

IRSA is insufficient because it depends on the pod identity webhook which only supports AWS. CCO aims to support all cloud providers.

What does:

             envFrom:
                - configMapRef:
                    name: ack-s3-user-config
                    optional: false
                - secretRef:
                    name: ack-s3-user-secrets
                    optional: true

do and what do each of these configmap and secrets look like? (where can I read more about this?) Because maybe I just need to add the config from the secret to a configMap for the pod to consume.

gallettilance commented 1 year ago

I just want the ACK controller to use the config specified in the secret. This can be by including the secret ref in an env var or any other way (I'm open to suggestions). I just thought setting the AWS_CONFIG_FILE to the config path would be sufficient but it's not - and I'm not sure I understand why.

acornett21 commented 1 year ago

@gallettilance Here are the docs on how to install on OpenShift. The existence of these is so that a user can provide configuration: ConfigMap for how to run the controller and Secret containing auth information so the controller can authenticate with AWS, if using an access key, that's why this is optional.

These are envFrom since a cluster admin cannot modify a Deployment after it has been installed by OLM, since OLM will reconcile the Deployment and the changes they made will be lost. I hope this is clearer on what is going on.

gallettilance commented 1 year ago

Sure - but in this case I am creating / customizing the CSV so I have full control of the deployment. All I want is to give the controller the above config so that it can assume the role with web identity and get STS creds. Are you saying if I just put the contents of the above secret in a configmap, the ack controller will be able to use STS creds? If not where is this config suppose to go?

Or are you saying that the ack controller doesn't support this approach - the only way to get sts creds is to annotate the service account with the role ARN and have the pod identity webhook project the bound sa token into the pod?

gallettilance commented 1 year ago

Or can I use the above secret in lieu of one that is supposed to contain credentials information?

(sorry for all the questions I'm just having a hard time understanding)

wdonne commented 1 year ago

I also use this model in my AWS Assume Role Operator and it works with the AWS Load Balancer Controller, which is not part of ACK. However, the next problem I face is that when the token is refreshed in the secret, it is not reloaded, not by the Load Balancer Controller anyway. This should also be checked with the ACK controllers, otherwise the model won't work.

ack-bot commented 9 months ago

Issues go stale after 180d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 60d of inactivity and eventually close. If this issue is safe to close now please do so with /close. Provide feedback via https://github.com/aws-controllers-k8s/community. /lifecycle stale

ack-bot commented 3 months ago

Issues go stale after 180d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 60d of inactivity and eventually close. If this issue is safe to close now please do so with /close. Provide feedback via https://github.com/aws-controllers-k8s/community. /lifecycle stale

aws-controllers-k8s / community