search

kedacore / keda

KEDA is a Kubernetes-based Event Driven Autoscaling component. It provides event driven scale for any container running in Kubernetes
https://keda.sh
Apache License 2.0
8.56k stars 1.08k forks source link

AWS SecretsManager TriggerAuth Unable to get secret #6249

Open sreeboppana opened 1 month ago

sreeboppana commented 1 month ago

Report

Keda version: keda:2.14.0 ghcr.io/kedacore/keda:2.14.0 kubenetes version: 1.29

keda TriggerAuth is unable to read the Secrets from AWS secretsManager though provided the correct secretName & secret parameter.

Expected Behavior

Keda should have read the secrets from AWS secrets Manager.

Verified that the secret Name is valid and present with in the AWS region's SecretsManager matches to that of the one configured within TriggerAuth.yaml

The secret Name : service/test-svc/svc-api-key-path existed within the intended aws regions's SecretsManager.

the target AWS secrets manager secret (service/test-svc/svc-api-key-path) is of the form: { "api-key":"MY-SECRET-KEY", "api-secret":"MY-SECRET--VAL" }

           triggerauth.yaml
apiVersion: v1
items:
  kind: TriggerAuthentication
  metadata:
    **name: test-triggerauth**
    namespace: my-microservice-namespace
   spec:
    awsSecretManager:
      podIdentity:
        provider: aws
        roleArn: arn:aws:iam::<accountId>:test-role/<test-role-ID>
      region: us-east-1
      secrets:
      - name: service/test-svc/svc-api-key-path
        parameter: api-key
      - name: service/test-svc/svc-api-key-path
        parameter: api-secret

Actual Behavior

2024-10-18T14:39:47Z    ERROR   **scale_handler error resolving auth params**   
   {
         "type": "ScaledObject", 
           "namespace": "my-microservice-namespace", 
            "name": "my-microservice", 
            "triggerIndex": 3, 
             "error": "error creating kafka client: kafka: client has run out of available brokers to talk to: read tcp 172.22.47.114:32914->100.66.23.62:9092: read: connection reset by peer"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).buildScalers
    /workspace/pkg/scaling/scalers_builder.go:99
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).performGetScalersCache
    /workspace/pkg/scaling/scale_handler.go:357
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScalersCache
    /workspace/pkg/scaling/scale_handler.go:282
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).getScaledObjectState
    /workspace/pkg/scaling/scale_handler.go:611
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
    /workspace/pkg/scaling/scale_handler.go:243
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
    /workspace/pkg/scaling/scale_handler.go:182

Steps to Reproduce the Problem

Goal: Use Keda triggerAuth AWs secretsManager to read secrets and be able to scale Kafka.

Keda version: keda:2.14.0 ghcr.io/kedacore/keda:2.14.0 kubenetes version: 1.29

  1. STEP-1 - triggerauth.yaml

    apiVersion: v1
items:
kind: TriggerAuthentication
metadata:
**name: test-triggerauth**
namespace: my-microservice-namespace
spec:
awsSecretManager:
  podIdentity:
    provider: aws
    roleArn: arn:aws:iam::<accountId>:test-role/<test-role-ID>
  region: us-east-1
  secrets:
  - name: service/test-svc/svc-api-key-path
    parameter: api-key
  - name: service/test-svc/svc-api-key-path
    parameter: api-secret
  2.         ScaledObject.yaml
    Name:         my-microservice
Namespace:    my-microservice-namespace
API Version:  keda.sh/v1alpha1
Kind:         ScaledObject
Spec:
Triggers:
Metadata:
  Value:      10
Metric Type:  Utilization
Type:         cpu
Metadata:
  Value:      10
Metric Type:  Utilization
Type:         memory
Authentication Ref:
  **Name:  test-triggerauth**
Metadata:
  Bootstrap Servers:  serverHostNames:9092
  Consumer Group:     xyz
  Lag Threshold:      30
  Topic:              event.xyz
Type:                 kafka

  3. the target AWS secrets manager secret (service/test-svc/svc-api-key-path) is of the form: { "api-key":"MY-SECRET-KEY", "api-secret":"MY-SECRET--VAL" }

    basically we are trying retrieve the api-key and api-secret values for that response above.

  4. Deployed the resources and then keda operator logs are shown below: as per the documentation https://keda.sh/docs/2.14/concepts/authentication/

The secrets list within awsSecretManager defines the mapping between the AWS Secret Manager secret and the authentication parameter used in your application, including the parameter name, AWS Secret Manager secret name, and an optional version parameter, defaulting to the latest version if unspecified.

2024-10-18T14:39:47Z    ERROR   **scale_handler error resolving auth params**   
   {
         "type": "ScaledObject", 
           "namespace": "my-microservice-namespace", 
            "name": "my-microservice", 
            "triggerIndex": 3, 
             "error": "error creating kafka client: kafka: client has run out of available brokers to talk to: read tcp 172.22.47.114:32914->100.66.23.62:9092: read: connection reset by peer"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).buildScalers
    /workspace/pkg/scaling/scalers_builder.go:99
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).performGetScalersCache
    /workspace/pkg/scaling/scale_handler.go:357
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScalersCache
    /workspace/pkg/scaling/scale_handler.go:282
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).getScaledObjectState
    /workspace/pkg/scaling/scale_handler.go:611
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
    /workspace/pkg/scaling/scale_handler.go:243
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
    /workspace/pkg/scaling/scale_handler.go:182

Logs from KEDA operator

2024-10-18T14:39:47Z    ERROR   **scale_handler error resolving auth params**   
   {
         "type": "ScaledObject", 
           "namespace": "my-microservice-namespace", 
            "name": "my-microservice", 
            "triggerIndex": 3, 
             "error": "error creating kafka client: kafka: client has run out of available brokers to talk to: read tcp 172.22.47.114:32914->100.66.23.62:9092: read: connection reset by peer"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).buildScalers
    /workspace/pkg/scaling/scalers_builder.go:99
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).performGetScalersCache
    /workspace/pkg/scaling/scale_handler.go:357
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScalersCache
    /workspace/pkg/scaling/scale_handler.go:282
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).getScaledObjectState
    /workspace/pkg/scaling/scale_handler.go:611
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
    /workspace/pkg/scaling/scale_handler.go:243
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
    /workspace/pkg/scaling/scale_handler.go:182

2024-10-18T14:39:47Z    ERROR   scale_handler   error getting state of scaledObject 
{
   "scaledObject.Namespace": "my-microservice-namespace", 
     "scaledObject.Name": "my-microservice", 
      "error": "error getting scalers cache error creating kafka client: kafka: client has run out of available brokers to talk to: read tcp 172.22.47.114:32914->100.66.23.62:9092: read: connection reset by peer"}

KEDA Version

2.14.0

Kubernetes Version

1.29

Platform

Amazon Web Services

Scaler Details

kafka

Anything else?

Verified that the secret Name is valid and present with in the AWS region's SecretsManager matches to that of the one configured within TriggerAuth.yaml The secret Name : service/test-svc/svc-api-key-path existed within the intended aws regions's SecretsManager.

           triggerauth.yaml
apiVersion: v1
items:
  kind: TriggerAuthentication
  metadata:
    **name: test-triggerauth**
    namespace: my-microservice-namespace
   spec:
    awsSecretManager:
      podIdentity:
        provider: aws
        roleArn: arn:aws:iam::<accountId>:test-role/<test-role-ID>
      region: us-east-1
      secrets:
      - name: service/test-svc/svc-api-key-path
        parameter: api-key
      - name: service/test-svc/svc-api-key-path
        parameter: api-secret
JorTurFer commented 3 weeks ago

Hello Could you verify if KEDA has the required environment variables added by the eks mutating webhook? Please, share the output of kubectl describe pod KEDA_OPERATOR_POD -n KEDA_NAMESPACE removing sensitive data

sreeboppana commented 2 weeks ago

thanks @JorTurFer please refer to the output of kubectl describe po keda-operator-5b48dfb687-b7xcx -n keda

$ kubectl describe po keda-operator-5b48dfb687-b7xcx -n keda

Name:         keda-operator-5b48dfb687-b7xcx
Namespace:    keda
Priority:     0
Node:         ip-100-66-9-70.ec2.internal/100.66.9.70
Start Time:   Tue, 05 Nov 2024 07:40:10 -0500
Labels:       app=keda-operator
              app.kubernetes.io/component=operator
              app.kubernetes.io/instance=keda-operator
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=keda-operator
              app.kubernetes.io/part-of=keda-operator
              app.kubernetes.io/version=2.14.0
              helm.sh/chart=keda-2.14.2
              name=keda-operator
              pod-template-hash=5b48dfb687
Annotations:  kubectl.kubernetes.io/restartedAt: 2024-10-31T09:06:59+01:00
Status:       Running
IP:           172.22.199.48
IPs:
  IP:           172.22.199.48
Controlled By:  ReplicaSet/keda-operator-5b48dfb687
Containers:
  keda-operator:
    Container ID:  containerd://94334ff68a5dab55f2c32a1f10a54230eaa2139ff2654fa20c858cf5b31c5dc8
    Image:         ghcr.io/kedacore/keda:2.14.0
    Image ID:      ghcr.io/kedacore/keda@sha256:c74847dd7c2a62d6a3ad0631208dab53264cec4eb0fc5ac3f41d4852ee363bf6
    Ports:         8080/TCP, 9666/TCP
    Host Ports:    0/TCP, 0/TCP
    Command:
      /keda
    Args:
      --leader-elect
      --disable-compression=true
      --zap-log-level=info
      --zap-encoder=console
      --zap-time-encoding=rfc3339
      --cert-dir=/certs
      --enable-cert-rotation=true
      --cert-secret-name=kedaorg-certs
      --operator-service-name=keda-operator
      --metrics-server-service-name=keda-operator-metrics-apiserver
      --webhooks-service-name=keda-admission-webhooks
      --k8s-cluster-name=kubernetes-default
      --k8s-cluster-domain=cluster.local
      --enable-prometheus-metrics=true
      --metrics-bind-address=:8080
    State:          Running
      Started:      Tue, 05 Nov 2024 07:40:13 -0500
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     2
      memory:  8Gi
    Requests:
      cpu:      1
      memory:   8Gi
    Liveness:   http-get http://:8081/healthz delay=25s timeout=1s period=10s #success=1 #failure=3
    Readiness:  http-get http://:8081/readyz delay=20s timeout=1s period=3s #success=1 #failure=3
    Environment:
      WATCH_NAMESPACE:              
      POD_NAME:                     keda-operator-5b48dfb687-b7xcx (v1:metadata.name)
      POD_NAMESPACE:                keda (v1:metadata.namespace)
      OPERATOR_NAME:                keda-operator
      KEDA_HTTP_DEFAULT_TIMEOUT:    3000
      KEDA_HTTP_MIN_TLS_VERSION:    TLS12
      AWS_STS_REGIONAL_ENDPOINTS:   regional
      AWS_DEFAULT_REGION:           us-east-1
      AWS_REGION:                   us-east-1
      AWS_ROLE_ARN:                 arn:aws:iam::REDACTED:role/keda-operator-iam-role-us-east-1
      AWS_WEB_IDENTITY_TOKEN_FILE:  /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    Mounts:
      /certs from certificates (ro)
      /mnt/kafka-secret from kafka-secret (ro)
      /var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-m9vcm (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       True 
  ContainersReady             True 
  PodScheduled                True 
Volumes:
  aws-iam-token:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  86400
  certificates:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  kedaorg-certs
    Optional:    true
  kafka-secret:
    Type:              CSI (a Container Storage Interface (CSI) volume source)
    Driver:            secrets-store.csi.k8s.io
    FSType:            
    ReadOnly:          true
    VolumeAttributes:      secretProviderClass=keda-kafka-secretproviderclass
  kube-api-access-m9vcm:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:                      <none>
sreeboppana commented 1 week ago

@JorTurFer please verify the output of the describe pod output above and advise. thanks

sreeboppana commented 1 week ago

i just gone through this https://keda.sh/docs/2.14/operate/cluster/ are you recommending adding the below env to keda-operator (keda namespace) ?

env:
  - name: KEDA_RESTRICT_SECRET_ACCESS
    value: "true"
JorTurFer commented 1 day ago

Do you see any connection error on your Kafka that brings any extra info? The envs vars look correct

JorTurFer commented 1 day ago

i just gone through this keda.sh/docs/2.14/operate/cluster are you recommending adding the below env to keda-operator (keda namespace) ?

env:
  - name: KEDA_RESTRICT_SECRET_ACCESS
    value: "true"

It depends on your scenario, limiting the envs can break your SO if they read secrets from the cluster