TriggerAuthentication : AWS Secret Manager is not working with awsSecretManager.podIdentity

Report

TriggerAuthentication AWS Secret Manager podIdentity is not working.

KEDA is not using ROLE provided in awsSecretManager.podIdentity.roleArn .
Even tried with awsSecretManager.identityOwner.workload ,but still KEDA is not using ROLE NAME provided in workload Service account annotation.

Even though awsSecretManager.podIdentity.roleArn is set, KEDA is still using IAM role which the KEDA operator uses

Expected Behavior

Expected Behavior is when ,

awsSecretManager.podIdentity.roleArn is set ,KEDA should use that role to make call to AWS secret manager to get secret value
OR when awsSecretManager.podIdentity.identityOwner is set to workload then KEDA should use role name used in service account associated with workload pod to make call to AWS secret manager to get secret value

Actual Behavior

Even though awsSecretManager.podIdentity.roleArn is set, KEDA is still using IAM role which the KEDA operator uses to get secret vale from AWS secret manager. Same behavior is observed even when we set awsSecretManager.podIdentity.identityOwner to workload

Steps to Reproduce the Problem

1.Create TriggerAuthentication with AWS Secret Manager as Authentication provider, sample below. The role in "roleArn" is in different AWS account and TriggerAuthentication is in different AWS EKS cluster

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: sample-test-auth
spec:
  awsSecretManager:
    podIdentity:                                                          
      provider: aws                                                       
      roleArn: arn:aws:iam::awsaccountid:role/TargetRoleName
    region: us-east-2                                                 
    secrets:                                                              
    - parameter: userName
      name: postgre-username
    - parameter: password
      name: postgre-password

2.Create ScaledObject, sample below

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: keda-postgre
spec:
  scaleTargetRef:
    name: app1
  triggers:
  -type: "postgresql"
    metadata:
      host: host
      port: port
      dbName: dbname
      sslmode: disable
      query: "query removed"
      targetQueryValue: "1"
    authenticationRef:
      name: sample-test-auth

3.Create sample deploy with name "app1" ,this is the target for scaledobject

Logs from KEDA operator

2024-06-18T10:30:53Z    ERROR   scale_handler   Error getting credentials       {"type": "ScaledObject", "namespace": "<removed>", "name": "keda-postgre", "error": "operation error Secrets Manager: GetSecretValue, https response error StatusCode: 400, RequestID: <removed>, api error AccessDeniedException: User: arn:aws:sts::<eks aws accountid removed>:assumed-role/keda-operator-role/<id removed> is not authorized to perform: secretsmanager:GetSecretValue on resource: postgre-username because no identity-based policy allows the secretsmanager:GetSecretValue action"}
github.com/kedacore/keda/v2/pkg/scaling/resolver.(*AwsSecretManagerHandler).Read
        /workspace/pkg/scaling/resolver/aws_secretmanager_handler.go:60
github.com/kedacore/keda/v2/pkg/scaling/resolver.resolveAuthRef
        /workspace/pkg/scaling/resolver/scale_resolvers.go:344
github.com/kedacore/keda/v2/pkg/scaling/resolver.ResolveAuthRefAndPodIdentity
        /workspace/pkg/scaling/resolver/scale_resolvers.go:183
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).buildScalers.func1
        /workspace/pkg/scaling/scalers_builder.go:72
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).buildScalers
        /workspace/pkg/scaling/scalers_builder.go:96
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).performGetScalersCache
        /workspace/pkg/scaling/scale_handler.go:357
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScalersCache
        /workspace/pkg/scaling/scale_handler.go:282
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).getScaledObjectMetricSpecs
        /workspace/controllers/keda/hpa.go:217
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).newHPAForScaledObject
        /workspace/controllers/keda/hpa.go:72
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).createAndDeployNewHPA
        /workspace/controllers/keda/hpa.go:45
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).ensureHPAForScaledObjectExists
        /workspace/controllers/keda/scaledobject_controller.go:441
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).reconcileScaledObject
        /workspace/controllers/keda/scaledobject_controller.go:280
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
        /workspace/controllers/keda/scaledobject_controller.go:191
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
        /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:119
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227

KEDA Version

2.13.0

Kubernetes Version

1.28

Platform

Amazon Web Services

Scaler Details

postgresql

Anything else?

Below 2 are running in one AWS EKS cluster Account

KEDA operator is running in its own namespace
TriggerAuthentication,ScaledObject and target Deployment is running in different namespace

AWS secret manger holding secrets is in different AWS account

I am trying cross account same region integration between KEDA and AWS secret manager

kedacore / keda