kedacore / keda

KEDA is a Kubernetes-based Event Driven Autoscaling component. It provides event driven scale for any container running in Kubernetes
https://keda.sh
Apache License 2.0
8.29k stars 1.05k forks source link

AWS credentials cache key needs to include the region #6128

Open maxbog opened 1 week ago

maxbog commented 1 week ago

Report

Hello, I have two deployments using ScaledObjects based on SQS queues in different regions (say, eu-central-1 and us-east-1) and I want to authenticate to AWS using pod identity. The first ScaledObject authenticates correctly (one running on eu-central-1), and then the AWS config (with region included) is cached in the config cache. The second ScaledObject fails to start, because the operator tries to connect to a queue in another region (us-east-1), but the cached config includes the region from the first queue (eu-central-1). If I understand the code correctly, the getCacheKey function here: https://github.com/kedacore/keda/blob/85d4dca17f9e2e58bdc91f046e6dbe8e6235e78f/pkg/scalers/aws/aws_config_cache.go#L71 needs to include region in the returned string so that the configs are cached per region.

Expected Behavior

Both ScaledObjects report as Ready

Actual Behavior

Only the first ScaledObject is ready, the second one never authenticates successfully.

Steps to Reproduce the Problem

1.Create two queues in different region 2.Create ScaledObjects for them using pod identity as auth mechanism

Logs from KEDA operator

{"level":"error","ts":"2024-09-03T14:22:34Z","logger":"scale_handler","msg":"error getting metric for trigger","scaledObject.Namespace":"**REDACTED**","scaledObject.Name":"**REDACTED**","trigger":"awsSqsQueueScaler","error":"operation error SQS: GetQueueAttributes, https response error StatusCode: 400, RequestID: d511bcd7-6b16-5a65-bb0d-e38676fca9a3, AWS.SimpleQueueService.NonExistentQueue: The specified queue does not exist or you do not have access to it.","stacktrace":"github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScaledObjectMetrics\n\t/workspace/pkg/scaling/scale_handler.go:553\ngithub.com/kedacore/keda/v2/pkg/metricsservice.(*GrpcServer).GetMetrics\n\t/workspace/pkg/metricsservice/server.go:48\ngithub.com/kedacore/keda/v2/pkg/metricsservice/api._MetricsService_GetMetrics_Handler.func1\n\t/workspace/pkg/metricsservice/api/metrics_grpc.pb.go:106\ngithub.com/kedacore/keda/v2/pkg/metricsservice.(*GrpcServer).Start.(*ServerMetrics).UnaryServerInterceptor.UnaryServerInterceptor.func6\n\t/workspace/vendor/github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/server.go:22\ngithub.com/kedacore/keda/v2/pkg/metricsservice/api._MetricsService_GetMetrics_Handler\n\t/workspace/pkg/metricsservice/api/metrics_grpc.pb.go:108\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/workspace/vendor/google.golang.org/grpc/server.go:1369\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/workspace/vendor/google.golang.org/grpc/server.go:1780\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/workspace/vendor/google.golang.org/grpc/server.go:1019"}

KEDA Version

2.15.0

Kubernetes Version

1.30

Platform

Amazon Web Services

Scaler Details

AWS SQS

Anything else?

No response

JorTurFer commented 1 week ago

Hello, Interesting and you're probably right. WDYT @ThaSami ?

ndlanier commented 2 days ago

I believe I am seeing this issue as well. Definitely paying attention to that PR.

maxbog commented 2 days ago

@JorTurFer any chance for a review and, hopefully, merge of the attached PR?