kedacore / keda

KEDA is a Kubernetes-based Event Driven Autoscaling component. It provides event driven scale for any container running in Kubernetes
https://keda.sh
Apache License 2.0
8.59k stars 1.08k forks source link

AWS credentials cache key needs to include the region #6128

Open maxbog opened 3 months ago

maxbog commented 3 months ago

Report

Hello, I have two deployments using ScaledObjects based on SQS queues in different regions (say, eu-central-1 and us-east-1) and I want to authenticate to AWS using pod identity. The first ScaledObject authenticates correctly (one running on eu-central-1), and then the AWS config (with region included) is cached in the config cache. The second ScaledObject fails to start, because the operator tries to connect to a queue in another region (us-east-1), but the cached config includes the region from the first queue (eu-central-1). If I understand the code correctly, the getCacheKey function here: https://github.com/kedacore/keda/blob/85d4dca17f9e2e58bdc91f046e6dbe8e6235e78f/pkg/scalers/aws/aws_config_cache.go#L71 needs to include region in the returned string so that the configs are cached per region.

Expected Behavior

Both ScaledObjects report as Ready

Actual Behavior

Only the first ScaledObject is ready, the second one never authenticates successfully.

Steps to Reproduce the Problem

1.Create two queues in different region 2.Create ScaledObjects for them using pod identity as auth mechanism

Logs from KEDA operator

{"level":"error","ts":"2024-09-03T14:22:34Z","logger":"scale_handler","msg":"error getting metric for trigger","scaledObject.Namespace":"**REDACTED**","scaledObject.Name":"**REDACTED**","trigger":"awsSqsQueueScaler","error":"operation error SQS: GetQueueAttributes, https response error StatusCode: 400, RequestID: d511bcd7-6b16-5a65-bb0d-e38676fca9a3, AWS.SimpleQueueService.NonExistentQueue: The specified queue does not exist or you do not have access to it.","stacktrace":"github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScaledObjectMetrics\n\t/workspace/pkg/scaling/scale_handler.go:553\ngithub.com/kedacore/keda/v2/pkg/metricsservice.(*GrpcServer).GetMetrics\n\t/workspace/pkg/metricsservice/server.go:48\ngithub.com/kedacore/keda/v2/pkg/metricsservice/api._MetricsService_GetMetrics_Handler.func1\n\t/workspace/pkg/metricsservice/api/metrics_grpc.pb.go:106\ngithub.com/kedacore/keda/v2/pkg/metricsservice.(*GrpcServer).Start.(*ServerMetrics).UnaryServerInterceptor.UnaryServerInterceptor.func6\n\t/workspace/vendor/github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/server.go:22\ngithub.com/kedacore/keda/v2/pkg/metricsservice/api._MetricsService_GetMetrics_Handler\n\t/workspace/pkg/metricsservice/api/metrics_grpc.pb.go:108\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/workspace/vendor/google.golang.org/grpc/server.go:1369\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/workspace/vendor/google.golang.org/grpc/server.go:1780\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/workspace/vendor/google.golang.org/grpc/server.go:1019"}

KEDA Version

2.15.0

Kubernetes Version

1.30

Platform

Amazon Web Services

Scaler Details

AWS SQS

Anything else?

No response

JorTurFer commented 3 months ago

Hello, Interesting and you're probably right. WDYT @ThaSami ?

ndlanier commented 2 months ago

I believe I am seeing this issue as well. Definitely paying attention to that PR.

maxbog commented 2 months ago

@JorTurFer any chance for a review and, hopefully, merge of the attached PR?

ndlanier commented 2 months ago

@JorTurFer bumping for review on the PR

stale[bot] commented 2 weeks ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

stale[bot] commented 1 week ago

This issue has been automatically closed due to inactivity.

ndlanier commented 9 hours ago

Can this be re-opened until the next release occurs?