search

kedacore / keda

KEDA is a Kubernetes-based Event Driven Autoscaling component. It provides event driven scale for any container running in Kubernetes
https://keda.sh
Apache License 2.0
8.38k stars 1.06k forks source link

Scale SQS based on age of oldest message #5754

Closed furstenheim-goodnotes closed 2 months ago

furstenheim-goodnotes commented 5 months ago

Proposal

I would like to be able to scale sqs consumers based on age of oldest message, instead of length of the queue.

Use-Case

In my use case I enqueue tasks in batches as sqs messages. I don't care much about the processing time, meaning not immediate, as long as they are processed more or less on time. I definitely don't care about the length of the queue, and I would like not to have to estimate the necessary length of the queue.

If set my batches daily, I could trigger an upscale if oldest message hasn't been processed for 2 days.

Is this a feature you are interested in implementing yourself?

No

Anything else?

No response

JorTurFer commented 5 months ago

Hello, It sounds legit, does the SQS API support to query this information?

furstenheim-goodnotes commented 5 months ago

According to this issue: https://github.com/aws/aws-sdk-go/issues/2526 it seems to be available via cloudwatch api, not via sqs api. Would that be a blocker?

JorTurFer commented 5 months ago

The problem that I see here, is that the update frequency of a monitoring API isn't the same as the SQS API and it can produce misunderstandings. The required permissions are different too, as the roles won't be SQS related but CloudWatch related and it can be confusing...

KEDA already supports a CloudWatch scaler that can be used for calling directly to CloudWatch, isn't it enough for the use case? I'm not saying a strong NO to this feature request, I'm trying to understand the use case because it can make sense

stale[bot] commented 3 months ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

stale[bot] commented 2 months ago

This issue has been automatically closed due to inactivity.