search

awspring / spring-cloud-aws

The New Home for Spring Cloud AWS
http://awspring.io
Apache License 2.0
885 stars 301 forks source link

SQS: Possibility to manage back pressure based on Rate Limiter #481

Open jeroenvandevelde opened 2 years ago

jeroenvandevelde commented 2 years ago

Type: Feature

Is your feature request related to a problem? Please describe. We want to be able to manage the max number of messages that are being polled from SQS so that we can protect external systems. Our application processes SQS messages which end up in HTTP calls to external systems. Those external systems have a maximum number of HTTP requests per second they can manage. So we would like a way to be certain we never poll more messages than that we are allowed to send out to those external systems.

Eg. External system A has a limit of 50 messages per second, then we would like to be able to tweak application B which consumes the SQS messages which result in HTTP calls to system A so that it at max polls 50 messages per second.

Describe the solution you'd like We would like a way to configure the SQS Listener to be certain that they don't poll more than a configurable x messages per a configurable x seconds. I am not certain how this solution would best look like. As i still need to think and investigate how this library currently handles the polling mechanism internally and how we can best fit it in.

@tomazfernandes and @maciejwalkowiak already gave some ideas on how this can be implemented in this discussion.

Describe alternatives you've considered We are currently using java-dynamic-sqs-listener for this purpose which we have configured in this manner.

Additional context https://github.com/awspring/spring-cloud-aws/discussions/479 https://github.com/JaidenAshmore/java-dynamic-sqs-listener/issues/372#issuecomment-927348168

sureshkmit commented 6 months ago

Thanks for the ask and the wonderful library. We are having a similar use case. Waiting for this to be implemented.