search

awspring / spring-cloud-aws

The New Home for Spring Cloud AWS
http://awspring.io
Apache License 2.0
869 stars 300 forks source link

Enable SqsTemplate to send more than 10 messages at once #1042

Open tomazfernandes opened 7 months ago

tomazfernandes commented 7 months ago

As discussed here, currently SqsTemplate fails if more than 10 messages are provided in a batch with an error from AWS SDK.

We can add support for this use case by partitioning the message list in batches of 10, while also sending the batches in parallel.

This change would likely be around here:

https://github.com/awspring/spring-cloud-aws/blob/562b9334821ea99d7eeac886e60dd59ef73451b3/spring-cloud-aws-sqs/src/main/java/io/awspring/cloud/sqs/operations/SqsTemplate.java#L350-L356

levys17 commented 7 months ago

@tomazfernandes I started to work on it...If it's ok, can you assign this issue to me?

techadri commented 3 months ago

@tomazfernandes I'd be happy to provide a PR for this topic, in case you (@levys17) aren't working on it anymore.

levys17 commented 3 months ago

@tomazfernandes I'd be happy to provide a PR for this topic, in case you (@levys17) aren't working on it anymore.

I didn't have time to finish that and I've barely worked on it in last weeks....so for me it'll be nice if you do that :D

techadri commented 3 months ago

@tomazfernandes Currently, doSendBatchAsync sends all messages in a single batch request using the sqsAsyncClient, ensuring that FIFO queues maintain the order of messages. If we change to partitioning messages and sending batches in parallel, only messages within each batch will retain order, but the order of the batches may vary. What are your thoughts on this?

tomazfernandes commented 3 months ago

Oh, that's a great point.

I think the ideal solution would be partitioning batches per message group for fifo queues and then sending each batch in parallel.

A simpler solution would be sending batches in parallel for Standard queues and sequentially for FIFO queues.

WDYT?