search

awspring / spring-cloud-aws

The New Home for Spring Cloud AWS
http://awspring.io
Apache License 2.0
867 stars 297 forks source link

Support for sending and receiving large message in SQS #858

Open vijayrcse opened 1 year ago

vijayrcse commented 1 year ago

Type: Feature

Is your feature request related to a problem? Please describe. We have a use case where our message size is larger than 254kb. is there a support in spring-cloud-sqs-messaging to send and receive large size messages using SQS extended client library

tomazfernandes commented 1 year ago

Hi @vijayrcse, thanks for bringing this up.

It should be possible to send larger messages by using S3 to store the payload, and sending only the reference in the message

There's a java-extended-client-lib that has this feature built in, as described here

Unfortunately it implements the blocking SqsClient, which is not supported by the framework. It might be possible to create an adapter or wrapper to use it with the SqsTemplate.

Let me know how it works for you.

Thanks.

NightFox7 commented 1 year ago

Hello ! I am also interested. I cannot find a way to make the extended client work the spring cloud aws sqs. Can anyone provide some tips please ? Thanks in advance;

tomazfernandes commented 1 year ago

Hi @NightFox7, if anyone is interested in creating a prototype of an adapter to use the client I linked above, or a separate implementation, I'd be happy to take a look.

vijayrcse commented 1 year ago

We used SQS Extended client library to publish the message to the queue. We didn't use SQStemplate of spring cloud aws However on the consumer side, we still use the SQSListener to consumer the message as string and process. If the message size is greater than 254kb, the message will have a link to the S3 bucket. Within the listener implementation we check for a specific header( it will be only available if the message is in greater than 254 kb ) and based on that we have conditional logic to retrieve the actual message from s3 reference.

NightFox7 commented 1 year ago

Ok I see. Thank you @vijayrcse

outcomes-josef-salyer commented 3 months ago

@vijayrcse do you have an example of your workaround that you can share?

vijayrcse commented 2 months ago

hi @outcomes-josef-salyer , we referred only java-extended-client-lib. You will find example code snippets in AWS documentation

agarcia-te commented 4 weeks ago

Unfortunately it implements the blocking SqsClient, which is not supported by the framework.

@tomazfernandes I believe async support has been added to the amazon-sqs-java-extended-client-lib, is this what you mean? https://github.com/awslabs/amazon-sqs-java-extended-client-lib/releases/tag/2.1.0

https://github.com/awslabs/amazon-sqs-java-extended-client-lib/blob/07d988c424dea7e4e7d128b217182e1414310560/src/main/java/com/amazon/sqs/javamessaging/AmazonSQSExtendedAsyncClient.java#L78

tomazfernandes commented 4 weeks ago

Looks like it @agarcia-te!

I wonder what we we'd need to do here to support that though, maybe the user can simply declare such client as a bean and the framework would use it?