awslabs / amazon-sqs-java-extended-client-lib

An extension to the Amazon SQS client that enables sending and receiving messages up to 2GB via Amazon S3.
Apache License 2.0
213 stars 109 forks source link

Support for SQS Lambda source event #31

Closed Paviel closed 4 years ago

Paviel commented 6 years ago

AWS now supports using SQS as a Lambda event source. https://aws.amazon.com/blogs/aws/aws-lambda-adds-amazon-simple-queue-service-to-supported-event-sources/ Events containing messages created with the extended client are hard to handle. Would be great to have a mechanism of deserializing the message and fetching the document from S3 if the message contains a reference. Thanks.

jmcazaux commented 5 years ago

Second that !

Slava96 commented 4 years ago

+1

Spiderpig86 commented 4 years ago

+1

msailes commented 4 years ago

How would you want this to work? Would you like a AWS provided library which had a special handler which fetched the S3 object behind the scenes and passed you it?

Spiderpig86 commented 4 years ago

For documentation purposes, I believe the ask is to make it easier for anyone currently consuming messages that were published by an SQSExtendedClient library and sent to an SQS queue easier.

The flow looks like this: SQSExtendedClient -> SQS Queue -> Lambda

As of now, I don't believe I have found any particular info regarding this use case. Probably this SO question can shed some light: https://stackoverflow.com/questions/57042235/how-to-use-amazonsqsextendedclient-to-recieve-data-from-an-sqsevent

We cannot directly use the SQSExtendedClient to consume messages sent to Lambda, so it sounds like a hacky solution is needed in the mean time to determine if the message is an S3 pointer or the actual payload.

This is especially painful when the SQSExtendedClient is configured to send regular messages via SQS and oversized messages via S3.

msailes commented 4 years ago

Lambda Powertools for Java now provides a seamless way to consume large messages in Lambda.

https://github.com/awslabs/aws-lambda-powertools-java

You can find the docs here: https://awslabs.github.io/aws-lambda-powertools-java/utilities/sqs_large_message_handling/

adam-aws commented 4 years ago

Thanks a lot @msailes, closing this issue