search

aws / aws-sdk-js-v3

Modularized AWS SDK for JavaScript.
Apache License 2.0
2.98k stars 560 forks source link

SQS - Dedup messages in a batch when using deleteMessageBatch #5950

Closed shyam2794 closed 3 months ago

shyam2794 commented 3 months ago

Describe the feature

Remove the duplicate messages within a single batch when deleting in sqs.

This has been handled in the aws sdk for ruby - https://github.com/aws/aws-sdk-ruby/blob/version-3/gems/aws-sdk-sqs/lib/aws-sdk-sqs/queue_poller.rb#L434

https://github.com/aws/aws-sdk-ruby/discussions/2908#discussioncomment-7082645

PR - https://github.com/aws/aws-sdk-ruby/pull/2918

Use Case

When there are messages with same id in a single batch , deletion fails with the error BatchEntryIdsNotDistinct . This is already handled in AWS SDK ruby , we can do the same for js as well .

Proposed Solution

From AWS docs ,

If you receive a message more than once, each time you receive it, you get a different receipt handle. You must provide the most recently received receipt handle when you request to delete the message (otherwise, the message might not be deleted).

We could reverse the order of message Id's remove the duplicate and reverse the message to the original order . This should retain the duplicate message with recent receipt handle .

Other Information

No response

Acknowledgements

SDK version used

v3

Environment details (OS name and version, etc.)

All OS

RanVaknin commented 3 months ago

Hi @shyam2794 ,

Thanks for reaching out. I have raised this with the SDK team and we have decided that we have no plans implementing this for multiple reasons.

  1. SDKs have moved away from customizations exactly for this reason. Unless there is a unified, customer driven, cross SDK feature request that standardizes high level libraries (for example lib-storage or lib-dynamo) implementing features on a per SDK level can create feature parity gaps.

  2. Since this would be a handwritten customization (not code generated unlike 95% of the SDK), this means we will have to port it over to the next major version otherwise will cause feature parity between the two major versions.

The solution here is to write a helper function to dedupe the batch list yourself.

Thanks again, Ran~

github-actions[bot] commented 2 months ago

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.