search

logstash-plugins / logstash-input-sqs

Apache License 2.0
16 stars 40 forks source link

max max_messages configurable #30

Open jlambert121 opened 8 years ago

jlambert121 commented 8 years ago

I was running into issues with this paling failing with a AWS.SimpleQueueService.BatchEntryIdsNotDistinct error. After some digging it appears (based on my limited knowledge) that this is a threading issue with the AWS SDK. Adding some debugging output to the AWS SDK this is the list of message IDs that are being attempted to be deleted from SQS:

0736bcf8-ae4b-45d6-b628-eab3db875214
8eab91cb-7c20-4100-9b99-8458370d9535
8eab91cb-7c20-4100-9b99-8458370d9535
8eab91cb-7c20-4100-9b99-8458370d9535
7e3ba44f-c43f-48d9-ac56-49b824942388
7e3ba44f-c43f-48d9-ac56-49b824942388
7e3ba44f-c43f-48d9-ac56-49b824942388
34fe6fdd-1cce-4321-833f-8d35368e982a
34fe6fdd-1cce-4321-833f-8d35368e982a
34fe6fdd-1cce-4321-833f-8d35368e982a

rather than the 10 unique ones it received. It looks like this can be fixed when we get to Ruby >= 2. I could be completely wrong on the actual issue, but it does appear this plugin is currently broken but processing messages one at a time does resolve the issue.

More discussion greatly appreciated!