search

awslabs / amazon-sqs-java-messaging-lib

This Amazon SQS Java Messaging Library holds the Java Message Service compatible classes, that are used for communicating with Amazon Simple Queue Service.
http://aws.amazon.com/sqs
Apache License 2.0
170 stars 147 forks source link

JMS Listener threads block one another after deleteMessage failure #65

Open justinfalk opened 6 years ago

justinfalk commented 6 years ago

After a couple of days and thousands of successfully processed messages our JMS message listener encountered a server-side AWS error. While attempting to recover it blocked all of the threads which required the application to be restarted.

2018-07-26 10:06:49,172 WARN  -> (handler-6) [DefaultMessageListenerContainer] Setup of JMS message listener invoker failed for destination 'MY_EXAMPLE_QUEUE.fifo' - trying to recover. Cause: AmazonServiceException: deleteMessage. RequestId: e4796bb3-1242-5fc0-9277-3860f7f7dbf8
HTTPStatusCode: 500 AmazonErrorCode: InternalError
2018-07-26 10:06:49,172 INFO  -> (handler-6) [SQSMessageConsumer] Shutting down ConsumerPrefetch executor
2018-07-26 10:06:49,172 INFO  -> (handler-6) [SQSMessageConsumer] Shutting down ConsumerPrefetch executor
2018-07-26 10:06:55,309 INFO  -> (handler-6) [SQSSession] Shutting down SessionCallBackScheduler executor
2018-07-26 10:06:55,310 INFO  -> (handler-6) [SQSMessageConsumer] Shutting down ConsumerPrefetch executor
2018-07-26 10:07:00,833 INFO  -> (handler-6) [SQSSession] Shutting down SessionCallBackScheduler executor
2018-07-26 10:07:00,833 INFO  -> (handler-6) [SQSMessageConsumer] Shutting down ConsumerPrefetch executor
2018-07-26 10:07:18,390 INFO  -> (handler-6) [SQSSession] Shutting down SessionCallBackScheduler executor
2018-07-26 10:07:18,390 INFO  -> (handler-6) [SQSMessageConsumer] Shutting down ConsumerPrefetch executor
2018-07-26 10:07:18,411 INFO  -> (handler-6) [SQSSession] Shutting down SessionCallBackScheduler executor
2018-07-26 10:07:18,411 INFO  -> (handler-6) [SQSMessageConsumer] Shutting down ConsumerPrefetch executor
2018-07-26 10:07:22,651 INFO  -> (handler-6) [SQSSession] Shutting down SessionCallBackScheduler executor
2018-07-26 10:07:22,651 INFO  -> (handler-6) [SQSMessageConsumer] Shutting down ConsumerPrefetch executor
2018-07-26 10:07:28,260 INFO  -> (handler-6) [SQSSession] Shutting down SessionCallBackScheduler executor
2018-07-26 10:07:28,260 INFO  -> (handler-6) [SQSMessageConsumer] Shutting down ConsumerPrefetch executor
2018-07-26 10:07:44,331 INFO  -> (handler-6) [SQSSession] Shutting down SessionCallBackScheduler executor
2018-07-26 10:07:44,331 INFO  -> (handler-6) [SQSMessageConsumer] Shutting down ConsumerPrefetch executor
2018-07-26 10:07:50,191 INFO  -> (handler-6) [SQSSession] Shutting down SessionCallBackScheduler executor
2018-07-26 10:07:50,192 INFO  -> (handler-6) [SQSMessageConsumer] Shutting down ConsumerPrefetch executor
2018-07-26 10:07:50,211 INFO  -> (handler-6) [SQSSession] Shutting down SessionCallBackScheduler executor
2018-07-26 10:07:50,211 INFO  -> (handler-6) [SQSMessageConsumer] Shutting down ConsumerPrefetch executor
2018-07-26 10:07:50,234 INFO  -> (handler-6) [SQSSession] Shutting down SessionCallBackScheduler executor
2018-07-26 10:07:50,240 INFO  -> (handler-6) [DefaultMessageListenerContainer] Successfully refreshed JMS Connection
justinfalk commented 6 years ago

Appears similar to #46

rkass commented 6 years ago

@justinfalk Were you ever able to get more insight on this? We're experiencing the exact same log stream without the access denied errors of #46

stevenzyang commented 5 years ago

This has been fixed as of https://github.com/awslabs/amazon-sqs-java-messaging-lib/pull/41