search

awslabs / amazon-sqs-java-messaging-lib

This Amazon SQS Java Messaging Library holds the Java Message Service compatible classes, that are used for communicating with Amazon Simple Queue Service.
http://aws.amazon.com/sqs
Apache License 2.0
170 stars 147 forks source link

SQSSession recover causes listener threads to block one another #46

Open thorxiong-nm opened 7 years ago

thorxiong-nm commented 7 years ago

We are seeing a issue where our jms listener threads end up blocking each other when the SQSSession#recover throws a Exception. The exception occurs in the recover process where it attempts to change the visibility timeout for a message. Since we do not have the permission SQS:ChangeMessageVisibility enabled on our queue, the request to change the message's visibility time out throws a AmazonServiceException - Access Denied. This results in a CPU spike that does not go away and continually grows if the same issue is encountered. From reading the SQS docs, my understanding is that the reason the visibility time out message is invoke during the recover process is to allow the message to be place immediately back into the queue. In instead of having this exception thrown when the queue does not have the SQS:ChangeMessageVisibility permission, is it possible to catch and log the response instead of having the error bubble up? Or, somewhere in the documentation, there should be a note in regards to having this permission enabled for SQSSession#recover to work properly.

tainer-1] ERROR - AmazonServiceException: changeMessageVisibilityBatch. RequestId: 20151ac2-2260-5a13-a37e-48a62a47d705
HTTPStatusCode: 403 AmazonErrorCode: AccessDenied
com.amazonaws.services.sqs.model.AmazonSQSException: Access to the resource https://sqs.us-east-1.amazonaws.com/070420408495/PX-NON-CLIENT-EXTRACT-POC1 is denied. (Service: AmazonSQS; Status Code: 403; Error Code: AccessDenied; Request ID: 20151ac2-2260-5a13-a37e-48a62a47d705)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1588)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1258)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1030)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
    at com.amazonaws.services.sqs.AmazonSQSClient.doInvoke(AmazonSQSClient.java:1792)
    at com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:1768)
    at com.amazonaws.services.sqs.AmazonSQSClient.executeChangeMessageVisibilityBatch(AmazonSQSClient.java:600)
    at com.amazonaws.services.sqs.AmazonSQSClient.changeMessageVisibilityBatch(AmazonSQSClient.java:576)
    at com.amazon.sqs.javamessaging.AmazonSQSMessagingClientWrapper.changeMessageVisibilityBatch(AmazonSQSMessagingClientWrapper.java:377)
    at com.amazon.sqs.javamessaging.acknowledge.NegativeAcknowledger.action(NegativeAcknowledger.java:90)
    at com.amazon.sqs.javamessaging.acknowledge.BulkSQSOperation.bulkAction(BulkSQSOperation.java:61)
    at com.amazon.sqs.javamessaging.SQSSession.recover(SQSSession.java:485)
    at org.springframework.jms.listener.AbstractMessageListenerContainer.rollbackOnExceptionIfNecessary(AbstractMessageListenerContainer.java:804)
    at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:658)
    at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:319)
    at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:257)
    at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1166)
    at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1158)
    at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1055)
    at java.lang.Thread.run(Thread.java:745)
10/20/2017 09:19:20:313 ageListenerContainer [tainer-1] ERROR - Application exception overridden by rollback exception
tamademicheli commented 5 years ago

I have the same...

stevenzyang commented 5 years ago

This has been fixed as of https://github.com/awslabs/amazon-sqs-java-messaging-lib/pull/41