search

aws / aws-sdk-java

The official AWS SDK for Java 1.x. The AWS SDK for Java 2.x is available here: https://github.com/aws/aws-sdk-java-v2/
https://aws.amazon.com/sdkforjava
Apache License 2.0
4.12k stars 2.83k forks source link

java.lang.IllegalStateException: Connection pool shut down #2497

Closed ragnar-lothbrok closed 3 years ago

ragnar-lothbrok commented 3 years ago

I am using aws-java-sdk-sqs-1.11.832.jar. When I am sending a payload to SQS I am getting java.lang.IllegalStateException: Connection pool shut down exception.

Describe the bug

When sending payload to SQS Queue getting below exception intermittently java.lang.IllegalStateException: Connection pool shut down org.apache.http.util.Asserts.check(Asserts.java:34) org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:191) …n.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:267) sun.reflect.GeneratedMethodAccessor92.invoke(Unknown Source) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) …ttp.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76) com.amazonaws.http.conn.$Proxy191.requestConnection(Unknown Source) org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:176) org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) …g.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) …rg.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) …rg.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) …mazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) …ttp.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1297) …ws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1113) …zonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:770) …http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:744) …mazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726) …onaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686) …p.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668) com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532) com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512) com.amazonaws.services.sqs.AmazonSQSClient.doInvoke(AmazonSQSClient.java:2215) com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:2182) com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:2171) …zonaws.services.sqs.AmazonSQSClient.executeGetQueueUrl(AmazonSQSClient.java:1163) com.amazonaws.services.sqs.AmazonSQSClient.getQueueUrl(AmazonSQSClient.java:1136) …amessaging.AmazonSQSMessagingClientWrapper.getQueueUrl(AmazonSQSMessagingClientWrapper.java:294) …amessaging.AmazonSQSMessagingClientWrapper.getQueueUrl(AmazonSQSMessagingClientWrapper.java:265) com.amazon.sqs.javamessaging.SQSSession.createQueue(SQSSession.java:636) …rt.destination.DynamicDestinationResolver.resolveQueue(DynamicDestinationResolver.java:85) …tion.DynamicDestinationResolver.resolveDestinationName(DynamicDestinationResolver.java:59) …tination.JmsDestinationAccessor.resolveDestinationName(JmsDestinationAccessor.java:115) org.springframework.jms.core.JmsTemplate.lambda$send$3(JmsTemplate.java:585) org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:504) org.springframework.jms.core.JmsTemplate.send(JmsTemplate.java:584)

Expected Behavior

Ideally, I should get 200 and should be able to send the payload successfully to SQS.

Current Behavior

Getting java.lang.IllegalStateException: Connection pool shut down Exception.

Steps to Reproduce

Possible Solution

Context

Due to this Customer is not able to submit jobs and retrying multiple times.

Your Environment

debora-ito commented 3 years ago

@ragnar-lothbrok IllegalStateException: Connection pool shut down can happen if the client is closed by some part of the system by calling shutdown() (which closes the underlying connection pool), but some other part of the system continues to try and use that client to make requests.

Check if your application isn't closing the clients either directly or indirectly. For example, if you are using Spring @Bean to obtain the client, Spring will automatically close objects if they have a close() or shutdown() method: https://docs.spring.io/spring/docs/current/javadoc-api/org/springframework/context/annotation/Bean.html#destroyMethod--

ragnar-lothbrok commented 3 years ago

@debora-ito I checked in the code we are not calling shutdown anywhere. It is possible it's happening because of SQS connections are fewer? Can I add exponential retry logic to fix it.

debora-ito commented 3 years ago

Do you have any @Bean method?

github-actions[bot] commented 3 years ago

It looks like this issue hasn’t been active in longer than a week. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please add a comment to prevent automatic closure, or if the issue is already closed please feel free to reopen it.

ragnar-lothbrok commented 3 years ago

@debora-ito

EndpointConfiguration endpointConfiguration; endpointConfiguration = new EndpointConfiguration( sqsProperties.getEndpoint() + sqsProperties.getHighPriorityJobsQueueName(), sqsProperties.getRegion()); connectionFactory = new SQSConnectionFactory(new ProviderConfiguration(), AmazonSQSClientBuilder.standard().withCredentials( new AWSStaticCredentialsProvider(new BasicAWSCredentials( sqsProperties.getAccessKey(), sqsProperties.getSecretKey()))) .withEndpointConfiguration(endpointConfiguration).build());

`@Bean(name = "jmsJobsTemplate") public JmsTemplate defaultJmsTemplate() { JmsTemplate jmsTemplate = new JmsTemplate(this.connectionFactory); jmsTemplate.setMessageConverter(messageConverter()); return jmsTemplate;

}`

We are using JMS.

debora-ito commented 3 years ago

Is it possible that JmsTemplate connection pool is being automatically closed by @Bean destroyMethod?

As a convenience to the user, the container will attempt to infer a destroy method against an object returned from the @Bean method

Also, I see you are using SQSConnectionFactory which is part of the amazon-sqs-java-messaging-lib library. I suggest you open an issue in their repo - https://github.com/awslabs/amazon-sqs-java-messaging-lib

ragnar-lothbrok commented 3 years ago

@debora-ito I have raised the issue to the repo you mentioned. Thank you

debora-ito commented 3 years ago

Great, in this case I'll close this. Feel free to reach out if you have further questions.

github-actions[bot] commented 3 years ago

COMMENT VISIBILITY WARNING

Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.