search

awslabs / amazon-sqs-java-messaging-lib

This Amazon SQS Java Messaging Library holds the Java Message Service compatible classes, that are used for communicating with Amazon Simple Queue Service.
http://aws.amazon.com/sqs
Apache License 2.0
170 stars 147 forks source link

java.lang.IllegalStateException: Connection pool shut down #96

Open ragnar-lothbrok opened 3 years ago

ragnar-lothbrok commented 3 years ago

I am using amazon-sqs-java-messaging-lib-1.0.8.jar. When I am sending a payload to SQS I am getting java.lang.IllegalStateException: Connection pool shut down exception.

Describe the bug When sending payload to SQS Queue getting below exception intermittently java.lang.IllegalStateException: Connection pool shut down org.apache.http.util.Asserts.check(Asserts.java:34) org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:191) …n.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:267) sun.reflect.GeneratedMethodAccessor92.invoke(Unknown Source) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) …ttp.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76) com.amazonaws.http.conn.$Proxy191.requestConnection(Unknown Source) org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:176) org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) …g.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) …rg.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) …rg.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) …mazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) …ttp.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1297) …ws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1113) …zonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:770) …http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:744) …mazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726) …onaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686) …p.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668) com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532) com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512) com.amazonaws.services.sqs.AmazonSQSClient.doInvoke(AmazonSQSClient.java:2215) com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:2182) com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:2171) …zonaws.services.sqs.AmazonSQSClient.executeGetQueueUrl(AmazonSQSClient.java:1163) com.amazonaws.services.sqs.AmazonSQSClient.getQueueUrl(AmazonSQSClient.java:1136) …amessaging.AmazonSQSMessagingClientWrapper.getQueueUrl(AmazonSQSMessagingClientWrapper.java:294) …amessaging.AmazonSQSMessagingClientWrapper.getQueueUrl(AmazonSQSMessagingClientWrapper.java:265) com.amazon.sqs.javamessaging.SQSSession.createQueue(SQSSession.java:636) …rt.destination.DynamicDestinationResolver.resolveQueue(DynamicDestinationResolver.java:85) …tion.DynamicDestinationResolver.resolveDestinationName(DynamicDestinationResolver.java:59) …tination.JmsDestinationAccessor.resolveDestinationName(JmsDestinationAccessor.java:115) org.springframework.jms.core.JmsTemplate.lambda$send$3(JmsTemplate.java:585) org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:504) org.springframework.jms.core.JmsTemplate.send(JmsTemplate.java:584)

Expected Behavior Ideally, I should get 200 and should be able to send the payload successfully to SQS.

Current Behavior Getting java.lang.IllegalStateException: Connection pool shut down Exception.

Your Environment amazon-sqs-java-messaging-lib-1.0.8.jar aws-java-sdk-sqs-1.11.832.jar Java 11 Linux

public JmsConfigurataion(JobsProperties sqsProperties) {

    EndpointConfiguration endpointConfiguration;
    endpointConfiguration = new EndpointConfiguration(
        endPoint + queueName, region);
    SQSConnectionFactory = connectionFactory = new SQSConnectionFactory(new ProviderConfiguration(),
        AmazonSQSClientBuilder.standard().withCredentials(
            new AWSStaticCredentialsProvider(new BasicAWSCredentials(
                accessKey(),
                secretKey())))
            .withEndpointConfiguration(endpointConfiguration).build());
}

@Override
@Bean(name = "jmsJobsContainerFactory")
public DefaultJmsListenerContainerFactory jmsListenerContainerFactory() {

    return super.jmsListenerContainerFactory();
}

@Bean
public DefaultJmsListenerContainerFactory jmsListenerContainerFactory() {
    DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory();
    factory.setConnectionFactory(this.connectionFactory);
    factory.setDestinationResolver(new DynamicDestinationResolver());
    factory.setConcurrency("3-10");
    factory.setMessageConverter(messageConverter());
    factory.setErrorHandler(new ErrorHandler() {
        @Override
        public void handleError(Throwable t) {
            logger.error(t.getMessage(), t);
        }
    });
    return factory;
}

@Bean(name = "jmsJobsTemplate")
public JmsTemplate defaultJmsTemplate() {
    JmsTemplate jmsTemplate = new JmsTemplate(this.connectionFactory);
    jmsTemplate.setMessageConverter(messageConverter());
    return jmsTemplate;

}

Link - https://github.com/aws/aws-sdk-java/issues/2497

rndbblnn commented 3 years ago

following. same issue happened here on our production environment

ragnar-lothbrok commented 3 years ago

following. same issue happened here on our production environment

How did you resolve?

ragnar-lothbrok commented 3 years ago

@rndbblnn In this (https://github.com/aws/aws-sdk-java/issues/2337#issuecomment-807985862) some folks mentioned increasing heap but it didn't work.

geheim01 commented 3 years ago

I am facing the same problem. Has anyone found a solution?

ddrouin commented 3 years ago

Same here with 10 long lived threads long polling indefinitely - and then... We would see this on shutting down an instance but now are also seeing this oocur on one of our dev ecs instances after about an hour running 20 sec long polls. CPU spike and kill the app is killed. Not sure if this itself is the cause or a side effect of the app shutting down yet.

kevinashaw commented 3 years ago

We are seeing a similar error with a Java 15 application. We are not using Spring or SQS, but are seeing the error on DynamoDB requests. This is on a production server instance after about 3 days of uptime and about 477K writes to DyanmoDB. We are getting errors on S3 writes also.
We are using aws-java-sdk-core version 1.12.51 and dynamodb version 2.17.24, as defined in our pom.xml file. The DDB error is:

Failed to batchWriteItem to DDB; General Exception received.
java.lang.IllegalStateException: Connection pool shut down
    at org.apache.http.util.Asserts.check(Asserts.java:34)
    at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:269)
    at jdk.internal.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:564)
    at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76)
    at com.amazonaws.http.conn.$Proxy9.requestConnection(Unknown Source)
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:176)
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
    at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1343)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1154)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:811)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:779)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:753)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:713)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:695)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:559)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:539)
    at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:6214)
    at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:6181)
    at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.executeBatchWriteItem(AmazonDynamoDBClient.java:768)
    at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.batchWriteItem(AmazonDynamoDBClient.java:732)
    at com.amazonaws.services.dynamodbv2.document.internal.BatchWriteItemImpl.doBatchWriteItem(BatchWriteItemImpl.java:113)
    at com.amazonaws.services.dynamodbv2.document.internal.BatchWriteItemImpl.batchWriteItem(BatchWriteItemImpl.java:53)
    at com.amazonaws.services.dynamodbv2.document.DynamoDB.batchWriteItem(DynamoDB.java:178)
    at com.algoint.XXX.DDB.writeRecord(DDB.java:155)
    at com.algoint.XXX.timers.Timer_WriteDDB_GeneralStatus$1.run(Timer_WriteDDB_GeneralStatus.java:79)
    at java.base/java.util.TimerThread.mainLoop(Timer.java:556)
    at java.base/java.util.TimerThread.run(Timer.java:506)
AnkitKadam567 commented 2 years ago

Does anyone find any solution here? I'm facing similar issue while fetching objects from S3

lucaalex87 commented 2 years ago

Any updates on this?

According to some AWS Documentation, if we increase the number of MaxConnections this should resolve itself or at least happen less often.

Another fix is to catch the IllegalStateException, and renew your client at that point and try again.

Can this be built into the library maybe?

Facing this in PROD also, when there is high concurrency. Affects SQS and S3Client (I know this issue is for SQS only).

Thanks for any potential future update :)

BTW this is how I create the client to allow for more connections - yes it's an app wide Bean. I'll also try to create it manually not as a bean, but with a regular singleton. (and maybe have a method that recreates the client, and call that in catch blocks... or something :) like that).

@Bean
public SqsClient sqsClient() {

    SqsClient sqs = SqsClient.builder()
            .region(Region.of(region))
            .credentialsProvider(() -> AwsBasicCredentials.create(awsId, awsKey))
            .httpClient(ApacheHttpClient.builder().maxConnections(150).build())
            .build();

    return sqs;
}
mrclrchtr commented 2 years ago

We have experienced the same problem and would be happy about an update or workaround.

krishnar700 commented 2 years ago

I am facing the same problem. while uploading the objects into S3 by using S3Publisher plugin through the Jenkins. image

Silence-Wang commented 2 years ago

I sold my problem that be same with you. I shutdown my client manually. Because the whole project use the only one client, when i start my server, it can be get data once, then i try to get the data from the same interface, it will throw this exception.

So I just remove the code "client.shutdown()", It's OK.

antique3158 commented 1 year ago

Does anyone find any solution here?

lucaalex87 commented 1 year ago

I made this workaround which seems to work for my use case:

I run all my SQS calls through something like this: the new SQSClient method just creates a new client. this is like a wrapper over the AWS class with a retry built in. whatToThrow just converts everything to a RuntimeException.

public SendMessageResponse sendMessage(SendMessageRequest request) {
        return runAWSCall(() -> getSQSClient().sendMessage(request));
    }

private <T> T runAWSCall(Callable<T> toCall) {
        try {
            return toCall.call();
        } catch (IllegalStateException | Error illegalStateException) {
            // new client and retry ONCE only
            sqsClient = newSQSClient();
            try {
                return toCall.call();
            } catch (Exception e) {
                throw whatToThrow(e);
            }
        } catch (Exception e) {
            throw whatToThrow(e);
        }

    }

You can also specify more connections for the SQS connection pool:

        private SqsClient newSQSClient() {

        SqsClient sqs = SqsClient.builder()
                .region(Region.of(region))
                .credentialsProvider(() -> AwsBasicCredentials.create(awsId, awsKey))
                .httpClient(ApacheHttpClient.builder().maxConnections(150).build())
                .build();

        return sqs;
    }

I have a similar wrapper over the S3Client too.

more info here: https://blog.adebski.com/posts/apache-http-client-shutting-down/ I got the idea from the link above: "one of the possible solution could be:

Implement a wrapper around the client libraries that use Apache HTTP under the hood. That wrapper would handle the java.lang.Errors and replace old clients with new clients."

antique3158 commented 1 year ago

Hi @lucaalex87 I use the @SqsListener annotation to consume the sqs message, so what should I do with this retry.

yasamin93 commented 1 year ago

Thanks @lucaalex87. My issue was with another AWS service, Rekognition. but setting the .httpClient(ApacheHttpClient.builder().maxConnections(150).build()) solved my issue. I also had to override the config. 

import software.amazon.awssdk.core.client.config.ClientOverrideConfiguration;
import software.amazon.awssdk.core.retry.RetryMode;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.http.apache.ApacheHttpClient;

RekognitionClient.builder()
                .region(Region.of(region))
                .httpClient(ApacheHttpClient.builder().maxConnections(150).build())
                .overrideConfiguration(
                        ClientOverrideConfiguration.builder().retryPolicy(RetryMode.STANDARD).build());