search

awslabs / amazon-sqs-java-messaging-lib

This Amazon SQS Java Messaging Library holds the Java Message Service compatible classes, that are used for communicating with Amazon Simple Queue Service.
http://aws.amazon.com/sqs
Apache License 2.0
170 stars 147 forks source link

Spring DMLC - SQSSessionCallbackScheduler too many zombie threads #47

Open yeshodhan opened 6 years ago

yeshodhan commented 6 years ago

We've been facing high CPU utilization after putting in SQS JMS based implementation on production.

The root cause turned out to SQS connectivity issues and the use of CacheLevel: CACHE_CONSUMER and a Fixed BackOff. Turns out CACHE_CONSUMER is too aggressive and even with an ExponentialBackOff, it will try to retry quite aggressively. After changing the CacheLevel to CACHE_CONNECTION, the CPU utilization is stable in our test environments.

We are facing another issue though. To simulate SQS connectivity issues, we kill the network connection and monitor using VisualVM. The no. of threads created for SQSSessionCallbackScheduler are quite high and they never exit.

See screenshot below:

screenshot 2017-11-26 11 12 54 screenshot 2017-11-26 11 21 18

After resuming network connectivity, the threads don't seem to go down and new threads are created after this point.

We are using Spring DMLC with the below configuration:

ConcurrentConsumers: 10 MaxConcurrentConsumers: 25 CacheLevel: CACHE_CONNECTION (1) AcknowledgeMode: CLIENT BackOff: Exponential ( InitialInterval: 5000, Multiplier: 1.5) IdleConsumerLimit: 1 IdleTaskExecutionLimit: 5 ReceiveTimeout: 1000

Any help?

yeshodhan commented 6 years ago

I've created #48 to solve this issue.