java.util.concurrent.RejectedExecutionException

[jieap2]

This is issue is happening in prodcution recently when processing messages this is coming up.

Request to please help this urgently as can see many messages getting to dead letters in production I am running the spring boot app in Cloud run

I have atached the file also for more clarity , if you can help in this will be great. @elefeint

java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@7a136d38 rejected from java.util.concurrent.ScheduledThreadPoolExecutor@12860c34[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 614] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.reject(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(Unknown Source) at org.springframework.cloud.sleuth.instrument.async.TraceableScheduledExecutorService.schedule(TraceableScheduledExecutorService.java:48) at com.google.cloud.pubsub.v1.Publisher.setupAlarm(Publisher.java:320) at com.google.cloud.pubsub.v1.Publisher.publish(Publisher.java:267) at org.springframework.cloud.gcp.pubsub.core.publisher.PubSubPublisherTemplate.publish(PubSubPublisherTemplate.java:100) at org.springframework.cloud.gcp.pubsub.core.publisher.PubSubPublisherTemplate.publish(PubSubPublisherTemplate.java:86) at org.springframework.cloud.gcp.pubsub.core.PubSubTemplate.publish(PubSubTemplate.java:124) at .PubSubApplication.publishMessage(PubSubApplication.java:95) at .PubSubApplication.lambda$soDisptach27$1(PubSubApplication.java:393) at java.util.HashMap.forEach(Unknown Source) at .PubSubApplication.soDisptach27(PubSubApplication.java:385) at .PubSubApplication.transformToXML(PubSubApplication.java:235) at ***.PubSubApplication.lambda$messageReceiver$0(PubSubApplication.java:178) at org.springframework.integration.handler.ReplyProducingMessageHandlerWrapper.handleRequestMessage(ReplyProducingMessageHandlerWrapper.java:49) at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:127) at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:170) at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:115) at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:133) at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:106) at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:73) at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:453) at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:403) at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:187) at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:166) at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:47) at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:109) at org.springframework.integration.endpoint.MessageProducerSupport.sendMessage(MessageProducerSupport.java:198) at org.springframework.cloud.gcp.pubsub.integration.inbound.PubSubInboundChannelAdapter.consumeMessage(PubSubInboundChannelAdapter.java:148) at org.springframework.cloud.gcp.pubsub.core.subscriber.PubSubSubscriberTemplate.lambda$subscribeAndConvert$1(PubSubSubscriberTemplate.java:191) at com.google.cloud.pubsub.v1.MessageDispatcher$4.run(MessageDispatcher.java:392) at org.springframework.cloud.sleuth.instrument.async.TraceRunnable.run(TraceRunnable.java:68) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source)

[elefeint]

@jieap2 This is interesting -- `ScheduledThreadPoolExecutor@12860c34[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 614] . It looks like the publisher executor is being shut down somewhere in your application.

What version of spring-cloud-gcp are you on? Have you recently upgraded versions?

[elefeint]

Also, is this a lengthy process triggered by a single request? About how long does the processing for a single request take?

[jieap2]

@elefeint yes i have upgraded after this issue to 1.2.5.RELEASE but then also i am getting this error. Also yes a message transformation and splitting all happens for the message . But this issue doens't happen when i run in my local machine so i am not sure why this happens in cloud.

[elefeint]

It sounds like your Cloud Run container is being automatically scaled down. How long does a single process take? Cloud Run has limitations on that: https://cloud.google.com/run/docs/configuring/request-timeout

[jieap2]

when this issue happened it was taking almost more than 10 mins and Rejected exception and , Executor been shutdown was thrown.But if the processing takes time it would happen in local system but that i am not facing its not even taking 2 -3 mins to process big messages, then why only in cloud this is happening, is there any thing to be configure for cloud to have the same speed?

[elefeint]

That explains the original RejectedExecutionException -- Cloud Run is timing out your request and closing the process, per https://cloud.google.com/run/docs/configuring/request-timeout. As a workaround, you could try to set the request timeout to the maximum value of 900 seconds (15 minutes) if you are on fully managed Cloud Run.

Now, as for slowness, where I would begin to look is in your Cloud Run instance configuration -- how much memory and CPU does it have compared to your local computer.

[jieap2]

ok thanks alot for the information , i will look in that in cloud run .

[elefeint]

Great! I will close the issue for now, since it's environmental.

[elefeint]

@jieap2 You may be interested in Ray Tsang's office hours -- he knows everything about everything in GCP, and can help point to environmental best practices: http://saturnism.me/office-hour