I'm getting the RejectedExecutionException issue on my Bitbucket Server instance, and following the instructions on that ticket, I've identified External Hooks as the likely culprit.
When the event queue gets full, the stack traces for the event queue threads are all from external hooks, like this one:
2017-03-16 00:02:39,897 WARN [http-nio-7990-exec-3] @1XK1CD1x2x329362x0 10.3.0.61,127.0.0.1 "GET /rest/api/1.0/projects/RR/repos/core/
pull-requests/1283/merge HTTP/1.0" c.a.s.i.e.EventThreadPoolExecutor Stack trace for AtlassianEvent::thread-2
com.atlassian.stash.internal.util.StackException: Stack trace for AtlassianEvent::thread-2
at java.lang.Object.wait(Native Method) ~[na:1.8.0_45]
at java.lang.Object.wait(Object.java:502) ~[na:1.8.0_45]
at java.lang.UNIXProcess.waitFor(UNIXProcess.java:396) ~[na:1.8.0_45]
at com.ngs.stash.externalhooks.hook.ExternalPreReceiveHook.runExternalHooks(ExternalPreReceiveHook.java:195) ~[na:na]
at com.ngs.stash.externalhooks.hook.ExternalPreReceiveHook.onReceive(ExternalPreReceiveHook.java:67) ~[na:na]
at com.ngs.stash.externalhooks.hook.ExternalAsyncPostReceiveHook.postReceive(ExternalAsyncPostReceiveHook.java:46) ~[na:na]
at com.atlassian.stash.internal.hook.repository.AsyncPostReceiveRepositoryHookAdapter$1.visit(AsyncPostReceiveRepositoryHookAda
pter.java:49) ~[bitbucket-service-impl-4.13.0.jar:na]
at com.atlassian.stash.internal.hook.repository.AsyncPostReceiveRepositoryHookAdapter$1.visit(AsyncPostReceiveRepositoryHookAda
pter.java:45) ~[bitbucket-service-impl-4.13.0.jar:na]
at com.atlassian.stash.internal.hook.repository.DefaultRepositoryHookService$RepositoryHookPagedTransactionCallback.doInTransac
tion(DefaultRepositoryHookService.java:573) ~[bitbucket-service-impl-4.13.0.jar:na]
at com.atlassian.stash.internal.hook.repository.DefaultRepositoryHookService$RepositoryHookPagedTransactionCallback.doInTransac
tion(DefaultRepositoryHookService.java:549) ~[bitbucket-service-impl-4.13.0.jar:na]
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:133) ~[spring-tx-4.3.4.RELEASE.
jar:4.3.4.RELEASE]
at com.atlassian.stash.internal.hook.repository.DefaultRepositoryHookService.visitEnabledHooks(DefaultRepositoryHookService.jav
a:279) ~[bitbucket-service-impl-4.13.0.jar:na]
at com.atlassian.stash.internal.hook.repository.AsyncPostReceiveRepositoryHookAdapter.postReceive(AsyncPostReceiveRepositoryHoo
kAdapter.java:45) ~[bitbucket-service-impl-4.13.0.jar:na]
at com.atlassian.stash.internal.hook.repository.AsyncPostReceiveRepositoryHookAdapter.onRefsChangedEvent(AsyncPostReceiveReposi
toryHookAdapter.java:40) ~[bitbucket-service-impl-4.13.0.jar:na]
at com.atlassian.event.internal.SingleParameterMethodListenerInvoker.invoke(SingleParameterMethodListenerInvoker.java:40) ~[atl
assian-event-3.1.3.jar:na]
at com.atlassian.stash.internal.event.AsyncBatchingInvokersTransformer$AsyncInvokerBatch.invoke(AsyncBatchingInvokersTransforme
r.java:109) ~[bitbucket-platform-4.13.0.jar:na]
at com.atlassian.event.internal.AsynchronousAbleEventDispatcher$1$1.run(AsynchronousAbleEventDispatcher.java:38) ~[atlassian-ev
ent-3.1.3.jar:na]
at com.atlassian.sal.core.executor.ThreadLocalDelegateRunnable.run(ThreadLocalDelegateRunnable.java:34) ~[sal-core-3.0.6.jar:na
]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
... 29 frames trimmed
That seems to indicate External Hooks is using the event thread to perform the work, instead of queueing it for execution out of its own thread pool.
I'm getting the RejectedExecutionException issue on my Bitbucket Server instance, and following the instructions on that ticket, I've identified External Hooks as the likely culprit.
When the event queue gets full, the stack traces for the event queue threads are all from external hooks, like this one:
That seems to indicate External Hooks is using the event thread to perform the work, instead of queueing it for execution out of its own thread pool.