RejectedExecutionException with externalhooks flooding the event queue

[dcsobral]

I'm getting the RejectedExecutionException issue on my Bitbucket Server instance, and following the instructions on that ticket, I've identified External Hooks as the likely culprit.

When the event queue gets full, the stack traces for the event queue threads are all from external hooks, like this one:

2017-03-16 00:02:39,897 WARN  [http-nio-7990-exec-3] @1XK1CD1x2x329362x0 10.3.0.61,127.0.0.1 "GET /rest/api/1.0/projects/RR/repos/core/
pull-requests/1283/merge HTTP/1.0" c.a.s.i.e.EventThreadPoolExecutor Stack trace for AtlassianEvent::thread-2
com.atlassian.stash.internal.util.StackException: Stack trace for AtlassianEvent::thread-2
        at java.lang.Object.wait(Native Method) ~[na:1.8.0_45]
        at java.lang.Object.wait(Object.java:502) ~[na:1.8.0_45]
        at java.lang.UNIXProcess.waitFor(UNIXProcess.java:396) ~[na:1.8.0_45]
        at com.ngs.stash.externalhooks.hook.ExternalPreReceiveHook.runExternalHooks(ExternalPreReceiveHook.java:195) ~[na:na]
        at com.ngs.stash.externalhooks.hook.ExternalPreReceiveHook.onReceive(ExternalPreReceiveHook.java:67) ~[na:na]
        at com.ngs.stash.externalhooks.hook.ExternalAsyncPostReceiveHook.postReceive(ExternalAsyncPostReceiveHook.java:46) ~[na:na]
        at com.atlassian.stash.internal.hook.repository.AsyncPostReceiveRepositoryHookAdapter$1.visit(AsyncPostReceiveRepositoryHookAda
pter.java:49) ~[bitbucket-service-impl-4.13.0.jar:na]
        at com.atlassian.stash.internal.hook.repository.AsyncPostReceiveRepositoryHookAdapter$1.visit(AsyncPostReceiveRepositoryHookAda
pter.java:45) ~[bitbucket-service-impl-4.13.0.jar:na]
        at com.atlassian.stash.internal.hook.repository.DefaultRepositoryHookService$RepositoryHookPagedTransactionCallback.doInTransac
tion(DefaultRepositoryHookService.java:573) ~[bitbucket-service-impl-4.13.0.jar:na]
        at com.atlassian.stash.internal.hook.repository.DefaultRepositoryHookService$RepositoryHookPagedTransactionCallback.doInTransac
tion(DefaultRepositoryHookService.java:549) ~[bitbucket-service-impl-4.13.0.jar:na]
        at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:133) ~[spring-tx-4.3.4.RELEASE.
jar:4.3.4.RELEASE]
        at com.atlassian.stash.internal.hook.repository.DefaultRepositoryHookService.visitEnabledHooks(DefaultRepositoryHookService.jav
a:279) ~[bitbucket-service-impl-4.13.0.jar:na]
        at com.atlassian.stash.internal.hook.repository.AsyncPostReceiveRepositoryHookAdapter.postReceive(AsyncPostReceiveRepositoryHoo
kAdapter.java:45) ~[bitbucket-service-impl-4.13.0.jar:na]
        at com.atlassian.stash.internal.hook.repository.AsyncPostReceiveRepositoryHookAdapter.onRefsChangedEvent(AsyncPostReceiveReposi
toryHookAdapter.java:40) ~[bitbucket-service-impl-4.13.0.jar:na]
        at com.atlassian.event.internal.SingleParameterMethodListenerInvoker.invoke(SingleParameterMethodListenerInvoker.java:40) ~[atl
assian-event-3.1.3.jar:na]
        at com.atlassian.stash.internal.event.AsyncBatchingInvokersTransformer$AsyncInvokerBatch.invoke(AsyncBatchingInvokersTransforme
r.java:109) ~[bitbucket-platform-4.13.0.jar:na]
        at com.atlassian.event.internal.AsynchronousAbleEventDispatcher$1$1.run(AsynchronousAbleEventDispatcher.java:38) ~[atlassian-ev
ent-3.1.3.jar:na]
        at com.atlassian.sal.core.executor.ThreadLocalDelegateRunnable.run(ThreadLocalDelegateRunnable.java:34) ~[sal-core-3.0.6.jar:na
]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_45]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
        ... 29 frames trimmed

That seems to indicate External Hooks is using the event thread to perform the work, instead of queueing it for execution out of its own thread pool.

[seletskiy]

@dcsobral: Can you propose a fix for this?

[seletskiy]

Closed due architectural changes made both in BB & External Hooks.