Support on blocked InfluxDbNotifier

Versions

Jenkins 2.289.2 github-autostatus:3.6.2 See other plugins in plugins.txt.

Issue

I have tried to kill a build and now the thread finalizing the build is blocked on InfluxDbNotifier. Excerpt from /threadDump:

"Handling POST /job/A/job/B/job/C/228/kill from <IP> : Jetty (winstone)-1375720" Id=1375720 Group=main BLOCKED on org.jenkinsci.plugins.githubautostatus.notifiers.InfluxDbNotifier@18915493 owned by "Running CpsFlowExecution[Owner[A/B/C/228:A/B/C #228]]" Id=1350112
    at org.jenkinsci.plugins.githubautostatus.notifiers.InfluxDbNotifier.postData(InfluxDbNotifier.java:317)
    -  blocked on org.jenkinsci.plugins.githubautostatus.notifiers.InfluxDbNotifier@18915493
    at org.jenkinsci.plugins.githubautostatus.notifiers.InfluxDbNotifier.notifyBuildStageStatus(InfluxDbNotifier.java:164)
    at org.jenkinsci.plugins.githubautostatus.notifiers.BuildNotifierManager.lambda$notifyBuildStageStatus$0(BuildNotifierManager.java:131)
    at org.jenkinsci.plugins.githubautostatus.notifiers.BuildNotifierManager$$Lambda$691/231652661.accept(Unknown Source)
    at java.util.ArrayList.forEach(ArrayList.java:1259)
    at org.jenkinsci.plugins.githubautostatus.notifiers.BuildNotifierManager.notifyBuildStageStatus(BuildNotifierManager.java:130)
    at org.jenkinsci.plugins.githubautostatus.BuildStatusAction.updateBuildStatusForStage(BuildStatusAction.java:276)
    at org.jenkinsci.plugins.githubautostatus.BuildStatusAction.updateBuildStatusForStage(BuildStatusAction.java:288)
    at org.jenkinsci.plugins.githubautostatus.BuildStatusAction.lambda$close$1(BuildStatusAction.java:175)
    at org.jenkinsci.plugins.githubautostatus.BuildStatusAction$$Lambda$706/2120780652.accept(Unknown Source)
    at java.util.HashMap.forEach(HashMap.java:1289)
    at org.jenkinsci.plugins.githubautostatus.BuildStatusAction.close(BuildStatusAction.java:173)
    at org.jenkinsci.plugins.githubautostatus.BuildStatusAction.updateBuildStatusForJob(BuildStatusAction.java:298)
    at org.jenkinsci.plugins.githubautostatus.BuildStatusJobListener.onCompleted(BuildStatusJobListener.java:83)
    at hudson.model.listeners.RunListener.fireCompleted(RunListener.java:209)
    at org.jenkinsci.plugins.workflow.job.WorkflowRun.finish(WorkflowRun.java:617)
    at org.jenkinsci.plugins.workflow.job.WorkflowRun.doKill(WorkflowRun.java:486)
    ...

When trying

Jenkins.instance.getItemByFullName("A/B/C")
.getBuildByNumber(228)
.finish(hudson.model.Result.ABORTED, new java.io.IOException("Aborting build"));

any subsequent attempts block the same:

"Handling POST /script from <IP> : Jetty (winstone)-1437697" Id=1437697 Group=main BLOCKED on org.jenkinsci.plugins.githubautostatus.notifiers.InfluxDbNotifier@18915493 owned by "Running CpsFlowExecution[Owner[A/B/C/228:A/B/C #228]]" Id=1350112
    at org.jenkinsci.plugins.githubautostatus.notifiers.InfluxDbNotifier.postData(InfluxDbNotifier.java:317)
    -  blocked on org.jenkinsci.plugins.githubautostatus.notifiers.InfluxDbNotifier@18915493
    at org.jenkinsci.plugins.githubautostatus.notifiers.InfluxDbNotifier.notifyFinalBuildStatus(InfluxDbNotifier.java:198)
    at org.jenkinsci.plugins.githubautostatus.notifiers.BuildNotifierManager.lambda$notifyFinalBuildStatus$1(BuildNotifierManager.java:143)
    at org.jenkinsci.plugins.githubautostatus.notifiers.BuildNotifierManager$$Lambda$707/142026376.accept(Unknown Source)
    at java.util.ArrayList.forEach(ArrayList.java:1259)
    at org.jenkinsci.plugins.githubautostatus.notifiers.BuildNotifierManager.notifyFinalBuildStatus(BuildNotifierManager.java:142)
    at org.jenkinsci.plugins.githubautostatus.BuildStatusAction.updateBuildStatusForJob(BuildStatusAction.java:299)
    at org.jenkinsci.plugins.githubautostatus.BuildStatusJobListener.onCompleted(BuildStatusJobListener.java:83)
    at hudson.model.listeners.RunListener.fireCompleted(RunListener.java:209)
    at org.jenkinsci.plugins.workflow.job.WorkflowRun.finish(WorkflowRun.java:617)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoCachedMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:233)
    at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:133)
    at Script1.run(Script1.groovy:3)
    ...

The owning thread seems to be stuck at reading from a socket (that probably doesn't reply anymore, due to network issues or timeouts):

"Running CpsFlowExecution[Owner[A/B/C/228:A/B/C #228]]" Id=1350112 Group=main RUNNABLE (in native)
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
    at java.net.SocketInputStream.read(SocketInputStream.java:171)
    at java.net.SocketInputStream.read(SocketInputStream.java:141)
    at sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:457)
    at sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:165)
    at sun.security.ssl.SSLTransport.decode(SSLTransport.java:109)
    at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1383)
    at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1291)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:435)
    at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:436)
    at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:384)
    at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
    at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
    at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
    at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
    at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
    at org.jenkinsci.plugins.githubautostatus.notifiers.InfluxDbNotifier.postData(InfluxDbNotifier.java:326)
    -  locked org.jenkinsci.plugins.githubautostatus.notifiers.InfluxDbNotifier@18915493
    at org.jenkinsci.plugins.githubautostatus.notifiers.InfluxDbNotifier.notifyBuildStageStatus(InfluxDbNotifier.java:164)
    at org.jenkinsci.plugins.githubautostatus.notifiers.BuildNotifierManager.lambda$notifyBuildStageStatus$0(BuildNotifierManager.java:131)
    at org.jenkinsci.plugins.githubautostatus.notifiers.BuildNotifierManager$$Lambda$691/231652661.accept(Unknown Source)
    at java.util.ArrayList.forEach(ArrayList.java:1259)
    at org.jenkinsci.plugins.githubautostatus.notifiers.BuildNotifierManager.notifyBuildStageStatus(BuildNotifierManager.java:130)
    at org.jenkinsci.plugins.githubautostatus.BuildStatusAction.updateBuildStatusForStage(BuildStatusAction.java:276)
    at org.jenkinsci.plugins.githubautostatus.GithubBuildStatusGraphListener.onNewHead(GithubBuildStatusGraphListener.java:132)
    at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1473)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:489)
    at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:38)
    at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139)
    at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
    at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

    Number of locked synchronizers = 2
    - java.util.concurrent.ThreadPoolExecutor$Worker@26857132
    - java.util.concurrent.locks.ReentrantLock$NonfairSync@664d1da2

As the build hasn't finished for Jenkins any subsequent build attempts get queued.

Questions

So how do I get the Thread unstuck without restarting the Jenkins Controller instance? Is finding and closing the socket an option?
I'm not sure how unlucky I got that I ended up in this situation. In Python when I'm in a comparable situation I normally put a timeout (and retries after some back-off with a counter) around socket based operations so I can't get stuck there. Especially if a lock is acquired around the socket operation. My experience is that (our) networks aren't as reliable as we want them to be...

If you need more information, please let me know.

jenkinsci / github-autostatus-plugin