[GelfTcpTransport] Connection failed: executor not accepting a task error

[darouwan]

When I tried to initialize a gelf transport and send some logs to gelf tcp input, it throw the error "[GelfTcpTransport] Connection failed: executor not accepting a task ". It seems to be threw by netty but I cannot locate where it threw.

What is the reason it may be caused by? How to solve it?

[joschi]

@darouwan Please provide the complete log message (and some context around it) and ideally a minimal project to reproduce the issue.

[ghost]

I'm running into this too. After modifying the GelfClient source to log the stack trace, I see this.

2018-06-27T18:05:10,508Z [sessionId:] [workflowExecutionId:] [gelfTcpTransport-1-1] ERROR o.g.g.transport.GelfTcpTransport - Connection failed: executor not accepting a task
java.lang.IllegalStateException: executor not accepting a task
    at io.netty.resolver.AddressResolverGroup.getResolver(AddressResolverGroup.java:60)
    at io.netty.bootstrap.Bootstrap.doResolveAndConnect0(Bootstrap.java:200)
    at io.netty.bootstrap.Bootstrap.access$000(Bootstrap.java:49)
    at io.netty.bootstrap.Bootstrap$1.operationComplete(Bootstrap.java:188)
    at io.netty.bootstrap.Bootstrap$1.operationComplete(Bootstrap.java:174)
    at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511)
    at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:485)
    at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424)
    at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:103)
    at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84)
    at io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetSuccess(AbstractChannel.java:978)
    at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:512)
    at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:423)
    at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:482)
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:465)
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run(Thread.java:748)

This is possible related to this netty issue but I am still investigating.

The code in question in netty:

  if (executor.isShuttingDown()) {
           throw new IllegalStateException("executor not accepting a task");
       }

I'm running inside a Docker container in case that is relevant info.

[ghost]

So for my use case the issue was that the GelfAppender had been stopped. DropWizard was taking over the logging after it had been initialized and stopped all appenders and then did its own thing. It wasn't an issue with the GelfAppender or the Graylog client.