search

graylog-labs / graylog-contentpack-nginx

A nginx content pack for Graylog
Apache License 2.0
75 stars 32 forks source link

Graylog reports FAILED Inputs with "Address already in use", but ports are open and logs are coming in #11

Closed juliohm1978 closed 6 years ago

juliohm1978 commented 6 years ago

Just tried installing your content pack and it all seems to be working well.

Except, Graylog keeps showing the Inputs as FAILED.

screenshot_003

At the overview panel:

screenshot_004

Even though there seems to be a problem, the ports are open and logs are coming in from remote nginx instances. I can confirm this looking at Graylog streams and by looking at the host where Graylog is installed using netstat.

I can also confirm the ports are in use by the Graylog JVM because this installation runs inside a Docker container in a Kubernetes setup. Graylog is the only process in that container.

Any ideas of why this error is coming up? I can change the Input ports manually and the message goes away. However, the error reappears after restarting the graylog instance.

joschi commented 6 years ago

@juliohm1978 "Triggered 2 hours ago" (next to "An input has failed to start"). Are you sure the problem is still current?

If so, what's in the logs of your Graylog node(s)? ➡️http://docs.graylog.org/en/2.4/pages/configuration/file_location.html

juliohm1978 commented 6 years ago

Hi @joschi,

Regardless of the message on the UI panel, I just triggered a restart of the graylog container. That means, a clean fresh instance with new logs coming up. And here's the exception I see within the first minute:

2018-02-10 00:04:04,121 ERROR: org.graylog2.plugin.inputs.transports.NettyTransport - Error in Input [Syslog UDP/5a4bc0be12522a5ae7bf6fb4] (channel [id: 0x4ecdefa4])
java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind0(Native Method) ~[?:1.8.0_72-internal]
    at sun.nio.ch.Net.bind(Net.java:433) ~[?:1.8.0_72-internal]
    at sun.nio.ch.DatagramChannelImpl.bind(DatagramChannelImpl.java:691) ~[?:1.8.0_72-internal]
    at sun.nio.ch.DatagramSocketAdaptor.bind(DatagramSocketAdaptor.java:91) ~[?:1.8.0_72-internal]
    at org.jboss.netty.channel.socket.nio.NioDatagramPipelineSink.bind(NioDatagramPipelineSink.java:129) [graylog.jar:?]
    at org.jboss.netty.channel.socket.nio.NioDatagramPipelineSink.eventSunk(NioDatagramPipelineSink.java:77) [graylog.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:779) [graylog.jar:?]
    at org.jboss.netty.channel.SimpleChannelHandler.bindRequested(SimpleChannelHandler.java:299) [graylog.jar:?]
    at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:265) [graylog.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) [graylog.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784) [graylog.jar:?]
    at org.jboss.netty.channel.SimpleChannelHandler.bindRequested(SimpleChannelHandler.java:299) [graylog.jar:?]
    at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:265) [graylog.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) [graylog.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582) [graylog.jar:?]
    at org.jboss.netty.channel.Channels.bind(Channels.java:561) [graylog.jar:?]
    at org.jboss.netty.channel.AbstractChannel.bind(AbstractChannel.java:197) [graylog.jar:?]
    at org.jboss.netty.bootstrap.ConnectionlessBootstrap.bind(ConnectionlessBootstrap.java:198) [graylog.jar:?]
    at org.graylog2.plugin.inputs.transports.NettyTransport.launch(NettyTransport.java:136) [graylog.jar:?]
    at org.graylog2.plugin.inputs.MessageInput.launch(MessageInput.java:155) [graylog.jar:?]
    at org.graylog2.shared.inputs.InputLauncher$1.run(InputLauncher.java:84) [graylog.jar:?]
    at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_72-internal]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_72-internal]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72-internal]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72-internal]
    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72-internal]
2018-02-10 00:04:04,127 ERROR: org.graylog2.shared.inputs.InputLauncher - The [org.graylog2.inputs.syslog.udp.SyslogUDPInput] input with ID <5a4bc0be12522a5ae7bf6fb4> misfired. Reason: Address already in use.
org.graylog2.plugin.inputs.MisfireException: org.graylog2.plugin.inputs.MisfireException: org.jboss.netty.channel.ChannelException: Failed to bind to: /0.0.0.0:12319
    at org.graylog2.plugin.inputs.MessageInput.launch(MessageInput.java:158) ~[graylog.jar:?]
    at org.graylog2.shared.inputs.InputLauncher$1.run(InputLauncher.java:84) [graylog.jar:?]
    at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_72-internal]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_72-internal]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72-internal]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72-internal]
    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72-internal]
Caused by: org.graylog2.plugin.inputs.MisfireException: org.jboss.netty.channel.ChannelException: Failed to bind to: /0.0.0.0:12319
    at org.graylog2.plugin.inputs.transports.NettyTransport.launch(NettyTransport.java:155) ~[graylog.jar:?]
    at org.graylog2.plugin.inputs.MessageInput.launch(MessageInput.java:155) ~[graylog.jar:?]
    ... 7 more
Caused by: org.jboss.netty.channel.ChannelException: Failed to bind to: /0.0.0.0:12319
    at org.jboss.netty.bootstrap.ConnectionlessBootstrap.bind(ConnectionlessBootstrap.java:204) ~[graylog.jar:?]
    at org.graylog2.plugin.inputs.transports.NettyTransport.launch(NettyTransport.java:136) ~[graylog.jar:?]
    at org.graylog2.plugin.inputs.MessageInput.launch(MessageInput.java:155) ~[graylog.jar:?]
    ... 7 more
Caused by: java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind0(Native Method) ~[?:1.8.0_72-internal]
    at sun.nio.ch.Net.bind(Net.java:433) ~[?:1.8.0_72-internal]
    at sun.nio.ch.DatagramChannelImpl.bind(DatagramChannelImpl.java:691) ~[?:1.8.0_72-internal]
    at sun.nio.ch.DatagramSocketAdaptor.bind(DatagramSocketAdaptor.java:91) ~[?:1.8.0_72-internal]
    at org.jboss.netty.channel.socket.nio.NioDatagramPipelineSink.bind(NioDatagramPipelineSink.java:129) ~[graylog.jar:?]
    at org.jboss.netty.channel.socket.nio.NioDatagramPipelineSink.eventSunk(NioDatagramPipelineSink.java:77) ~[graylog.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:779) ~[graylog.jar:?]
    at org.jboss.netty.channel.SimpleChannelHandler.bindRequested(SimpleChannelHandler.java:299) ~[graylog.jar:?]
    at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:265) ~[graylog.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) ~[graylog.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784) ~[graylog.jar:?]
    at org.jboss.netty.channel.SimpleChannelHandler.bindRequested(SimpleChannelHandler.java:299) ~[graylog.jar:?]
    at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:265) ~[graylog.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) ~[graylog.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582) ~[graylog.jar:?]
    at org.jboss.netty.channel.Channels.bind(Channels.java:561) ~[graylog.jar:?]
    at org.jboss.netty.channel.AbstractChannel.bind(AbstractChannel.java:197) ~[graylog.jar:?]
    at org.jboss.netty.bootstrap.ConnectionlessBootstrap.bind(ConnectionlessBootstrap.java:198) ~[graylog.jar:?]
    at org.graylog2.plugin.inputs.transports.NettyTransport.launch(NettyTransport.java:136) ~[graylog.jar:?]
    at org.graylog2.plugin.inputs.MessageInput.launch(MessageInput.java:155) ~[graylog.jar:?]
    ... 7 more
2018-02-10 00:04:04,132 INFO : org.graylog2.inputs.InputStateListener - Input [Syslog UDP/5a4bc0be12522a5ae7bf6fb4] is now FAILED
2018-02-10 00:04:04,167 ERROR: org.graylog2.plugin.inputs.transports.NettyTransport - Error in Input [Syslog UDP/5a4bc0be12522a5ae7bf6fb3] (channel [id: 0x1b19c779])
java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind0(Native Method) ~[?:1.8.0_72-internal]
    at sun.nio.ch.Net.bind(Net.java:433) ~[?:1.8.0_72-internal]
    at sun.nio.ch.DatagramChannelImpl.bind(DatagramChannelImpl.java:691) ~[?:1.8.0_72-internal]
    at sun.nio.ch.DatagramSocketAdaptor.bind(DatagramSocketAdaptor.java:91) ~[?:1.8.0_72-internal]
    at org.jboss.netty.channel.socket.nio.NioDatagramPipelineSink.bind(NioDatagramPipelineSink.java:129) [graylog.jar:?]
    at org.jboss.netty.channel.socket.nio.NioDatagramPipelineSink.eventSunk(NioDatagramPipelineSink.java:77) [graylog.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:779) [graylog.jar:?]
    at org.jboss.netty.channel.SimpleChannelHandler.bindRequested(SimpleChannelHandler.java:299) [graylog.jar:?]
    at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:265) [graylog.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) [graylog.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784) [graylog.jar:?]
    at org.jboss.netty.channel.SimpleChannelHandler.bindRequested(SimpleChannelHandler.java:299) [graylog.jar:?]
    at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:265) [graylog.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) [graylog.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582) [graylog.jar:?]
    at org.jboss.netty.channel.Channels.bind(Channels.java:561) [graylog.jar:?]
    at org.jboss.netty.channel.AbstractChannel.bind(AbstractChannel.java:197) [graylog.jar:?]
    at org.jboss.netty.bootstrap.ConnectionlessBootstrap.bind(ConnectionlessBootstrap.java:198) [graylog.jar:?]
    at org.graylog2.plugin.inputs.transports.NettyTransport.launch(NettyTransport.java:136) [graylog.jar:?]
    at org.graylog2.plugin.inputs.MessageInput.launch(MessageInput.java:155) [graylog.jar:?]
    at org.graylog2.shared.inputs.InputLauncher$1.run(InputLauncher.java:84) [graylog.jar:?]
    at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_72-internal]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_72-internal]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72-internal]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72-internal]
    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72-internal]
2018-02-10 00:04:04,169 ERROR: org.graylog2.shared.inputs.InputLauncher - The [org.graylog2.inputs.syslog.udp.SyslogUDPInput] input with ID <5a4bc0be12522a5ae7bf6fb3> misfired. Reason: Address already in use.
org.graylog2.plugin.inputs.MisfireException: org.graylog2.plugin.inputs.MisfireException: org.jboss.netty.channel.ChannelException: Failed to bind to: /0.0.0.0:12318
    at org.graylog2.plugin.inputs.MessageInput.launch(MessageInput.java:158) ~[graylog.jar:?]
    at org.graylog2.shared.inputs.InputLauncher$1.run(InputLauncher.java:84) [graylog.jar:?]
    at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_72-internal]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_72-internal]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72-internal]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72-internal]
    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72-internal]
Caused by: org.graylog2.plugin.inputs.MisfireException: org.jboss.netty.channel.ChannelException: Failed to bind to: /0.0.0.0:12318
    at org.graylog2.plugin.inputs.transports.NettyTransport.launch(NettyTransport.java:155) ~[graylog.jar:?]
    at org.graylog2.plugin.inputs.MessageInput.launch(MessageInput.java:155) ~[graylog.jar:?]
    ... 7 more
Caused by: org.jboss.netty.channel.ChannelException: Failed to bind to: /0.0.0.0:12318
    at org.jboss.netty.bootstrap.ConnectionlessBootstrap.bind(ConnectionlessBootstrap.java:204) ~[graylog.jar:?]
    at org.graylog2.plugin.inputs.transports.NettyTransport.launch(NettyTransport.java:136) ~[graylog.jar:?]
    at org.graylog2.plugin.inputs.MessageInput.launch(MessageInput.java:155) ~[graylog.jar:?]
    ... 7 more
Caused by: java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind0(Native Method) ~[?:1.8.0_72-internal]
    at sun.nio.ch.Net.bind(Net.java:433) ~[?:1.8.0_72-internal]
    at sun.nio.ch.DatagramChannelImpl.bind(DatagramChannelImpl.java:691) ~[?:1.8.0_72-internal]
    at sun.nio.ch.DatagramSocketAdaptor.bind(DatagramSocketAdaptor.java:91) ~[?:1.8.0_72-internal]
    at org.jboss.netty.channel.socket.nio.NioDatagramPipelineSink.bind(NioDatagramPipelineSink.java:129) ~[graylog.jar:?]
    at org.jboss.netty.channel.socket.nio.NioDatagramPipelineSink.eventSunk(NioDatagramPipelineSink.java:77) ~[graylog.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:779) ~[graylog.jar:?]
    at org.jboss.netty.channel.SimpleChannelHandler.bindRequested(SimpleChannelHandler.java:299) ~[graylog.jar:?]
    at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:265) ~[graylog.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) ~[graylog.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784) ~[graylog.jar:?]
    at org.jboss.netty.channel.SimpleChannelHandler.bindRequested(SimpleChannelHandler.java:299) ~[graylog.jar:?]
    at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:265) ~[graylog.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) ~[graylog.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582) ~[graylog.jar:?]
    at org.jboss.netty.channel.Channels.bind(Channels.java:561) ~[graylog.jar:?]
    at org.jboss.netty.channel.AbstractChannel.bind(AbstractChannel.java:197) ~[graylog.jar:?]
    at org.jboss.netty.bootstrap.ConnectionlessBootstrap.bind(ConnectionlessBootstrap.java:198) ~[graylog.jar:?]
    at org.graylog2.plugin.inputs.transports.NettyTransport.launch(NettyTransport.java:136) ~[graylog.jar:?]
    at org.graylog2.plugin.inputs.MessageInput.launch(MessageInput.java:155) ~[graylog.jar:?]
    ... 7 more

Filtering down to the last couple of Exceptions on the stack

Caused by: org.jboss.netty.channel.ChannelException: Failed to bind to: /0.0.0.0:12318
    at org.jboss.netty.bootstrap.ConnectionlessBootstrap.bind(ConnectionlessBootstrap.java:204) ~[graylog.jar:?]
    at org.graylog2.plugin.inputs.transports.NettyTransport.launch(NettyTransport.java:136) ~[graylog.jar:?]
    at org.graylog2.plugin.inputs.MessageInput.launch(MessageInput.java:155) ~[graylog.jar:?]
    ... 7 more
Caused by: java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind0(Native Method) ~[?:1.8.0_72-internal]
    at sun.nio.ch.Net.bind(Net.java:433) ~[?:1.8.0_72-internal]
    at sun.nio.ch.DatagramChannelImpl.bind(DatagramChannelImpl.java:691) ~[?:1.8.0_72-internal]
    at sun.nio.ch.DatagramSocketAdaptor.bind(DatagramSocketAdaptor.java:91) ~[?:1.8.0_72-internal]
    at org.jboss.netty.channel.socket.nio.NioDatagramPipelineSink.bind(NioDatagramPipelineSink.java:129) ~[graylog.jar:?]
    at org.jboss.netty.channel.socket.nio.NioDatagramPipelineSink.eventSunk(NioDatagramPipelineSink.java:77) ~[graylog.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:779) ~[graylog.jar:?]
    at org.jboss.netty.channel.SimpleChannelHandler.bindRequested(SimpleChannelHandler.java:299) ~[graylog.jar:?]
    at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:265) ~[graylog.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) ~[graylog.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784) ~[graylog.jar:?]
    at org.jboss.netty.channel.SimpleChannelHandler.bindRequested(SimpleChannelHandler.java:299) ~[graylog.jar:?]
    at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:265) ~[graylog.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) ~[graylog.jar:?]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582) ~[graylog.jar:?]
    at org.jboss.netty.channel.Channels.bind(Channels.java:561) ~[graylog.jar:?]
    at org.jboss.netty.channel.AbstractChannel.bind(AbstractChannel.java:197) ~[graylog.jar:?]
    at org.jboss.netty.bootstrap.ConnectionlessBootstrap.bind(ConnectionlessBootstrap.java:198) ~[graylog.jar:?]
    at org.graylog2.plugin.inputs.transports.NettyTransport.launch(NettyTransport.java:136) ~[graylog.jar:?]
    at org.graylog2.plugin.inputs.MessageInput.launch(MessageInput.java:155) ~[graylog.jar:?]
    ... 7 more

I manged to upload complete log file from this instance to pastebin if you want to look at the complete dump.

https://pastebin.com/raw/ahyp1stK

joschi commented 6 years ago

@juliohm1978 What's the complete output of the following commands on the machine running Graylog?

$ sudo netstat -tuplen
$ sudo ps -ef
$ sudo ifconfig -a
juliohm1978 commented 6 years ago

Here it goes

root@graylog-server-5fc77f68c8-hbqc7:/usr/share/graylog# netstat -tuplen
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       User       Inode       PID/Program name
tcp6       0      0 :::9000                 :::*                    LISTEN      1100       6884395     -               
tcp6       0      0 :::12201                :::*                    LISTEN      1100       6883484     -               
udp6       0      0 :::12318                :::*                                1100       6880078     -               
udp6       0      0 :::12319                :::*                                1100       6883360     -               
root@graylog-server-5fc77f68c8-hbqc7:/usr/share/graylog# ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
graylog      1     0  1 00:03 ?        00:21:13 /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xms1g -Xmx2g -XX:NewRatio=1 -XX:MaxMetaspaceSize=256m 
root       389     0  0 23:47 ?        00:00:00 bash
root       449   389  0 23:49 ?        00:00:00 ps -ef
root@graylog-server-5fc77f68c8-hbqc7:/usr/share/graylog# ifconfig -a
eth0      Link encap:Ethernet  HWaddr a2:26:2e:76:f0:bf  
          inet addr:10.233.118.214  Bcast:0.0.0.0  Mask:255.255.255.255
          inet6 addr: fe80::a026:2eff:fe76:f0bf/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1096947 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1427194 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:375180497 (357.8 MiB)  TX bytes:320867052 (306.0 MiB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

tunl0     Link encap:IPIP Tunnel  HWaddr   
          NOARP  MTU:1480  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
joschi commented 6 years ago

@juliohm1978 What's the output of the following commands on the machine running the Graylog container when the Graylog container is stopped?

$ sudo lsof -i : 12318
$ sudo lsof -i : 12319
$ sudo ps -ef

What's the complete configuration of the Graylog container? Is there only a single Graylog container in the Kubernetes cluster and on the Kubernetes pod?

juliohm1978 commented 6 years ago

When the Graylog container is stopped, lsof shows nothing opened on the host.

In any case, my staging environment where this is running was left untouched for the weekend. When I came back to it today, I restarted the Graylog server to run the commands you requested. Funny enough the problem disappeared by itself!

selection_001

All inputs are reported as RUNNING and it's back to normal.

I'm chalking this up to whatever inconsistency the Graylog server held for a couple of days. Like I said, even with the Inputs marked as FAILED, the ports were opened inside the container and on the host, and logs were coming in ok.

Whatever it was, it seems to be have resolved on its own during the weekend.

I apologize for the waste of time, and thank you for the help! Best regards!

lpohl commented 6 years ago

I've exactly the same problem, random input ports binding fails with Java exception in server.log, but JVM is binding nevertheless and data is coming in. Manually starting the Input results in another bind exception, stopping isn't possible. If I restart the service or hole system, another random input is failing. Graylog Version: 2.3.2-1 OS: CentOS 7 SELinux: Permissive I have this on a some of my single Graylog instances, like 2 out of 10 atm., any idea what could cause this behavior?

joschi commented 6 years ago

We are using GitHub issues for tracking bugs in Graylog itself, but this doesn't look like one. Please post this issue to our discussion forum or join the #graylog channel on freenode IRC.

Thank you!

jam49 commented 6 years ago

Seen this issue as well; generally when adding Inputs, or reapplying inputs with a content pack. Same version, same error messages. Deleting the input and recreating does not correct the issue. Restarting the container does not help either. My conclusion is that there is something that gets written into the config DB that is either duplicated or not removed. As the only solution I have that works every time, is to delete the config DB, and then apply / create the inputs via content pack or by hand. This is painful due to other config data already set.