search

Graylog2 / graylog2-server

Free and open log management
https://www.graylog.org
Other
7.44k stars 1.07k forks source link

unable to start GELF input on IPv6 address #15177

Open dch opened 1 year ago

dch commented 1 year ago

Want to have IPv6 input for GELF data.

Expected Behavior

Input starts successfully

Current Behavior

An input has failed to start (triggered 3 minutes ago)
Input 640bce814232ab7b1843d60f has failed to start on node 2b572ad1-3d43-4707-a729-ccbcf6a16ee9 for this reason: »null«.

logs:

21:36:45.788 [eventbus-handler-1] INFO  org.graylog2.inputs.InputStateListener - Input [GELF TCP/6434816da0af4957c20395a4] is now STARTING
21:36:45.822 [netty-transport-2] WARN  org.graylog2.plugin.inputs.transports.AbstractTcpTransport - Failed to start channel for input GELFTCPInput{title=gelf, type=org.graylog2.inputs.gelf.tcp.GELFTCPInput, nodeId=e7deda98-c981-43dc-bc53-239950930eab}
java.nio.channels.UnsupportedAddressTypeException: null
        at sun.nio.ch.Net.checkAddress(Net.java:161) ~[?:?]
        at sun.nio.ch.ServerSocketChannelImpl.netBind(ServerSocketChannelImpl.java:330) ~[?:?]
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:294) ~[?:?]
        at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:141) ~[graylog.jar:?]
        at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:562) [graylog.jar:?]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1334) [graylog.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:600) [graylog.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:579) [graylog.jar:?]
        at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:973) [graylog.jar:?]
        at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:260) [graylog.jar:?]
        at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:356) [graylog.jar:?]
        at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174) [graylog.jar:?]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167) [graylog.jar:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) [graylog.jar:?]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569) [graylog.jar:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [graylog.jar:?]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180) [graylog.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
21:36:45.847 [inputs-0] ERROR org.graylog2.shared.inputs.InputLauncher - The [org.graylog2.inputs.gelf.tcp.GELFTCPInput] input with ID <6434816da0af4957c20395a4> misfired. Reason: null
org.graylog2.plugin.inputs.MisfireException: org.graylog2.plugin.inputs.MisfireException: java.nio.channels.UnsupportedAddressTypeException
        at org.graylog2.plugin.inputs.MessageInput.launch(MessageInput.java:161) ~[graylog.jar:?]
        at org.graylog2.shared.inputs.InputLauncher$1.run(InputLauncher.java:91) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180) [graylog.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: org.graylog2.plugin.inputs.MisfireException: java.nio.channels.UnsupportedAddressTypeException
        at org.graylog2.plugin.inputs.transports.AbstractTcpTransport.launch(AbstractTcpTransport.java:224) ~[graylog.jar:?]
        at org.graylog2.plugin.inputs.MessageInput.launch(MessageInput.java:158) ~[graylog.jar:?]
        ... 7 more
Caused by: java.nio.channels.UnsupportedAddressTypeException
        at sun.nio.ch.Net.checkAddress(Net.java:161) ~[?:?]
        at sun.nio.ch.ServerSocketChannelImpl.netBind(ServerSocketChannelImpl.java:330) ~[?:?]
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:294) ~[?:?]
        at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:141) ~[graylog.jar:?]
        at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:562) ~[graylog.jar:?]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1334) ~[graylog.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:600) ~[graylog.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:579) ~[graylog.jar:?]
        at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:973) ~[graylog.jar:?]
        at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:260) ~[graylog.jar:?]
        at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:356) ~[graylog.jar:?]
        at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174) ~[graylog.jar:?]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167) ~[graylog.jar:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[graylog.jar:?]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569) ~[graylog.jar:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[graylog.jar:?]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[graylog.jar:?]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180) ~[graylog.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) ~[graylog.jar:?]
        ... 1 more

Possible Solution

Steps to Reproduce (for bugs)

Add an IPv6 address GELF input, and try to start it.

Context

IPv6 address tried as [::1] and without ::1,or fca2:927d:4d50:715c:f0b4::5 same result.

    bind_address:
     [::1]
    charset_name:
     UTF-8
    decompress_size_limit:
     8388608
    max_message_size:
     2097152
    number_worker_threads:
     4
    override_source:
     <empty>
    port:
     11514
    recv_buffer_size:
     1048576
    tcp_keepalive:
     true
    tls_cert_file:
     <empty>
    tls_client_auth:
     disabled
    tls_client_auth_cert_file:
     <empty>
    tls_enable:
     false
    tls_key_file:
     <empty>
    tls_key_password:
    ********
    use_null_delimiter:
     true

Your Environment

dch commented 1 year ago

This used to work, IIRC under graylog 4.x and possibly an older JVM. An acceptable workaround for me has been to use socat

# /usr/local/etc/socat-instances.conf
[graylog]
flags="-ly TCP6-LISTEN:1514,reuseaddr,fork,reuseport TCP4:127.0.0.1:1514"

Which forks a process per inbound connection and proxies from IPv6 to IPv4. This may not be suitable for environments with many connections, or short-lived ones.

dch commented 1 year ago

still broken in 5.1.6

NiceRath commented 1 year ago

Greetings!

IPv6 input-listener also not working on 5.0.7 (debian) using GELF-TCP, Syslog-TCP and Syslog-UDP.

2023-06-29T11:24:18.331+02:00 INFO  [InputStateListener] Input [Syslog TCP/649d4dc20f63f62fd696a775] is now STARTING
2023-06-29T11:24:18.333+02:00 WARN  [AbstractTcpTransport] Failed to start channel for input SyslogTCPInput{title=test, type=org.graylog2.inputs.syslog.tcp.SyslogTCPInput, nodeId=c434835d-ffa2-44f3-9fd7-d194e476aae8}
io.netty.channel.unix.Errors$NativeIoException: bind(..) failed: Address family not supported by protocol
2023-06-29T11:24:18.333+02:00 ERROR [InputLauncher] The [org.graylog2.inputs.syslog.tcp.SyslogTCPInput] input with ID <649d4dc20f63f62fd696a775> misfired. Reason: bind(..) failed: Address family not supported by protocol.
org.graylog2.plugin.inputs.MisfireException: org.graylog2.plugin.inputs.MisfireException: io.netty.channel.unix.Errors$NativeIoException: bind(..) failed: Address family not supported by protocol
        at org.graylog2.plugin.inputs.MessageInput.launch(MessageInput.java:161) ~[graylog.jar:?]
        at org.graylog2.shared.inputs.InputLauncher$1.run(InputLauncher.java:91) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180) [graylog.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: org.graylog2.plugin.inputs.MisfireException: io.netty.channel.unix.Errors$NativeIoException: bind(..) failed: Address family not supported by protocol
        at org.graylog2.plugin.inputs.transports.AbstractTcpTransport.launch(AbstractTcpTransport.java:224) ~[graylog.jar:?]
        at org.graylog2.plugin.inputs.MessageInput.launch(MessageInput.java:158) ~[graylog.jar:?]
        ... 7 more
Caused by: io.netty.channel.unix.Errors$NativeIoException: bind(..) failed: Address family not supported by protocol
2023-06-29T11:24:18.337+02:00 INFO  [InputStateListener] Input [Syslog TCP/649d4dc20f63f62fd696a775] is now FAILED

- Rath

NiceRath commented 1 year ago

FYI: Working for me with Syslog-TCP in Graylog 5.1.4:

bind_address: [::1]

netstat -tulpn | grep 514
tcp6       0      0 ::1:514 :::*                    LISTEN      xxxx/java
NiceRath commented 1 year ago

Interestingly: Another server with the same version still shows the issue..

Works:

# version in WebUI
Graylog 5.1.5+993cd0f on xxx1 (Eclipse Adoptium 17.0.8 on Linux 5.10.0-25-amd64)

uname -a
> Linux xxx1 5.10.0-25-amd64 #1 SMP Debian 5.10.191-1 (2023-08-16) x86_64 GNU/Linux

apt policy graylog-server 
> graylog-server:
>   Installed: 5.1.5-1
>   Candidate: 5.1.5-1
>   Version table:
>  *** 5.1.5-1 500
>         500 https://packages.graylog2.org/repo/debian stable/5.1 amd64 Packages

apt list --installed | grep jdk
> openjdk-17-jre-headless/oldstable-security,now 17.0.7+7-1~deb11u1 amd64 [installed]

Does not work:

# version in WebUI
Graylog 5.1.5+993cd0f on xxx2 (Debian 17.0.7 on Linux 5.10.0-25-amd64)

uname -a
> Linux xxx2 5.10.0-25-amd64 #1 SMP Debian 5.10.191-1 (2023-08-16) x86_64 GNU/Linux

apt policy graylog-server
> graylog-server:
>   Installed: 5.1.5-1
>   Candidate: 5.1.5-1
>   Version table:
>  *** 5.1.5-1 500
>         500 https://packages.graylog2.org/repo/debian stable/5.1 amd64 Packages

apt list --installed | grep jdk
> openjdk-17-jre-headless/oldstable-security,now 17.0.7+7-1~deb11u1 amd64 [installed]

Note also: sysctl -a | grep '^net' shows the exact same values on both nodes (IPv6 is not disabled and works for other services)

Error:

2023-09-19T15:50:26.034+02:00 ERROR [InputLauncher] The [org.graylog2.inputs.syslog.tcp.SyslogTCPInput] input [Syslog TCP/IN_syslog_TLS_IP6/6509a324bc50f342a21b28bc] misfired. Reason: bind(..) failed: Address family not supported by protocol.
org.graylog2.plugin.inputs.MisfireException: org.graylog2.plugin.inputs.MisfireException: io.netty.channel.unix.Errors$NativeIoException: bind(..) failed: Address family not supported by protocol
        at org.graylog2.plugin.inputs.MessageInput.launch(MessageInput.java:167) ~[graylog.jar:?]
        at org.graylog2.shared.inputs.InputLauncher$1.run(InputLauncher.java:94) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180) [graylog.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: org.graylog2.plugin.inputs.MisfireException: io.netty.channel.unix.Errors$NativeIoException: bind(..) failed: Address family not supported by protocol
        at org.graylog2.plugin.inputs.transports.AbstractTcpTransport.launch(AbstractTcpTransport.java:225) ~[graylog.jar:?]
        at org.graylog2.plugin.inputs.transports.Transport.launch(Transport.java:35) ~[graylog.jar:?]
        at org.graylog2.plugin.inputs.MessageInput.launch(MessageInput.java:164) ~[graylog.jar:?]
        ... 7 more
Caused by: io.netty.channel.unix.Errors$NativeIoException: bind(..) failed: Address family not supported by protocol
2023-09-19T15:50:26.035+02:00 INFO  [InputStateListener] Input [Syslog TCP/IN_syslog_TLS_IP6/6509a324bc50f342a21b28bc] is now FAILED
NiceRath commented 1 year ago

We were able to solve the bind(..) failed: Address family not supported by protocol error by removing & purging graylog and re-installing it:

apt purge graylog-server
apt purge openjdk-17-jre-headless
# reboot

Now the WebUI also shows the version: Graylog 5.1.5+993cd0f on log (Eclipse Adoptium 17.0.8 on Linux 5.10.0-25-amd64)