search

elastic / logstash

Logstash - transport and process your logs, events, or other data
https://www.elastic.co/products/logstash
Other
14.21k stars 3.5k forks source link

JNR regression in Logstash 7.17.0 (Docker) #13703

Open kares opened 2 years ago

kares commented 2 years ago

hitting a code path which executes Socket.tcp_server_sockets(address, port) exposes a JNR bug.

NOTE: for now only seen this fail in tests (usually when WEBrick is used), the failure is intermittent at times.

sample reproducer

docker run -it --entrypoint "/bin/bash" docker.elastic.co/logstash/logstash:7.17.0

java.lang.RuntimeException: getprotobyname_r failed
    at org.jruby.dist/jnr.netdb.NativeProtocolsDB$LinuxNativeProtocolsDB.getProtocolByName(NativeProtocolsDB.java:180)
    at org.jruby.dist/jnr.netdb.NativeProtocolsDB.load(NativeProtocolsDB.java:80)
    at org.jruby.dist/jnr.netdb.NativeProtocolsDB.access$000(NativeProtocolsDB.java:40)
    at org.jruby.dist/jnr.netdb.NativeProtocolsDB$SingletonHolder.<clinit>(NativeProtocolsDB.java:47)
    at org.jruby.dist/jnr.netdb.NativeProtocolsDB.getInstance(NativeProtocolsDB.java:43)
    at org.jruby.dist/jnr.netdb.Protocol$ProtocolDBSingletonHolder.load(Protocol.java:107)
    at org.jruby.dist/jnr.netdb.Protocol$ProtocolDBSingletonHolder.<clinit>(Protocol.java:103)
    at org.jruby.dist/jnr.netdb.Protocol.getProtocolDB(Protocol.java:96)
    at org.jruby.dist/jnr.netdb.Protocol.getProtocolByNumber(Protocol.java:59)
    at org.jruby.dist/org.jruby.ext.socket.Addrinfo.<init>(Addrinfo.java:806)
    at org.jruby.dist/org.jruby.ext.socket.SocketUtils$2.addrinfo(SocketUtils.java:255)
    at org.jruby.dist/org.jruby.ext.socket.SocketUtils.buildAddrinfoList(SocketUtils.java:321)
    at org.jruby.dist/org.jruby.ext.socket.SocketUtils.getaddrinfoList(SocketUtils.java:231)
    at org.jruby.dist/org.jruby.ext.socket.Addrinfo.getaddrinfo(Addrinfo.java:307)
    at org.jruby.dist/org.jruby.ext.socket.Addrinfo$INVOKER$s$0$0$getaddrinfo.call(Addrinfo$INVOKER$s$0$0$getaddrinfo.gen)
    at org.jruby.dist/org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837)
    at org.jruby.dist/org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)
    at usr.share.logstash.vendor.jruby.lib.ruby.stdlib.socket.RUBY$method$foreach$0(/usr/share/logstash/vendor/jruby/lib/ruby/stdlib/socket.rb:229)
    at org.jruby.dist/org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)
    at org.jruby.dist/org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)
    at org.jruby.dist/org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:197)
    at usr.share.logstash.vendor.jruby.lib.ruby.stdlib.socket.RUBY$method$tcp_server_sockets$0(/usr/share/logstash/vendor/jruby/lib/ruby/stdlib/socket.rb:769)
    at org.jruby.dist/org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)
    at org.jruby.dist/org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)
    at org.jruby.dist/org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)
    at DashE.RUBY$script(-e:1)
    at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:710)
    at org.jruby.dist/org.jruby.ir.Compiler$1.load(Compiler.java:89)
    at org.jruby.dist/org.jruby.Ruby.runScript(Ruby.java:1205)
    at org.jruby.dist/org.jruby.Ruby.runNormally(Ruby.java:1128)
    at org.jruby.dist/org.jruby.Ruby.runNormally(Ruby.java:1146)
    at org.jruby.dist/org.jruby.Ruby.runFromMain(Ruby.java:958)
    at org.jruby.dist/org.jruby.Main.doRunFromMain(Main.java:400)
    at org.jruby.dist/org.jruby.Main.internalRun(Main.java:292)
    at org.jruby.dist/org.jruby.Main.run(Main.java:234)
    at org.jruby.dist/org.jruby.Main.main(Main.java:206)

Running the same against: docker run -it --entrypoint "/bin/bash" docker.elastic.co/logstash/logstash:7.16.3 seems to work just fine:

lukeyeager commented 2 years ago

I'm seeing the same error today when trying to follow these instructions to send data from logstash to logstash using lumberjack to beats: https://www.elastic.co/guide/en/logstash/8.2/ls-to-ls-lumberjack.html

Error on the sending side:

WARNING: Failed to load native protocols db
java.lang.RuntimeException: getprotobyname_r failed
       at jnr.netdb.NativeProtocolsDB$LinuxNativeProtocolsDB.getProtocolByName(NativeProtocolsDB.java:180)
       at jnr.netdb.NativeProtocolsDB.load(NativeProtocolsDB.java:80)
...
[2022-05-24T21:23:37,884][ERROR][logstash.outputs.lumberjack][main] All hosts unavailable, sleeping {:hosts=>[...

Error on the receiving side: [2022-05-24T21:23:37,848][INFO ][org.logstash.beats.BeatsHandler][beats][ae5c2442aab1062649e3de2da65fa67b910e91835a49b7e9331460b49a5b5122] [local: 192.168.98.7:5044, remote: 10.126.233.171:38970] Handling exception: io.netty.handler.codec.DecoderException: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 22 (caused by: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 22)

Could be unrelated to your bug, kares - but this was the only web search result for getprotobyname_r failed.

Logstash version is 8.2.0 on both sides.

kares commented 2 years ago

while the warning is present JNR (and thus Logstash) seems to be working fine.

your issue is not caused by what the issue report here is about but is rather smt is sending unexpected payload on the receiving input: io.netty.handler.codec.DecoderException: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 22 ... the protocol expected a specific format but got a different byte - likely smt else tried connecting to the port.

lukeyeager commented 2 years ago

Oh, thanks for parsing that for me! Ok, I'll look more closely at the codecs I'm using, etc.

yaauie commented 1 year ago

This will be fixed by adding the netbase package to the Docker image.

alvarofernandezavalos commented 11 months ago

USER root RUN apt install netbase -y