search

linux-on-ibm-z / docs

A collection of instructions and tutorials to help you get your favorite software up and running on Linux on z
147 stars 56 forks source link

Logstash - Missing JFFI Library #33

Open bendermIBM opened 5 years ago

bendermIBM commented 5 years ago

@cwsolee Trying to run through the Logstash build process for a customer and the automated build script does look like it worked and I am able to run logstash -v however once I try to run it I am getting a missing library error for JFFI

[2019-09-26T14:15:59,875][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (LoadError) Could not load FFI Provider: (NotImplementedError) FFI not available: java.lang.UnsatisfiedLinkError: could not locate stub library in jar file.  Tried [jni/s390x-Linux/libjffi-1.2.so, /jni/s390x-Linux/libjffi-1.2.so]

strace : https://gist.github.com/bendermIBM/76cddd0fa7b8da087be723fa5c881cde

Could your team take a look? Let me know if you need access to the machine

bendermIBM commented 5 years ago

@cwsolee I just tried with AdoptJDK 8 per our messages

java -version

openjdk version "1.8.0_222"
OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_222-b10)
OpenJDK 64-Bit Zero VM (AdoptOpenJDK)(build 25.222-b10, interpreted mode)

echo $JAVA_HOME

/opt/ibm/jdk8u222-b10

echo $PATH

/opt/ibm/jdk8u222-b10/bin:/opt/ibm/jdk8u222-b10/bin:/opt/ibm/java-s390x-80/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin

echo $LD_LIBRARY_PATH

/usr/local/jffi/build/jni:/usr/local/jffi/build/jni:/usr/local/jffi/build/jni:

We are still getting a JFFI error as with OpenJDK8, IBM Java 8 and OpenJDK11

logstash -f /etc/logstash/conf.d -l /var/log/logstash --log.level=debug --path.settings=/etc/logstash

Thread.exclusive is deprecated, use Thread::Mutex
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2019-09-27T11:44:54,424][DEBUG][logstash.modules.scaffold] Found module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2019-09-27T11:44:54,445][DEBUG][logstash.plugins.registry] Adding plugin to the registry {:name=>"fb_apache", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0x407bfc49 @directory="/usr/share/logstash/modules/fb_apache/configuration", @module_name="fb_apache", @kibana_version_parts=["6", "0", "0"]>}
[2019-09-27T11:44:54,461][DEBUG][logstash.modules.scaffold] Found module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2019-09-27T11:44:54,462][DEBUG][logstash.plugins.registry] Adding plugin to the registry {:name=>"netflow", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0x543d242e @directory="/usr/share/logstash/modules/netflow/configuration", @module_name="netflow", @kibana_version_parts=["6", "0", "0"]>}
[2019-09-27T11:44:59,022][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (LoadError) Could not load FFI Provider: (NotImplementedError) FFI not available: java.lang.UnsatisfiedLinkError: could not locate stub library in jar file.  Tried [jni/s390x-Linux/libjffi-1.2.so, /jni/s390x-Linux/libjffi-1.2.so]
    at com.kenai.jffi.internal.StubLoader.getStubLibraryStream(StubLoader.java:412)
    at com.kenai.jffi.internal.StubLoader.loadFromJar(StubLoader.java:355)
    at com.kenai.jffi.internal.StubLoader.load(StubLoader.java:258)
    at com.kenai.jffi.internal.StubLoader.<clinit>(StubLoader.java:449)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:348)
    at com.kenai.jffi.Init.load(Init.java:68)
    at com.kenai.jffi.Foreign$InstanceHolder.getInstanceHolder(Foreign.java:49)
    at com.kenai.jffi.Foreign$InstanceHolder.<clinit>(Foreign.java:45)
    at com.kenai.jffi.Foreign.getInstance(Foreign.java:103)
    at com.kenai.jffi.Type$Builtin.lookupTypeInfo(Type.java:242)
    at com.kenai.jffi.Type$Builtin.getTypeInfo(Type.java:237)
    at com.kenai.jffi.Type.resolveSize(Type.java:155)
    at com.kenai.jffi.Type.size(Type.java:138)
    at jnr.ffi.provider.jffi.NativeRuntime$TypeDelegate.size(NativeRuntime.java:178)
    at jnr.ffi.provider.AbstractRuntime.<init>(AbstractRuntime.java:48)
    at jnr.ffi.provider.jffi.NativeRuntime.<init>(NativeRuntime.java:57)
    at jnr.ffi.provider.jffi.NativeRuntime.<init>(NativeRuntime.java:41)
    at jnr.ffi.provider.jffi.NativeRuntime$SingletonHolder.<clinit>(NativeRuntime.java:53)
    at jnr.ffi.provider.jffi.NativeRuntime.getInstance(NativeRuntime.java:49)
    at jnr.ffi.provider.jffi.Provider.<init>(Provider.java:29)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at java.lang.Class.newInstance(Class.java:442)
    at jnr.ffi.provider.FFIProvider$SystemProviderSingletonHolder.getInstance(FFIProvider.java:68)
    at jnr.ffi.provider.FFIProvider$SystemProviderSingletonHolder.<clinit>(FFIProvider.java:57)
    at jnr.ffi.provider.FFIProvider.getSystemProvider(FFIProvider.java:35)
    at jnr.ffi.Library.loadLibrary(Library.java:114)
    at jnr.posix.POSIXFactory$DefaultLibCProvider$SingletonHolder.<clinit>(POSIXFactory.java:281)
    at jnr.posix.POSIXFactory$DefaultLibCProvider.getLibC(POSIXFactory.java:310)
    at jnr.posix.BaseNativePOSIX.<init>(BaseNativePOSIX.java:38)
    at jnr.posix.LinuxPOSIX.<init>(LinuxPOSIX.java:19)
    at jnr.posix.POSIXFactory.loadLinuxPOSIX(POSIXFactory.java:146)
    at jnr.posix.POSIXFactory.loadNativePOSIX(POSIXFactory.java:124)
    at jnr.posix.POSIXFactory.loadPOSIX(POSIXFactory.java:93)
    at jnr.posix.LazyPOSIX.loadPOSIX(LazyPOSIX.java:38)
    at jnr.posix.LazyPOSIX.posix(LazyPOSIX.java:32)
    at jnr.posix.LazyPOSIX.isNative(LazyPOSIX.java:402)
    at org.jruby.RubyGlobal.initSTDIO(RubyGlobal.java:287)
    at org.jruby.RubyGlobal.createGlobals(RubyGlobal.java:211)
    at org.jruby.Ruby.init(Ruby.java:1260)
    at org.jruby.Ruby.newInstance(Ruby.java:370)
    at org.logstash.Logstash.<init>(Logstash.java:88)
    at org.logstash.Logstash.main(Logstash.java:43)

 See http://jira.codehaus.org/browse/JRUBY-4583

Error accessing temp directory: /tmp
This often occurs because the temp directory has been mounted with NOEXEC or
the Logstash user has insufficient permissions on the directory. Possible
workarounds include setting the -Djava.io.tmpdir property in the jvm.options
file to an alternate directory or correcting the Logstash user's permissions.
bendermIBM commented 5 years ago

Also, I'm not sure AdoptJDK 8 has a JIT

printing help takes 156 seconds

logstash --help  156.18s user 0.41s system 98% cpu 2:39.77 total
joransiu commented 5 years ago

Not sure about the libjffi issue w.r.t logstash, but can confirm that OpenJDK 8 with HotSpot VM does not have a JIT on Linux on Z. It was only added in JDK9 or newer. You can consider using OpenJDK 8 with OpenJ9 VM, which will be much more performant: https://adoptopenjdk.net/releases.html?variant=openjdk8&jvmVariant=openj9#s390x_linux

bendermIBM commented 5 years ago

@joransiu that does look to have the JIT thanks

logstash --help  28.25s user 1.18s system 180% cpu 16.286 total

Still same JFFI error w/ J9 VM

logstash -f /etc/logstash/conf.d -l /var/log/logstash --log.level=debug --path.settings=/etc/logstash

Thread.exclusive is deprecated, use Thread::Mutex
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2019-09-27T12:16:07,243][DEBUG][logstash.modules.scaffold] Found module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2019-09-27T12:16:07,246][DEBUG][logstash.plugins.registry] Adding plugin to the registry {:name=>"fb_apache", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0xffffffff975d9559 @directory="/usr/share/logstash/modules/fb_apache/configuration", @module_name="fb_apache", @kibana_version_parts=["6", "0", "0"]>}
[2019-09-27T12:16:07,247][DEBUG][logstash.modules.scaffold] Found module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2019-09-27T12:16:07,247][DEBUG][logstash.plugins.registry] Adding plugin to the registry {:name=>"netflow", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0x4fd3b9dd @directory="/usr/share/logstash/modules/netflow/configuration", @module_name="netflow", @kibana_version_parts=["6", "0", "0"]>}
[2019-09-27T12:16:07,429][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (LoadError) Could not load FFI Provider: (NotImplementedError) FFI not available: java.lang.UnsatisfiedLinkError: could not locate stub library in jar file.  Tried [jni/s390x-Linux/libjffi-1.2.so, /jni/s390x-Linux/libjffi-1.2.so]
    at com.kenai.jffi.internal.StubLoader.getStubLibraryStream(StubLoader.java:412)
    at com.kenai.jffi.internal.StubLoader.loadFromJar(StubLoader.java:355)
    at com.kenai.jffi.internal.StubLoader.load(StubLoader.java:258)
    at com.kenai.jffi.internal.StubLoader.<clinit>(StubLoader.java:449)
    at java.lang.Class.forNameImpl(Native Method)
    at java.lang.Class.forName(Class.java:403)
    at com.kenai.jffi.Init.load(Init.java:68)
    at com.kenai.jffi.Foreign$InstanceHolder.getInstanceHolder(Foreign.java:49)
    at com.kenai.jffi.Foreign$InstanceHolder.<clinit>(Foreign.java:45)
    at com.kenai.jffi.Foreign.getInstance(Foreign.java:103)
    at com.kenai.jffi.Type$Builtin.lookupTypeInfo(Type.java:242)
    at com.kenai.jffi.Type$Builtin.getTypeInfo(Type.java:237)
    at com.kenai.jffi.Type.resolveSize(Type.java:155)
    at com.kenai.jffi.Type.size(Type.java:138)
    at jnr.ffi.provider.jffi.NativeRuntime$TypeDelegate.size(NativeRuntime.java:178)
    at jnr.ffi.provider.AbstractRuntime.<init>(AbstractRuntime.java:48)
    at jnr.ffi.provider.jffi.NativeRuntime.<init>(NativeRuntime.java:57)
    at jnr.ffi.provider.jffi.NativeRuntime.<init>(NativeRuntime.java:41)
    at jnr.ffi.provider.jffi.NativeRuntime$SingletonHolder.<clinit>(NativeRuntime.java:53)
    at jnr.ffi.provider.jffi.NativeRuntime.getInstance(NativeRuntime.java:49)
    at jnr.ffi.provider.jffi.Provider.<init>(Provider.java:29)
    at java.lang.J9VMInternals.newInstanceImpl(Native Method)
    at java.lang.Class.newInstance(Class.java:1848)
    at jnr.ffi.provider.FFIProvider$SystemProviderSingletonHolder.getInstance(FFIProvider.java:68)
    at jnr.ffi.provider.FFIProvider$SystemProviderSingletonHolder.<clinit>(FFIProvider.java:57)
    at jnr.ffi.provider.FFIProvider.getSystemProvider(FFIProvider.java:35)
    at jnr.ffi.Library.loadLibrary(Library.java:114)
    at jnr.posix.POSIXFactory$DefaultLibCProvider$SingletonHolder.<clinit>(POSIXFactory.java:281)
    at jnr.posix.POSIXFactory$DefaultLibCProvider.getLibC(POSIXFactory.java:310)
    at jnr.posix.BaseNativePOSIX.<init>(BaseNativePOSIX.java:38)
    at jnr.posix.LinuxPOSIX.<init>(LinuxPOSIX.java:19)
    at jnr.posix.POSIXFactory.loadLinuxPOSIX(POSIXFactory.java:146)
    at jnr.posix.POSIXFactory.loadNativePOSIX(POSIXFactory.java:124)
    at jnr.posix.POSIXFactory.loadPOSIX(POSIXFactory.java:93)
    at jnr.posix.LazyPOSIX.loadPOSIX(LazyPOSIX.java:38)
    at jnr.posix.LazyPOSIX.posix(LazyPOSIX.java:32)
    at jnr.posix.LazyPOSIX.isNative(LazyPOSIX.java:402)
    at org.jruby.RubyGlobal.initSTDIO(RubyGlobal.java:287)
    at org.jruby.RubyGlobal.createGlobals(RubyGlobal.java:211)
    at org.jruby.Ruby.init(Ruby.java:1260)
    at org.jruby.Ruby.newInstance(Ruby.java:370)
    at org.logstash.Logstash.<init>(Logstash.java:88)
    at org.logstash.Logstash.main(Logstash.java:43)

 See http://jira.codehaus.org/browse/JRUBY-4583

Error accessing temp directory: /tmp
This often occurs because the temp directory has been mounted with NOEXEC or
the Logstash user has insufficient permissions on the directory. Possible
workarounds include setting the -Djava.io.tmpdir property in the jvm.options
file to an alternate directory or correcting the Logstash user's permissions.
joransiu commented 5 years ago

I found this issue: https://github.com/linux-on-ibm-z/docs/issues/18 that sounds related. Could you try the steps there? It suggests the libjffi libraries need to be rolled up into the jar? I'm not sure why...

bendermIBM commented 5 years ago

Hey @joransiu - tried it out.

zip -g jruby-complete-9.1.13.0.jar /usr/local/jffi/build/jni/libjffi-1.2.so

    zip warning: jruby-complete-9.1.13.0.jar not found or empty
  adding: usr/local/jffi/build/jni/libjffi-1.2.so (deflated 70%)

Same error

logstash -f /etc/logstash/conf.d -l /var/log/logstash --log.level=debug --path.settings=/etc/logstash

Thread.exclusive is deprecated, use Thread::Mutex
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2019-09-28T13:46:09,433][DEBUG][logstash.modules.scaffold] Found module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2019-09-28T13:46:09,440][DEBUG][logstash.plugins.registry] Adding plugin to the registry {:name=>"fb_apache", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0xffffffffc0a9e58e @directory="/usr/share/logstash/modules/fb_apache/configuration", @module_name="fb_apache", @kibana_version_parts=["6", "0", "0"]>}
[2019-09-28T13:46:09,441][DEBUG][logstash.modules.scaffold] Found module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2019-09-28T13:46:09,441][DEBUG][logstash.plugins.registry] Adding plugin to the registry {:name=>"netflow", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0xffffffffcaf57fab @directory="/usr/share/logstash/modules/netflow/configuration", @module_name="netflow", @kibana_version_parts=["6", "0", "0"]>}
[2019-09-28T13:46:09,637][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (LoadError) Could not load FFI Provider: (NotImplementedError) FFI not available: java.lang.UnsatisfiedLinkError: could not locate stub library in jar file.  Tried [jni/s390x-Linux/libjffi-1.2.so, /jni/s390x-Linux/libjffi-1.2.so]
    at com.kenai.jffi.internal.StubLoader.getStubLibraryStream(StubLoader.java:412)
    at com.kenai.jffi.internal.StubLoader.loadFromJar(StubLoader.java:355)
    at com.kenai.jffi.internal.StubLoader.load(StubLoader.java:258)
    at com.kenai.jffi.internal.StubLoader.<clinit>(StubLoader.java:449)
    at java.lang.Class.forNameImpl(Native Method)
    at java.lang.Class.forName(Class.java:403)
    at com.kenai.jffi.Init.load(Init.java:68)
    at com.kenai.jffi.Foreign$InstanceHolder.getInstanceHolder(Foreign.java:49)
    at com.kenai.jffi.Foreign$InstanceHolder.<clinit>(Foreign.java:45)
    at com.kenai.jffi.Foreign.getInstance(Foreign.java:103)
    at com.kenai.jffi.Type$Builtin.lookupTypeInfo(Type.java:242)
    at com.kenai.jffi.Type$Builtin.getTypeInfo(Type.java:237)
    at com.kenai.jffi.Type.resolveSize(Type.java:155)
    at com.kenai.jffi.Type.size(Type.java:138)
    at jnr.ffi.provider.jffi.NativeRuntime$TypeDelegate.size(NativeRuntime.java:178)
    at jnr.ffi.provider.AbstractRuntime.<init>(AbstractRuntime.java:48)
    at jnr.ffi.provider.jffi.NativeRuntime.<init>(NativeRuntime.java:57)
    at jnr.ffi.provider.jffi.NativeRuntime.<init>(NativeRuntime.java:41)
    at jnr.ffi.provider.jffi.NativeRuntime$SingletonHolder.<clinit>(NativeRuntime.java:53)
    at jnr.ffi.provider.jffi.NativeRuntime.getInstance(NativeRuntime.java:49)
    at jnr.ffi.provider.jffi.Provider.<init>(Provider.java:29)
    at java.lang.J9VMInternals.newInstanceImpl(Native Method)
    at java.lang.Class.newInstance(Class.java:1848)
    at jnr.ffi.provider.FFIProvider$SystemProviderSingletonHolder.getInstance(FFIProvider.java:68)
    at jnr.ffi.provider.FFIProvider$SystemProviderSingletonHolder.<clinit>(FFIProvider.java:57)
    at jnr.ffi.provider.FFIProvider.getSystemProvider(FFIProvider.java:35)
    at jnr.ffi.Library.loadLibrary(Library.java:114)
    at jnr.posix.POSIXFactory$DefaultLibCProvider$SingletonHolder.<clinit>(POSIXFactory.java:281)
    at jnr.posix.POSIXFactory$DefaultLibCProvider.getLibC(POSIXFactory.java:310)
    at jnr.posix.BaseNativePOSIX.<init>(BaseNativePOSIX.java:38)
    at jnr.posix.LinuxPOSIX.<init>(LinuxPOSIX.java:19)
    at jnr.posix.POSIXFactory.loadLinuxPOSIX(POSIXFactory.java:146)
    at jnr.posix.POSIXFactory.loadNativePOSIX(POSIXFactory.java:124)
    at jnr.posix.POSIXFactory.loadPOSIX(POSIXFactory.java:93)
    at jnr.posix.LazyPOSIX.loadPOSIX(LazyPOSIX.java:38)
    at jnr.posix.LazyPOSIX.posix(LazyPOSIX.java:32)
    at jnr.posix.LazyPOSIX.isNative(LazyPOSIX.java:402)
    at org.jruby.RubyGlobal.initSTDIO(RubyGlobal.java:287)
    at org.jruby.RubyGlobal.createGlobals(RubyGlobal.java:211)
    at org.jruby.Ruby.init(Ruby.java:1260)
    at org.jruby.Ruby.newInstance(Ruby.java:370)
    at org.logstash.Logstash.<init>(Logstash.java:88)
    at org.logstash.Logstash.main(Logstash.java:43)

 See http://jira.codehaus.org/browse/JRUBY-4583

Error accessing temp directory: /tmp
This often occurs because the temp directory has been mounted with NOEXEC or
the Logstash user has insufficient permissions on the directory. Possible
workarounds include setting the -Djava.io.tmpdir property in the jvm.options
file to an alternate directory or correcting the Logstash user's permissions.
bendermIBM commented 5 years ago

Hey @cwsolee I have pinged you the details of the clean machine. To assist in the recreate the command we are using to run logstash is :

logstash -f /etc/logstash/conf.d -l /var/log/logstash --log.level=debug --path.settings=/etc/logstash

I have also sent you the related config files