search

xerial / snappy-java

Snappy compressor/decompressor for Java
Apache License 2.0
1.04k stars 232 forks source link

Caused by: org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] no native library is found for os.name=zOS and os.arch=s390x #232

Open sunz21sunz opened 5 years ago

sunz21sunz commented 5 years ago

Hi Team, I am using version 1.1.7.3 of 'snappy-java' and getting below error:

Caused by: org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] no native library is found for os.name=zOS and os.arch=s390x

JVM version:: IBM J9 VM (build 2.9, JRE 1.8.0 z/OS s390x-64-Bit Compressed References 20190124_408237 (JIT enabled, AOT enabled)

odaira commented 5 years ago

snappy-java currently does not have a native binary for zOS/s390x. I contributed the binary for Linux/s390x, but I do not have access to zOS now. Let me poke around....

sunz21sunz commented 5 years ago

@odaira Did you find anything or any ETA, we have install around next month that requires this change

odaira commented 5 years ago

@sunz21sunz Would you have any contact in IBM (or other company supporting zOS) on your deal? If the deadline is critical, I would reach out to the contact if any. I'm working for snappy-java on a voluntary basis, so I'll try, but I cannot commit myself to any schedule.

By the way, I've got access to a zOS instance. It has nothing installed, so I'll start with installing git.

xerial commented 5 years ago

Another idea is adding a fallback to pure-java Snappy (like aircompressor) if no native library is found.

jamesbroadhead commented 4 years ago

@xerial do you have any benchmarks for a pure-java implementation vs this one?

fosslou commented 4 years ago

@odaira you can get access to a free LinuxOne (Linux on Z) environment in two ways.

  1. The Marist College Linux One Community CLoud https://linuxone.cloud.marist.edu/cloud/#/login
  2. The HyperProtect Virtual Server (Free tier) on the IBM public Cloud https://cloud.ibm.com

Good luck

xerial commented 4 years ago

Added a fallback to pure-java snappy implementation in #244 and this is available since snappy-java 1.1.76

n-marion commented 4 years ago

@xerial , the fallback that you implemented in #244 that is based on aircompressor doesn't support BIGENDIAN platforms 

static {
        ByteOrder order = ByteOrder.nativeOrder();
        if (!order.equals(ByteOrder.LITTLE_ENDIAN)) {
            throw new SnappyError(SnappyErrorCode.UNSUPPORTED_PLATFORM, format("pure-java snappy requires a little endian platform (found %s)", order));
        }
kiwi1969 commented 4 years ago

I came here to report the bug, but @n-marion beat me :) If run on Z/OS Unix shell with snappy-java 1.1.7.6, and use pure-java option on command line :

Exception in thread "main" org.xerial.snappy.SnappyError: [UNSUPPORTED_PLATFORM] pure-java snappy requires a little endian platform (found BIG_ENDIAN) at org.xerial.snappy.pure.UnsafeUtil.(UnsafeUtil.java:38) at org.xerial.snappy.pure.SnappyRawCompressor.writeUncompressedLength(SnappyRawCompressor.java:383) at org.xerial.snappy.pure.SnappyRawCompressor.compress(SnappyRawCompressor.java:89) at org.xerial.snappy.pure.PureJavaSnappy.rawCompress(PureJavaSnappy.java:102) at org.xerial.snappy.Snappy.rawCompress(Snappy.java:424) at org.xerial.snappy.Snappy.compress(Snappy.java:105) at com.equifax.usis.acro.Main.main(Main.java:12)

I really want to use snappy compression within Avro, but this LITTLE_ENDIAN bug in aircompressor is preventing us. [(https://github.com/airlift/aircompressor/blob/master/src/main/java/io/airlift/compress/snappy/UnsafeUtil.java)]

Specifically, it seems that the lines identified by @xerial for LITTLE_ENDIAN should be deleted.

xerial commented 4 years ago

It's not a bug, rather big-endian architecture is not supported in the pure-java fallback version. To use zOS, s390x we need to add a native library for this.

If https://github.com/dockcross/dockcross supports cross-compilation for zOS/s390s, it's the ideal way to support zOS in snappy java because the build will be reproducible, and we don't need to ask IBM team members like @odaira to rebuild native libraries.

kiwi1969 commented 4 years ago

I only referred to it as a bug in pure-java, as it seems the only reason why big endian isn't supported is because there was no effort to reverse the byte order to allow it to work. ie It doesn't seem to be something that is completely unresolvable

This link has examples of methods that could be leveraged to provide the big endian functionality []http://cr.openjdk.java.net/~jmelvin/7130404/jdk/webrev.04/raw_files/new/src/share/classes/sun/security/provider/ByteArrayAccess.java

xerial commented 4 years ago

Personally I don't have time for working on big-endian support by checking every detail of the compressor/decompressor implementation (e.g., where byte order matters).

If there is a docker image of g++ cross compiler for zOS, I'd be happy to included it as a part of the build process of snappy-java. Otherwise, I need to wait for contributions from open source community.

kiwi1969 commented 4 years ago

This link seems to mention a docker image for cross compile of gcc to Z/OS platform (at bottom of blog). [https://blog.the-leviathan.ch/?p=1344] My company blocks the link, so I am unable to validate.

In the meantime, I might play around with Pure Java and try to get a local version that for Big Endian, does the reversebytes alternative that seems to be needed around every UNSAFE.getXXX() or UNSAFE.putxxx invocation

fosslou commented 4 years ago

You should be able to test this on Linux on Z... which you can get a VM on the Marist College LInuxOne Community Cloud here: https://developer.ibm.com/components/ibm-linuxone/gettingstarted/

On Tue, Aug 25, 2020 at 12:26 PM kiwi1969 notifications@github.com wrote:

This link seems to mention a docker image for cross compile of gcc to Z/OS platform (at bottom of blog). [https://blog.the-leviathan.ch/?p=1344] My company blocks the link, so I am unable to validate.

In the meantime, I might play around with Pure Java and try to get a local version that for Big Endian, does the reversebytes alternative that seems to be needed around every UNSAFE.getXXX() or UNSAFE.putxxx invocation

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/xerial/snappy-java/issues/232#issuecomment-680130875, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABHJIVXR2AVESFQLKY55OEDSCPQZTANCNFSM4I7CYR3A .

-- I/T Architect

kiwi1969 commented 4 years ago

@fosslou - I am using Z/OS native, not ZLinux. So far I have a version of libsnappy.so natively compiled on z/os using c89, without any GNU tools, based upon snappy.c implementation here: [https://github.com/yitzikc/snappy-c]. I am yet to add the JNI stubs to allow JNI to work. I am not sure if this actually works properly yet, or if it is 100% compatible with dll that snappy-java uses, so I am thinking I may be better off (and perhaps easier) to attempting an update to pure-java for big endian.

xerial commented 4 years ago

libsnappy.so won't work as snappy-java needs its own extension (libsnappyjava.so). Running make native is the basic step to create snappy-java that should work at your target machine. https://github.com/xerial/snappy-java/blob/master/Makefile#L143

kiwi1969 commented 4 years ago

I have successfully updated snappy.purejava to work on both little and Big Endian, by providing new methods, instead of directly using UNSAFE.getXXX methods for short, int,long datatypes. ie Initial testing looks good.

I would like to note, that the 'fallback' doesn't actually occur when JNI DLL isn't being found. I can only get the pure java implementation to run by forcing it on the command line See below

java -cp "SnappyTest.jar:lib/*" com.equifax.usis.acro.Main
Exception in thread "main" org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] no native library is found for os.name=zOS and os.arch=s390x
    at org.xerial.snappy.SnappyLoader.findNativeLibrary(SnappyLoader.java:361)
    at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:195)
    at org.xerial.snappy.SnappyLoader.loadSnappyApi(SnappyLoader.java:167)
    at org.xerial.snappy.Snappy.init(Snappy.java:69)
    at org.xerial.snappy.Snappy.<clinit>(Snappy.java:46)
    at com.equifax.usis.acro.Main.main(Main.java:12)

java -Dorg.xerial.snappy.purejava=true -cp "SnappyTest.jar:lib/*" com.equifax.usis.acro.Main
Hello snappy-java! Snappy-java is a JNI-based wrapper of Snappy, a fast compresser/decompresser.
Original size = 96
Compressed size = 87
Uncompressed size = 96

My work won't let me attach to your Git repository from this PC, so I will have just pasted the 3 changed modules and my Test driver mainline here below snappytest.zip If someone could incorporate these into a later release, that would solve my issues.

kiwi1969 commented 4 years ago

I managed to create own fork of this project, apply my changes, and submit a pull request for this project to merge in changes from my fork : https://github.com/xerial/snappy-java/pull/254

Hoping this will solve this issue for systems without DLL for native Java (and usually do not have GNU toolsets etc in order to produce it themselves). They will just have to specify "-Dorg.xerial.snappy.purejava=true" to select pure Java.