LZ4HC compressor very slow with JNI

[subes]

See test case: https://github.com/subes/invesdwin-context-persistence/blob/master/invesdwin-context-persistence-parent/invesdwin-context-persistence-timeseries/src/test/java/de/invesdwin/context/persistence/timeseries/DatabasePerformanceTest.java

Switch LZ4Streams.setAllowJniCompressor(true); to false to see write rates going from 40k/sec to 4 million per sec.

The switch works like this:

public static LZ4BlockOutputStream newHighLZ4OutputStream(final OutputStream out, final int blockSize,
            final int compressionLevel) {
        if (allowJniCompressor) {
            return new LZ4BlockOutputStream(out, blockSize,
                    //fastestInstance picks jni which flushes too slow
                    LZ4Factory.fastestInstance().highCompressor(compressionLevel),
                    XXHashFactory.fastestInstance().newStreamingHash32(DEFAULT_SEED).asChecksum(), true);
        } else {
            return new LZ4BlockOutputStream(out, blockSize,
                    //fastestInstance picks jni which flushes too slow
                    LZ4Factory.fastestJavaInstance().highCompressor(compressionLevel),
                    XXHashFactory.fastestInstance().newStreamingHash32(DEFAULT_SEED).asChecksum(), true);
        }
    }

I guess JNI is using FS.sync on flush which makes this so slow? The problem does not happen with fast compressor, only high compressor is affected (though required for me due to higher compression which makes decompression faster due to less IO needed).

[subes]

You have to run testTimeSeriesDbPerformance() via junit in the abolve linked test case to reproduce.

[odaira]

Thanks for reporting it. I'll try to run it myself.

[odaira]

What blockSize and compressionLevel did you use? Was your output to a file? On what platform did you run your benchmark?

[subes]

64k, as well as 1MB, Linux (latest ubuntu, though happens for a few releases now), yes to a file. Compression level 99.

    public static final int DEFAULT_COMPRESSION_LEVEL = 99;
    public static final int LARGE_BLOCK_SIZE = Integers
            .checkedCast(new ByteSize(1D, ByteSizeScale.MEGABYTES).getValue(ByteSizeScale.BYTES));
    /*
     * 64KB is default in LZ4OutputStream (1 << 16) though 128K is almost the same speed with a bit better compression
     * on fast compressor
     * 
     * http://java-performance.info/performance-general-compression/
     */
    public static final int DEFAULT_BLOCK_SIZE = Integers
            .checkedCast(new ByteSize(64D, ByteSizeScale.KILOBYTES).getValue(ByteSizeScale.BYTES));
    public static final int DEFAULT_SEED = 0x9747b28c;

[odaira]

Thanks for the information. Which version of lz4-java did you use? Do you have any idea about the original uncompressed size and the size after compression by the high compressor (i.e. the size of your typical input data and how compressible it is)? I could not reproduce the problem with jvm-compressor-benchmark, so I'm trying to figure out the conditions.

[odaira]

OK, I've found that with compression level 99, the JNI version is slower than the Java Unsafe version. I'll investigate the reason. Did you try the default compression level of 9 for the high compressor? In most cases it is as good as and is faster than the maximum compression level, and its JNI version is faster than the Java versions.

[subes]

Yes, this fixes it. Thanks a lot. With Java performance is at 3.5 million writes per second, while JNI now reaches 4.2 million writes per second. Seems this problem was not obvious enough to be spotted easily. :D

Closing the issue.

[odaira]

FYI, up to compression level 9 of the JNI high compressor and all the compression levels of the safe/unsafe Java high compressors use the same algorithm. However, the JNI high compressor of level 10 or above uses quite a different algorithm with large overhead, so it is no surprise that it is slower than the Java equivalent.

Added a comment about it 7147f2185c6beb5e64cd07373dbc105835a8a2dd.