Native library loading error and version 6 picked by default?

[mchv]

Hi,

I get the following error when trying to compress and then uncompress random bits with 0.8.0 release. Apart from the native library loading error, what seems odd is the input stream decoder seems to detect zstd 0.6.

[info] - zstd should compress the content
[info] Exception encountered when attempting to run a suite with class name: com.gu.contentapi.porter.integration.flexible.ZstdCompressionTest *** ABORTED ***
[info]   java.lang.UnsatisfiedLinkError: com.github.luben.zstd.ZstdInputStreamV06.createDCtx()J
[info]   at com.github.luben.zstd.ZstdInputStreamV06.createDCtx(Native Method)
[info]   at com.github.luben.zstd.ZstdInputStreamV06.<init>(ZstdInputStreamV06.java:57)
[info]   at com.github.luben.zstd.ZstdInputStream.<init>(ZstdInputStream.java:80)
[info]   at com.gu.contentapi.porter.integration.ZstdCompression$.uncompress(Compression.scala:59)
[info]   at com.gu.contentapi.porter.integration.ZstdCompression$.uncompress(Compression.scala:68)
[info]   at com.gu.contentapi.porter.integration.flexible.ZstdCompressionTest$$anonfun$3.apply$mcV$sp(ZstdCompressionTests.scala:17)
[info]   at com.gu.contentapi.porter.integration.flexible.ZstdCompressionTest$$anonfun$3.apply(ZstdCompressionTests.scala:15)
[info]   at com.gu.contentapi.porter.integration.flexible.ZstdCompressionTest$$anonfun$3.apply(ZstdCompressionTests.scala:15)

My code:

import com.github.luben.zstd.ZstdInputStream
import com.github.luben.zstd.ZstdOutputStream

object ZstdCompression {

  def compress(data: Array[Byte]): Array[Byte] = {
    val bos = new ByteArrayOutputStream()
    val out = new ZstdOutputStream(bos)
    out.write(data)
    out.close()
    bos.toByteArray
  }

  def uncompress(buffer: ByteBuffer): ByteBuffer = {
    val bos = new ByteArrayOutputStream(8192)
    val in = new ZstdInputStream(new ByteBufferBackedInputStream(buffer))
    IOUtils.copy(in, bos)
    in.close()
    bos.close()
    ByteBuffer.wrap(bos.toByteArray())
  }

  def uncompress(data: Array[Byte]): Array[Byte] = {
    val buffer = ByteBuffer.wrap(data)
    val byteBuffer = uncompress(buffer)
    val result = new Array[Byte](byteBuffer.remaining)
    byteBuffer.get(result)
    result
  }

}

the test:

import com.gu.contentapi.porter.integration.ZstdCompression._
import org.scalatest.FunSuite
import org.scalatest.Matchers
import scala.util.Random

class ZstdCompressionTest extends FunSuite with Matchers {

  test("zstd should compress the content") {
    val blob = Array.fill(2048) { Random.nextInt(100).toByte }
    compress(blob).length should be < blob.length
  }

  test("ztsd and then unzstd should return the same content") {
    val blob = Array.fill(2048) { Random.nextInt(100).toByte }
    uncompress(compress(blob)) should be(blob)
  }
}

[luben]

Hmm, that's really strange. I will be traveling the weekend but will push a new version based on zstd-1.0 next week (the compression is done, decompression not yet). It will be based on the new streaming API provided by it and will simplify a lot the bindings code.

[luben]

BTW, on what platform you get the errors? It may be a problem with my build/packaging chain. It's strange because I have mostly the same test. What's the result if you build the zstd-jni locally from source?

[mchv]

Mac OS X 64 bits.

$ uname -a
Darwin 32570 15.0.0 Darwin Kernel Version 15.0.0: Sat Sep 19 15:53:46 PDT 2015; root:xnu-3247.10.11~1/RELEASE_X86_64 x86_64

I have been using the provided build binary:

"com.github.luben" % "zstd-jni" % "0.8.0"

[mchv]

@luben I will try to build locally on monday and see if I can reproduce

[shaheenkdr]

@luben , Sir, does this work in Android? Since space is considered a luxury in mobile, this would definitely help many developers. I need to compress a 300 MB file, would be glad to know if it does !

[luben]

@shaheenkdr, it should work on android but have to be re-compiled and packaged. This is good as the distributed JAR contains all types of architectures that will not be needed.

[mchv]

@luben compiling and testing locally works fine. Sorry for the following long log

32570:zstd-jni mchauvin$ sbt
Use of ~/.sbtconfig is deprecated, please migrate global settings to /usr/local/etc/sbtopts
[info] Loading global plugins from /Users/mchauvin/.sbt/0.13/plugins
[info] Loading project definition from /Users/mchauvin/Code/zstd-jni/project
[info] Updating {file:/Users/mchauvin/Code/zstd-jni/project/}zstd-jni-build...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] downloading http://dl.bintray.com/content/joprice/sbt-plugins/com.github.joprice/sbt-jni/scala_2.10/sbt_0.13/0.1.2/jars/sbt-jni.jar ...
[info]  [SUCCESSFUL ] com.github.joprice#sbt-jni;0.1.2!sbt-jni.jar (806ms)
[info] downloading https://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/com.typesafe.sbt/sbt-osgi/scala_2.10/sbt_0.13/0.8.0/jars/sbt-osgi.jar ...
[info]  [SUCCESSFUL ] com.typesafe.sbt#sbt-osgi;0.8.0!sbt-osgi.jar (2815ms)
[info] downloading https://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/de.johoop/jacoco4sbt/scala_2.10/sbt_0.13/2.1.6/jars/jacoco4sbt.jar ...
[info]  [SUCCESSFUL ] de.johoop#jacoco4sbt;2.1.6!jacoco4sbt.jar (1817ms)
[info] downloading https://repo1.maven.org/maven2/com/github/fommil/lion/agent/1.0.0/agent-1.0.0-assembly.jar ...
[info]  [SUCCESSFUL ] com.github.fommil.lion#agent;1.0.0!agent.jar (357ms)
[info] downloading https://repo1.maven.org/maven2/biz/aQute/bnd/bndlib/2.4.0/bndlib-2.4.0.jar ...
[info]  [SUCCESSFUL ] biz.aQute.bnd#bndlib;2.4.0!bndlib.jar (579ms)
[info] downloading https://repo1.maven.org/maven2/org/jacoco/org.jacoco.core/0.7.1.201405082137/org.jacoco.core-0.7.1.201405082137.jar ...
[info]  [SUCCESSFUL ] org.jacoco#org.jacoco.core;0.7.1.201405082137!org.jacoco.core.jar (55ms)
[info] downloading https://repo1.maven.org/maven2/org/jacoco/org.jacoco.report/0.7.1.201405082137/org.jacoco.report-0.7.1.201405082137.jar ...
[info]  [SUCCESSFUL ] org.jacoco#org.jacoco.report;0.7.1.201405082137!org.jacoco.report.jar (63ms)
[info] downloading https://repo1.maven.org/maven2/org/ow2/asm/asm-debug-all/5.0.1/asm-debug-all-5.0.1.jar ...
[info]  [SUCCESSFUL ] org.ow2.asm#asm-debug-all;5.0.1!asm-debug-all.jar (121ms)
[info] Done updating.
[info] Set current project to zstd-jni (in build file:/Users/mchauvin/Code/zstd-jni/)
> compile
[info] Updating {file:/Users/mchauvin/Code/zstd-jni/}zstd-jni...
[info] Resolving org.jacoco#org.jacoco.build;0.7.1.201405082137 ...
[info] downloading https://repo1.maven.org/maven2/org/jacoco/org.jacoco.agent/0.7.1.201405082137/org.jacoco.agent-0.7.1.201405082137.jar ...
[info]  [SUCCESSFUL ] org.jacoco#org.jacoco.agent;0.7.1.201405082137!org.jacoco.agent.jar (107ms)
[info] Done updating.
[info] Compiling 10 Java sources to /Users/mchauvin/Code/zstd-jni/target/classes...
[info] javah -d /Users/mchauvin/Code/zstd-jni/target/classes/include -classpath /Users/mchauvin/Code/zstd-jni/target/classes com.github.luben.zstd.Zstd com.github.luben.zstd.ZstdOutputStream com.github.luben.zstd.ZstdInputStream com.github.luben.zstd.ZstdInputStreamV07 com.github.luben.zstd.ZstdInputStreamV06 com.github.luben.zstd.ZstdInputStreamV05 com.github.luben.zstd.ZstdInputStreamV04
[info] mkdir -p /Users/mchauvin/Code/zstd-jni/target/classes/darwin/x86_64
[info] gcc -shared -fPIC -O3 -I/Users/mchauvin/Code/zstd-jni/target/classes/include -I/usr/include -L/usr/local/include -I/Library/Java/JavaVirtualMachines/jdk1.8.0_45.jdk/Contents/Home/jre/../include -I/Library/Java/JavaVirtualMachines/jdk1.8.0_45.jdk/Contents/Home/jre/../include/darwin -I/Users/mchauvin/Code/zstd-jni/src/main/native -I/Users/mchauvin/Code/zstd-jni/src/main/native/common -std=c99 -Wundef -Wshadow -Wcast-align -Wstrict-prototypes -Wno-unused-variable -msse4 -o /Users/mchauvin/Code/zstd-jni/target/classes/darwin/x86_64/libzstd.dylib /Users/mchauvin/Code/zstd-jni/src/main/native/jni_inputstream_zstd.c /Users/mchauvin/Code/zstd-jni/src/main/native/jni_inputstream_zstd_v04.c /Users/mchauvin/Code/zstd-jni/src/main/native/jni_inputstream_zstd_v05.c /Users/mchauvin/Code/zstd-jni/src/main/native/jni_inputstream_zstd_v06.c /Users/mchauvin/Code/zstd-jni/src/main/native/jni_inputstream_zstd_v07.c /Users/mchauvin/Code/zstd-jni/src/main/native/jni_outputstream_zstd.c /Users/mchauvin/Code/zstd-jni/src/main/native/jni_zdict.c /Users/mchauvin/Code/zstd-jni/src/main/native/jni_zstd.c /Users/mchauvin/Code/zstd-jni/src/main/native/common/entropy_common.c /Users/mchauvin/Code/zstd-jni/src/main/native/common/fse_decompress.c /Users/mchauvin/Code/zstd-jni/src/main/native/common/xxhash.c /Users/mchauvin/Code/zstd-jni/src/main/native/common/zstd_common.c /Users/mchauvin/Code/zstd-jni/src/main/native/compress/fse_compress.c /Users/mchauvin/Code/zstd-jni/src/main/native/compress/huf_compress.c /Users/mchauvin/Code/zstd-jni/src/main/native/compress/zbuff_compress.c /Users/mchauvin/Code/zstd-jni/src/main/native/compress/zstd_compress.c /Users/mchauvin/Code/zstd-jni/src/main/native/decompress/huf_decompress.c /Users/mchauvin/Code/zstd-jni/src/main/native/decompress/zbuff_decompress.c /Users/mchauvin/Code/zstd-jni/src/main/native/decompress/zstd_decompress.c /Users/mchauvin/Code/zstd-jni/src/main/native/dictBuilder/divsufsort.c /Users/mchauvin/Code/zstd-jni/src/main/native/dictBuilder/zdict.c /Users/mchauvin/Code/zstd-jni/src/main/native/legacy/zstd_v04.c /Users/mchauvin/Code/zstd-jni/src/main/native/legacy/zstd_v05.c /Users/mchauvin/Code/zstd-jni/src/main/native/legacy/zstd_v06.c /Users/mchauvin/Code/zstd-jni/src/main/native/legacy/zstd_v07.c
[success] Total time: 15 s, completed 05-Sep-2016 13:42:16
> test
[info] javah -d /Users/mchauvin/Code/zstd-jni/target/classes/include -classpath /Users/mchauvin/Code/zstd-jni/target/classes com.github.luben.zstd.Zstd com.github.luben.zstd.ZstdOutputStream com.github.luben.zstd.ZstdInputStream com.github.luben.zstd.ZstdInputStreamV07 com.github.luben.zstd.ZstdInputStreamV06 com.github.luben.zstd.ZstdInputStreamV05 com.github.luben.zstd.ZstdInputStreamV04
[info] mkdir -p /Users/mchauvin/Code/zstd-jni/target/classes/darwin/x86_64
[info] gcc -shared -fPIC -O3 -I/Users/mchauvin/Code/zstd-jni/target/classes/include -I/usr/include -L/usr/local/include -I/Library/Java/JavaVirtualMachines/jdk1.8.0_45.jdk/Contents/Home/jre/../include -I/Library/Java/JavaVirtualMachines/jdk1.8.0_45.jdk/Contents/Home/jre/../include/darwin -I/Users/mchauvin/Code/zstd-jni/src/main/native -I/Users/mchauvin/Code/zstd-jni/src/main/native/common -std=c99 -Wundef -Wshadow -Wcast-align -Wstrict-prototypes -Wno-unused-variable -msse4 -o /Users/mchauvin/Code/zstd-jni/target/classes/darwin/x86_64/libzstd.dylib /Users/mchauvin/Code/zstd-jni/src/main/native/jni_inputstream_zstd.c /Users/mchauvin/Code/zstd-jni/src/main/native/jni_inputstream_zstd_v04.c /Users/mchauvin/Code/zstd-jni/src/main/native/jni_inputstream_zstd_v05.c /Users/mchauvin/Code/zstd-jni/src/main/native/jni_inputstream_zstd_v06.c /Users/mchauvin/Code/zstd-jni/src/main/native/jni_inputstream_zstd_v07.c /Users/mchauvin/Code/zstd-jni/src/main/native/jni_outputstream_zstd.c /Users/mchauvin/Code/zstd-jni/src/main/native/jni_zdict.c /Users/mchauvin/Code/zstd-jni/src/main/native/jni_zstd.c /Users/mchauvin/Code/zstd-jni/src/main/native/common/entropy_common.c /Users/mchauvin/Code/zstd-jni/src/main/native/common/fse_decompress.c /Users/mchauvin/Code/zstd-jni/src/main/native/common/xxhash.c /Users/mchauvin/Code/zstd-jni/src/main/native/common/zstd_common.c /Users/mchauvin/Code/zstd-jni/src/main/native/compress/fse_compress.c /Users/mchauvin/Code/zstd-jni/src/main/native/compress/huf_compress.c /Users/mchauvin/Code/zstd-jni/src/main/native/compress/zbuff_compress.c /Users/mchauvin/Code/zstd-jni/src/main/native/compress/zstd_compress.c /Users/mchauvin/Code/zstd-jni/src/main/native/decompress/huf_decompress.c /Users/mchauvin/Code/zstd-jni/src/main/native/decompress/zbuff_decompress.c /Users/mchauvin/Code/zstd-jni/src/main/native/decompress/zstd_decompress.c /Users/mchauvin/Code/zstd-jni/src/main/native/dictBuilder/divsufsort.c /Users/mchauvin/Code/zstd-jni/src/main/native/dictBuilder/zdict.c /Users/mchauvin/Code/zstd-jni/src/main/native/legacy/zstd_v04.c /Users/mchauvin/Code/zstd-jni/src/main/native/legacy/zstd_v05.c /Users/mchauvin/Code/zstd-jni/src/main/native/legacy/zstd_v06.c /Users/mchauvin/Code/zstd-jni/src/main/native/legacy/zstd_v07.c
[info] Compiling 2 Scala sources to /Users/mchauvin/Code/zstd-jni/target/test-classes...
[warn] javaOptions will be ignored, fork is set to false

      Compressable data -1
      --
      Compression:        302 MB/s
      Decompression:      834 MB/s
      Compression Ratio:  6.055182768377895

      Compressable data -3
      --
      Compression:        216 MB/s
      Decompression:      718 MB/s
      Compression Ratio:  6.421596065870941

      Compressable data -6
      --
      Compression:        104 MB/s
      Decompression:      804 MB/s
      Compression Ratio:  7.077799527505906

      Compressable data -9
      --
      Compression:        54 MB/s
      Decompression:      824 MB/s
      Compression Ratio:  7.497057877238766

      Streaming at 1
      --
      Compression:        321 MB/s
      Decompression:      752 MB/s
      Compression Ratio:  7.722644230273517

      Streaming at 3
      --
      Compression:        266 MB/s
      Decompression:      706 MB/s
      Compression Ratio:  8.376652841065232

      Streaming at 6
      --
      Compression:        112 MB/s
      Decompression:      735 MB/s
      Compression Ratio:  9.191198126306046

      Streaming at 9
      --
      Compression:        68 MB/s
      Decompression:      796 MB/s
      Compression Ratio:  10.199024621783188

[info] ZstdPerfSpec:
[info] - should be fast for compressable data -1
[info] - should be fast for compressable data -3
[info] - should be fast for compressable data -6
[info] - should be fast for compressable data -9
[info] - should be fast with steaming -1
[info] - should be fast with steaming -3
[info] - should be fast with steaming -6
[info] - should be fast with steaming -9
[info] ZstdSpec:
[info] Zstd
[info] - should should round-trip compression/decompression at level 1
[info] Zstd
[info] - should should round-trip compression/decompression at level 3
[info] Zstd
[info] - should should round-trip compression/decompression at level 6
[info] Zstd
[info] - should should round-trip compression/decompression at level 9
[info] Zstd
[info] - should should round-trip compression/decompression at level 16
[info] Zstd
[info] - should should round-trip using streaming API at level 1
[info] Zstd
[info] - should should round-trip using streaming API at level 3
[info] Zstd
[info] - should should round-trip using streaming API at level 6
[info] Zstd
[info] - should should round-trip using streaming API at level 9
[info] Zstd
[info] - should should round-trip using streaming API at level 16
[info] Zstd
[info] - should should round-trip using streaming API with unfinished chunks at level 1
[info] Zstd
[info] - should should round-trip using streaming API with unfinished chunks at level 3
[info] Zstd
[info] - should should round-trip using streaming API with unfinished chunks at level 6
[info] Zstd
[info] - should should round-trip using streaming API with unfinished chunks at level 9
[info] Zstd
[info] - should should round-trip using streaming API with unfinished chunks at level 16
[info] ZstdInputStream
[info] - should be able to consume files compressed by the zstd binary at level 1
[info] ZstdInputStream
[info] - should be able to consume files compressed by the zstd binary at level 3
[info] ZstdInputStream
[info] - should be able to consume files compressed by the zstd binary at level 6
[info] ZstdInputStream
[info] - should be able to consume files compressed by the zstd binary at level 9
[info] ZstdInputStream
[info] - should be able to consume files compressed by the zstd binary at level 16
[info] ZstdOutputStream
[info] - should produce the same compressed file as zstd binary at level 1
[info] ZstdOutputStream
[info] - should produce the same compressed file as zstd binary at level 3
[info] ZstdOutputStream
[info] - should produce the same compressed file as zstd binary at level 6
[info] ZstdOutputStream
[info] - should produce the same compressed file as zstd binary at level 9
[info] ZstdOutputStream
[info] - should produce the same compressed file as zstd binary at level 16
[info] ZstdInputStream
[info] - should be able to consume files compressed by the zstd binary version 04
[info] ZstdInputStream
[info] - should be able to consume files compressed by the zstd binary version 05
[info] ZstdInputStream
[info] - should be able to consume files compressed by the zstd binary version 06
[info] ZstdInputStream
[info] - should be able to consume files compressed by the zstd binary version 07
[info] ZstdContinuousInputStream
[info] - should be able to consume files compressed by the zstd binary version 04
[info] ZstdContinuousInputStream
[info] - should be able to consume files compressed by the zstd binary version 05
[info] ZstdContinuousInputStream
[info] - should be able to consume files compressed by the zstd binary version 06
[info] ZstdContinuousInputStream
[info] - should be able to consume files compressed by the zstd binary version 07
[info] ScalaCheck
[info] Passed: Total 0, Failed 0, Errors 0, Passed 0
[info] ScalaTest
[info] Run completed in 1 minute, 9 seconds.
[info] Total number of tests run: 41
[info] Suites: completed 2, aborted 0
[info] Tests: succeeded 41, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[info] Passed: Total 41, Failed 0, Errors 0, Passed 41
[success] Total time: 89 s, completed 05-Sep-2016 13:43:47

[shaheenkdr]

@luben Sir , actually I am a college student, but I'd like to build the Android part of this. Some help would be really great . what all should be opted out? After completion, I also want to write an article for beginners to easily use this incredible compression in Android. Kindly help :)

[luben]

@shaheenkdr, I am not too familiar with the Android build system. I guess it will not need any code changes as there is nothing unusual in the code itself. Contributions are welcome.

BTW, I am going to release zstream branch as 1.0.0 really soon. If you start working on the Android build it will be better to use it as a base (instead of master).

[luben]

@mchv, if you publish it locally (sbt publish-local) is your test still failing? It should use the locally published jar.

[shaheenkdr]

@luben Android uses Gradle based build system. So I'll try using your .jar . And will share if there are any issues. By the way, can I get your email , so that I can ping you with you any updates or issues. Also, please do add an example of how to use this to compress file in the description. would be helpful to beginners like me. Thanks.

[luben]

@shaheenkdr , let's move the discussion to https://github.com/luben/zstd-jni/issues/10

I am coming back to github frequently so it's good to communicate here.

Regarding the examples, you can look in the src/test.

[luben]

@mchv , I just pushed zstd-jni-1.0.0 to Maven. It should appear in 2-3 hours in the index. You can try to compile your test with the updated dependency.

[luben]

Closing the issue as v1.0.0 was released and it handles the legacy decompression support in completely different way. Please open a new issue if you still encounter problems using the binary release zstd-jni-1.0.0.