search

luben / zstd-jni

JNI binding for Zstd
Other
854 stars 168 forks source link

Problematic frame (JVM Crash) #228

Closed iSilviu-debug closed 2 years ago

iSilviu-debug commented 2 years ago

Hello! I am writing this issue to report a crash of my JVM. I use zstd-jni to compress the chunks of my minecraft server, but in rare moments the JVM crashes and I don't understand what the problem is.

I also tried with openjdk 17 and ran into the same problem. I also tried with v1.5.2-3 and ran into the same problem.

JVM Dumped me this: https://paste.gg/p/anonymous/5d0cc4e5fe184c48b5faeb67de68de65

Thank you!

luben commented 2 years ago

Is there any chance one instance of ZstdOutputStreamNoFinalizer is used concurrently by multiple threads?

BTW, do you have the chunk that crashes the encoder when trying to compress it?

iSilviu-debug commented 2 years ago

Is there any chance one instance of ZstdOutputStreamNoFinalizer is used concurrently by multiple threads?

Yes exactly! The chunk system used is multi-thread and asynchronous! So can it be solved?

Do you have the chunk that crashes the encoder when trying to compress it?

Unfortunately not, also because the chunk, after the restart, is compressed correctly.

luben commented 2 years ago

Is there any chance one instance of ZstdOutputStreamNoFinalizer is used concurrently by multiple threads?

Yes exactly! The chunk system used is multi-thread and asynchronous! So can it be solved?

Unfortunately no. The native library (https://github.com/facebook/zstd) is not internally synchronized and one context (instance of Zstd(Input|Output)Stream) should not be used by multiple threads. From their documentation (https://facebook.github.io/zstd/zstd_manual.html):

  Note 2 : In multi-threaded environments,
         use one different context per thread for parallel execution.

typedef struct ZSTD_CCtx_s ZSTD_CCtx;
iSilviu-debug commented 2 years ago

I understand, thank you very much!

The thing is, this crash only happens once after a few hours, not right away.

iSilviu-debug commented 2 years ago

multi-threading parameters These parameters are only active if multi-threading is enabled (compiled with build macro ZSTD_MULTITHREAD).

  • Otherwise, trying to set any other value than default (0) will be a no-op and return an error.

  • In a situation where it's unknown if the linked library supports multi-threading or not,

  • setting ZSTD_c_nbWorkers to any value >= 1 and consulting the return value provides a quick way to check this property.

  • ZSTD_c_nbWorkers=400,

  • Select how many threads will be spawned to compres in parallel.

  • When nbWorkers >= 1, triggers asynchronous mode when invoking ZSTD_compressStream*() :

  • ZSTD_compressStream*() consumes input and flush output if possible, but immediately gives back control to caller,

  • while compression is performed in parallel, within worker thread(s).

  • (note : a strong exception to this rule is when first invocation of ZSTD_compressStream2() sets ZSTD_e_end :

  • in which case, ZSTD_compressStream2() delegates to ZSTD_compress2(), which is always a blocking call).

  • Default value is 0, aka "single-threaded mode" : no worker is spawned.

It can be useful?

luben commented 2 years ago

It can be useful?

Zstd can use internally multiple worker threads when compressing one file/stream. It is also supported by zstd-jni. If that is your case, then try it. My understanding is that you want to compress multiple independent chunks in parallel - it that case it will not help you.

iSilviu-debug commented 2 years ago

It can be useful?

Zstd can use internally multiple worker threads when compressing one file/stream. It is also supported by zstd-jni. If that is your case, then try it. My understanding is that you want to compress multiple independent chunks in parallel - it that case it will not help you.

I understand, thanks for understanding!