enquiry of compressor/decompressor

[lsching17]

1. In a multithreaded environment, what is better way for perfromance?

   a. share a factory between multiple threads, create compressor/decompressor in each thread b. share a compressor/decompressor between multiple threads c. do not share anything, create everything ad-hoc

2. What is the behaviour of safe decompressor/fast decompressor if the output size is larger than the specified decompressedLength/allocated_buffer_size?

   an exception is thrown or memory corruption occur?

[lsching17]

1. Do the output buffer need to be "clean" before each operation. i.e. can i reuse buffer for every compress/uncompress operation?

[broneill]

Javadocs indicate that compressors and decompressors are thread safe. Looking at the code I see no synchronization, and so no state is shared or re-used. I would expect to see the same performance no matter what you do. Measure it and go for the simplest thing.

[lsching17]

1. What is the behaviour of safe decompressor/fast decompressor if the output size is larger than the specified decompressedLength/allocated_buffer_size?

   an exception is thrown or memory corruption occur?

2. Do the output buffer need to be "clean" before each operation. i.e. can i reuse buffer for every compress/uncompress operation?

On 05/07/2014 08:10 AM, broneill wrote:

Javadocs indicate that compressors and decompressors are thread safe. Looking at the code I see no synchronization, and so no state is shared or re-used. I would expect to see the same performance no matter what you do. Measure it and go for the simplest thing.

— Reply to this email directly or view it on GitHub https://github.com/jpountz/lz4-java/issues/41#issuecomment-48075049.

[broneill]

Why not try it out? It's not that hard to write some test code.

[lsching17]

Other user may have similar enquiry about thread safety, scalability, error handling, buffer reuse..

if it is available in the documentation, it should benefit other user.

Furthermore, for unclean buffer: it is hard to write test scripts to prove the code work for EVERY unclean buffer, the only possible way for user is to study documentation or examine the related code himself (and he must be a C-JNI expert).

On 06/07/2014 07:04 AM, broneill wrote:

Why not try it out? It's not that hard to write some test code.

— Reply to this email directly or view it on GitHub https://github.com/jpountz/lz4-java/issues/41#issuecomment-48098636.

[broneill]

Good points. There are however, a few assumptions regarding the safety of Java APIs. Although the use of JNI can lead to problems, any memory corruption caused by the API is a bug. Documenting that checks are in place is just a promise anyhow, because the implementation might still be defective. The only way to be convinced of the safety is to look at the tests and look at the code.

[jpountz]

• all APIs are thread-safe
• if the output buffer is not large enough, an exception is supposed to be thrown. If not, that is a bug. :-)
• the output buffer doesn't need to be clean before compressing or decompressing.

[lsching17]

Thank you very much for all your help.

But why there is a need for "safe decompressor"? The fast decompressor throw exception when buffer is not large enough

[jpountz]

The additional guarantees of the safe decompressor are not about the output buffer but the input buffer: it allows you to specify the number of bytes that it is allowed to read in order to decompress data.

[lsching17]

thank a lot!