search

geosolutions-it / imageio-ext

Additional plugins and extension for the standard Java ImageIO library
Other
139 stars 80 forks source link

TIFFDeflator blocks GC due to long-held native critical section #292

Open davispw opened 10 months ago

davispw commented 10 months ago

TIFFDeflator calls ZipDeflateCompressor in a single chunk with a large output byte[], which makes a single call to Deflater.deflate(), whose native implementation calls GetPrimitiveArrayCritical to pin memory (openjdk source). This suspends GC until compression finishes, which can take some time, especially for larger images. In a busy multithreaded process, if younggen fills up, this can block all JVM memory allocation, which is very bad. (Explanation: https://shipilev.net/jvm/anatomy-quarks/9-jni-critical-gclocker/)

It looks like this would also be an issue with libdeflate-java: see compressor.c and LibdeflateCompressor.java.

davispw commented 10 months ago

An easy solution would be to use DeflaterOutputStream instead of Deflater, which uses tiny output chunks (and by the way solves #290 and #291, and also avoids an unnecessary jumbo byte[] allocation). Unfortunately the new deflate SPI (#262) is an awkward interface for this.

aaime commented 8 months ago

Did you check if there are performance side effects of using the stream-based approach, rather than working against a preallocated chunk?