search

ome / ome-codecs

OME image compression (coder-decoder implementations)
Other
4 stars 13 forks source link

Optimize LZW compression #14

Closed popiel closed 5 years ago

popiel commented 6 years ago

This collection of commits improves LZW compression speed by 5-15%. This is achieved by small improvements to the loop structure and bit shifting/masking behaviours in the compressor, plus providing another API method which can avoid an extra array copy (by returning a partially-filled ByteBuffer instead of copying into a new byte[] of the perfect size).

During development, I was running new and old code side by side in the unit tests and doing speed comparisons; you can see some of that in the commit history, if you're curious. In the final commits, I ripped that out so only the optimized code remains.

This work was motivated by the desire to speed up image format conversions for image files in excess of 50GB. In conjunction with additional changes in bioformats and ome-common-java, I have achieved a geometric to linear time asymptotic improvement, plus another ~30% speed improvement of just the linear time cost. Patches for bioformats and ome-common-java will be forthcoming in the next few days.

sbesson commented 6 years ago

Thanks @popiel for the contribution. We are investigating and will be reviewing various writing performance bottlenecks so this comes at a timely moment. Including in the daily CI builds to verify this does not cause any regression.