search

lz4 / lz4-java

LZ4 compression for Java
Apache License 2.0
1.11k stars 252 forks source link

How to compress file larger than 2GB? #101

Closed darouwan closed 6 years ago

darouwan commented 7 years ago

I am trying to compress file over 2GB size. As we know, both byte array and inputstream have 2^31-1 length limit, which is 2GB. So compressing file over 2GB size in one time seems impossible technically.

I ever tried to compress file in multi parts, where read one big file into some 2G length buffer and compress them respectively. However it is a little difficult to manage them.

So any better idea to achieve my purpose?

odaira commented 7 years ago

Although I have never tried it myself, how about using https://github.com/xerial/larray? I think you can get a direct ByteBuffer from LArray (or LBuffer?) and can give it to lz4-java.

odaira commented 6 years ago

Would you still need help for this?

darouwan commented 6 years ago

No more need for help, thanks!