lz4frame should correctly handle skipped frames

[GoogleCodeExporter]

As Suggested by Sergey Cherepanov :

lz4frame should correctly test for, inform about and handle skipped frames.

Original issue reported on code.google.com by yann.col...@gmail.com on 29 Dec 2014 at 8:12

[GoogleCodeExporter]

Here is my opinion how skipped frames should work.
99.9% of lz4 library users will not use skipped frames.Regular user (like me) 
just want to have simple interface to lz4 library without any complications. 
Hence, mainstream library interface should silently skip data from skippable 
frame.
Writing utility like lz4 that only compress/uncompress data on current most 
recent version of library should be no brainer. No checking on skipped frames. 
I also would rpefer not to check for legacy formats.

That to say, there will be 0.1% of users who actively will want and use skipped 
frames with their own data. Lz4 library should provide full support for them.
Lz4 library should provide ability to register handlers for skipped frames. 
Selection of handler shoudl be based on magic number in frame header. Handlers 
of skipped frame shoudl get input data and put processed result into output 
stream.
Lz4 library shoudl came with simplest "uncompressed" handler built-in. 
Uncompressed handler will put data from skipped frame into output without 
processing. It should be extremely easy to register "uncompressed" handler, 
with one call to library.

In short:
- Mainstream users should not be really aware or do any actions for skippable 
frames.
- Data from skippable frames should be silently skipped unless explicit actions 
from user.
- Library shoudl allow tor egister handlers for skippable frames
- Handlers receive data from skippable frame, and put data into output stream
- Library come with built-in "uncompressed" handler, that just copy data from 
skippable frame into output
- Registering "uncompressed" handler is an easy call to library.

P.S.
I am using lz4 library in my communication tools, and it is amazing. It is a 
first software that scale up to all hyperthreaded cores linearly! I can see lz4 
compression rate for single thread 220-280MB/sec, and it does scale up to 
number of huperthreaded cores, both on my PC and on servers.
I did quite a lot of coding of parallel multithreaded lz4 comrpession, if 
anybody wants I can creae some kind of "lz4p" utility or modify lz4 to do 
multithreaded compression. No need to multithread uncompression - it is fast 
enough!

Original comment by ser...@cherepanov.org on 9 Mar 2015 at 4:09

[GoogleCodeExporter]

Thanks Sergey.
I guess I'll start by taking good care of the perceived majority of users, 
which in your opinion, do not need to care about skippable frames.

I've a few ideas on how to handle the situation for those who would prefer to 
access those frames, but let's keep that for a later release. It's usually a 
bad idea to over-engineer a solution while the basics are not yet settled.

Thanks for your comment on lz4 multi-thread performance.
Indeed, a multi-threaded version of lz4 utility is one of the most important 
update remaining to complete. If you believe you can contribute, feel free to 
fork and branch the source code on github, and experiment with your own 
modifications. Merging could be done later on when you feel it's ready.

Best Regards

Original comment by yann.col...@gmail.com on 9 Mar 2015 at 5:29

[GoogleCodeExporter]

A first attempt at supporting skippable frames has been committed into dev 
branch :
https://github.com/Cyan4973/lz4/tree/dev

It is complemented by a few more test cases within frametests.

Regards

Original comment by yann.col...@gmail.com on 16 Mar 2015 at 4:55

[GoogleCodeExporter]

Integrated into r128

Original comment by yann.col...@gmail.com on 31 Mar 2015 at 1:37

• Changed state: Fixed