Android kernel not booting after using lz4 newer than r92

[GoogleCodeExporter]

What steps will reproduce the problem?
1.Modify Android kernel with LZ4 compression support
2.Use lz4c from this page
3.After building and testing, Android not booting.

What is the expected output? What do you see instead?
It should boot. Obviously.

What version of the product are you using? On what operating system?
Ubuntu 12.04.2 64 bit with lz4c r94.
with lz4demo r91, the Android booted well.

Please provide any additional information below.
I've added LZ4 compression support with the following commits:
https://bitbucket.org/arter97/arter97-cm-i9300/commits/d49015317a1d9e032937d24dc
17f2608d6f3a146
https://bitbucket.org/arter97/arter97-cm-i9300/commits/1ec031e7f83dead209dc5f232
4769eb715b303fb
https://bitbucket.org/arter97/arter97-cm-i9300/commits/0aa5dd8541268f21fb819f1e9
72c47c0df8e7324

I think the new "LZ4_decompress_safe() and LZ4_decompress_fast()" is not 
working properly with Android. Or I have to update the LZ4 source in my kernel.
Please look at the three commits above and tell me anything I should update.

Cheers :D

Original issue reported on code.google.com by qkrwngud...@gmail.com on 10 May 2013 at 12:39

[GoogleCodeExporter]

Alas, I'm not the creator/maintainer of this kernel patch. I'll try to help, 
but in any case, don't forget this is the work of Kyungsik Lee, he's probably 
in best position to help you.

One potential issue is there might be some confusion on the format.

LZ4 compression and decompression functions (such as LZ4_decompress_safe() and 
LZ4_decompress_fast())are designed for direct memory-to-memory applications.

This format is also used for I/O compression too, but in this case, it is 
encapsulated. This is necessary because, in many circumstances, files and 
streams to compress are much larger than allocated memory. Hence, input is cut 
into blocks, and each block is compressed, using LZ4.

This encapsulation format is named "LZ4 Streaming Format", and defined here :
http://fastcompression.blogspot.fr/2013/04/lz4-streaming-format-final.html

But the kernel patch was created before the LZ4 Streaming Format was formally 
defined, and therefore uses the Legacy Streaming format.

In any case, you shouldn't use LZ4_decompress_safe() or LZ4_decompress_fast() 
directly on encapsulated data : you have to decode it, extract the blocks, and 
then you can decode those blocks using the decompression functions.

The logic is present into lz4c.c, function decodeLegacyStream().

Last detail : 
LZ4_decompress_safe() and LZ4_decompress_fast() are only newer names.
The equivalence is :
LZ4_uncompress = LZ4_decompress_fast
LZ4_uncompress_unknownOutputSize = LZ4_decompress_safe

Original comment by yann.col...@gmail.com on 10 May 2013 at 1:07

[GoogleCodeExporter]

Last point, maybe the issue is not in the source code itself :

If you use latest version of LZ4c to compress your kernel, the resulting file 
will use the latest version of the Streaming format. Therefore, it will not be 
decoded by the kernel patch, because the kernel patch expects the Legacy 
streaming format.

In order to compress using the Legacy format, you need to use the hidden option 
-l :
example :

lz4c -l kernel.raw kernel.lz4

In this example, lz4c will use the Legacy streaming format to produce 
kernel.lz4, which will be decodable by the kernel patch.

Rgds

Original comment by yann.col...@gmail.com on 11 May 2013 at 11:30

[GoogleCodeExporter]

I've googled a bit and found a patch that will change scripts/Makefile.lib to 
use lz4c with -l flag.

Although, since it's named "Legacy", I want to figure out how to use the latest 
Streaming format in Linux kernel :(

Is there any new patches? I've tried to contact Kyungsik Lee but I haven't got 
any answers yet :(

Original comment by qkrwngud...@gmail.com on 11 May 2013 at 12:42

[GoogleCodeExporter]

Hi,
I guess the patch you found is:
https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git/commit/?id=b078
915464637215645a2dcf388a0402986f4a41

Like Yann mentioned above, Linux kernel only supports Legacy format so far.
That's why the kernel isn't decompressed when you try to use lz4c which
compresses the kernel with the new Streaming format.

I'm trying to make the Linux kernel support the Streaming format but
it will be on a slow track. By the way I'v got your email and this comment
is enough to explain what you asked, except that how long lz4c will support 
"-l".

Thanks,
Kyungsik

Original comment by raphael....@gmail.com on 13 May 2013 at 1:05

[GoogleCodeExporter]

I close this issue, since I believe the answer of Kyungsik sum it all.
Also : don't expect anything special from the "new streaming format". It's 
meant to support different scenarios, but doesn't bring any great improvement 
for kernel compression.

Original comment by yann.col...@gmail.com on 13 May 2013 at 9:42

• Changed state: Invalid