Kernel panic when decompressing a lz4-compressed initramfs

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Build an initramfs image for Arch Linux using the lz4 compressor via Arch's 
mkinitcpio version 16.
2. Reboot.
3. Kernel panic happens immediately following the bootloader.

What is the expected output? What do you see instead?
We expect to not get a kernel panic. :)

What version of the product are you using? On what operating system?
lz4 version 109-2 (see [1] below)
Arch Linux x86_64 using the 3.12.5 Linux kernel

Please provide any additional information below.
At least one other user and myself are getting kernel panics when attempting to 
use a lz4-compressed initramfs. I have experienced this issue on three Arch 
Linux x86_64 machines (one laptop, one desktop, and one VirtualBox VM). You can 
read the mailing list discussion at [2], below. The maintainer of mkinitcpio 
said to talk with lz4 upstream, hence this bug report. The Git repository for 
mkinitcpio can be found at [3].

I do not know if this is specific to Linux 3.12.5, as I have not tried another 
version of the kernel yet.

[1] https://www.archlinux.org/packages/community/x86_64/lz4/
[2] 
https://mailman.archlinux.org/pipermail/arch-projects/2013-December/004044.html
[3] https://projects.archlinux.org/mkinitcpio.git/

Original issue reported on code.google.com by stjohn.j...@gmail.com on 21 Dec 2013 at 2:36

[GoogleCodeExporter]

The real issue at hand here is that the kernel support was added based on the 
original streaming format. When lz4c dropped the -l flag, it effectively broke 
an existing user. I realize that its also possible for the kernel to update to 
the newer streaming format, but removing the possibility of using the legacy 
format when existing users are still prevalent seems short sighted.

So, this "bug" ends up being more like a feature request. Would you be willing 
to provide the legacy encoder in newer lz4 until the kernel code is consuming 
the newer format?

Original comment by d...@falconindy.com on 21 Dec 2013 at 3:47

[GoogleCodeExporter]

I also got a kernel panic after Syslinux bootloader. I'm using kernel 3.13-rc4.

Original comment by serdotli...@gmail.com on 22 Dec 2013 at 1:28

[GoogleCodeExporter]

It works when the additional and undocumented compression option "-l" (legacy?) 
is used for lz4 in the mkinitcpio.conf. I'm using Kernel 3.12.6 and lz4 109-2.

See: https://bbs.archlinux.org/viewtopic.php?pid=1362909#p1362909

Original comment by thorsten...@gmail.com on 22 Dec 2013 at 2:28

[GoogleCodeExporter]

There may be some confusion here.
lz4 (and lz4c) have always and still support the -l command.
-l is not documented, because it's not supposed to be used by "normal" (command 
line) users.
I was under the expectation that Linux Kernel compression was done by scripts, 
so the user should not need to manipulate directly this command.

Now, if not documenting the -l command is part of the reason for this 
confusion, maybe this choice could be revisited.

Original comment by yann.col...@gmail.com on 22 Dec 2013 at 4:18

[GoogleCodeExporter]

Kernel compression isn't the issue at hand here. Rather, it's initramfs 
creation. I'll point to mkinitcpio, dracut, and initramfs-tools. All of these 
are shell-based programs to create initramfs images and, when combined, cover a 
large number of popular linux distros -- Fedora, Ubuntu, Debian, OpenSUSE, and 
Arch Linux, among them.

I think the major point of confusion here has been that the development of 
kernel support appears to have been independent of this lz4 tools project. It's 
unfortunate that there wasn't collaboration between the two camps. I'm not 
trying to blame anyone or point fingers. It simply means we've found ourselves 
in a less than ideal situation and now we need to address it in a way that, 
hopefully, keeps everyone happy.

To that end, documenting the -l option would be fantastic for the sake of 
giving it some sort of permanence. I realize that some day it should very well 
go away, but unfortunately I suspect you're going to be stuck "maintaining" it 
as long as the kernel can't consume the newer format. I'd like to see the 
warnings that the -l option currently prints as documentation as well, rather 
than runtime warnings.

Original comment by d...@falconindy.com on 22 Dec 2013 at 6:27

[GoogleCodeExporter]

I'll look into it

Original comment by yann.col...@gmail.com on 22 Dec 2013 at 6:32

• Changed state: Accepted
• Added labels: Priority-Low
• Removed labels: Priority-minor

[GoogleCodeExporter]

The following attached file is an early release candidate for r110
which integrates the comments from this thread.

- -l command is documented
    + in the advanced section of help within command line tool
    + in the man page
- associated warning message is only displayed in verbose mode

provided here for tests and comments

Original comment by yann.col...@gmail.com on 22 Dec 2013 at 11:54

Attachments:

• lz4-r110.tar.gz

[GoogleCodeExporter]

Looks good to me.

There's an entirely unrelated segfault due to an invalid use of free at 
lz4cli.c:1252. The victim char* is never pointing to a heap allocation. This is 
simple enough to trigger with 'lz4 infile outfile'.

Original comment by d...@falconindy.com on 23 Dec 2013 at 12:17

[GoogleCodeExporter]

Thanks for the test.

Indeed, I made the wrong assumption that using free() on an invalid pointer was 
harmless.

The version in attached file should correct that.

Original comment by yann.col...@gmail.com on 23 Dec 2013 at 12:32

Attachments:

• lz4-r110.tar.gz

[GoogleCodeExporter]

-l switch documented into r110

Original comment by yann.col...@gmail.com on 30 Dec 2013 at 5:23

• Changed state: Fixed