search

ulikunitz / xz

Pure golang package for reading and writing xz-compressed files
Other
484 stars 45 forks source link

Cannot decompress some archives #8

Closed djmdjm closed 8 years ago

djmdjm commented 8 years ago

Some xz archives fail part way during decompression. Quite a few of the Linux kernel releases fall into this category.

You can reproduce it via:

[djm@demiurge ~]$ wget -q https://www.kernel.org/pub/linux/kernel/v3.0/linux-3.13.tar.xz                                                                    
[djm@demiurge ~]$ xzcat linux-3.13.tar.xz | wc -c
549816320
[djm@demiurge ~]$ gxz -dc linux-3.13.tar.xz | wc -c
201330688
[djm@demiurge ~]$ echo $?

Note that the truncation is silent - no error is written to stderr and the exit status is 0. The problem isn't in cmd/xz - I noticed it first using the library directly.

ulikunitz commented 8 years ago

Thank you for reporting the bug. The cause is that the xz file consists of three concatenated xz streams. You can verify it by doing xxd linux-3.13.tar.xz | grep 7zXZ. So far I haven't encountered such multi-stream files, but I have to admit that my biggest test file is 100 Megabyte large.

As a workaround you could just run another xz reader on the same file, because the streams are immediately following each other. Note that the spec allows gaps between streams and if I remember correctly they must be set to zero.

The API will require now some work. Multistream xz files must of course be supported in the default case. But I would also like to support a single xz stream reader for an embedded use case.

I may not be able to work on it next week, but I will definitely deal with it in the week after.

ulikunitz commented 8 years ago

The Reader in the dev branch supporrs now multiple xz streams. See TestReaderMultipleStreams. The API has been changed.

djmdjm commented 8 years ago

Thanks! Do you have any idea when this feature will make it into a release?

ulikunitz commented 8 years ago

My plan is to do it this week.

Damien Miller notifications@github.com schrieb am Mo., 4. Juli 2016 01:32:

Thanks! Do you have any idea when this feature will make it into a release?

— You are receiving this because you modified the open/close state.

Reply to this email directly, view it on GitHub https://github.com/ulikunitz/xz/issues/8#issuecomment-230182113, or mute the thread https://github.com/notifications/unsubscribe/AIEZFUNtc2rV9bi0AnB6Qzbr-QWdfjtuks5qSEaMgaJpZM4I5InO .

ulikunitz commented 8 years ago

The release v0.5 is now out. Please note that the API has changed.