search

hasse69 / rar2fs

FUSE file system for reading RAR archives
https://hasse69.github.io/rar2fs/
GNU General Public License v3.0
279 stars 27 forks source link

rar2fs fails to serve files using unrar-5.6.6+ #100

Closed KyleSanderson closed 5 years ago

KyleSanderson commented 6 years ago

This was a fun one to discover (thought it was Kodi... and then openssh... and then maybe an archive format change... and then reminiscent of the ancient corrupted caching issues...).

Anyways; when building rar2fs with unrar-5.5.8 lighttpd and sshfs rar2fs works no problem. When rar2fs is run against libunrar-5.6.6 and libunrar-5.6.8 rar2fs seems to fail to open any data. I've gotten anywhere from 0mb to 3mb out of the extracted archive so I'm thinking it may be data corruption... I don't have the tools to debug this currently so I'm hoping what was provided helps. This doesn't appear to be a fuse issue as I changed from 2.9.8 to various 3.x versions and it didn't make a difference.

hasse69 commented 6 years ago

Thanks for the issue report. Will look into it. Have not tried v5.6.6 myself so it is not unlikely that a change has been made that causes failures to process some archives.

Also interesting to note here is that you claim to have tested fuse3. I thought the fuse3 API was not backwards compatible with fuse2, hence the reason why both libraries are normally installed on systems that have uplifted to fuse3. The way that rar2fs uses the API definitely is not according to how fuse3 implements it. Are you sure that you did not have both libraries installed and rar2fs picked the older one? As for now, fuse3 is not supported by rar2fs.

KyleSanderson commented 6 years ago

Ahhhhhh you're right about fuse; I do indeed have both libraries present so that's crap for the STR :)

The issue was across a multitude of archives (I do the FS mount option) but I'm not certain if they were all impacted. A vast majority of them were however.

Gentoo only has those versions (5.5.8 and later); I didn't try a manual compilation.

hasse69 commented 6 years ago

Can you please check if Gentoo has by-passed the static linking of libunrar.so? The ldd command on the rar2fs binary should do it. If there is no mentioning of libunrar.so then it is using static linkage (which is preferred due to the simple reason that it is too easy to get compiles done towards one source version but link to another).

KyleSanderson commented 6 years ago 
        linux-vdso.so.1 (0x00007ffe0f3c0000)
        libfuse.so.2 => /usr/lib64/libfuse.so.2 (0x00007eff1ed13000)
        libunrar.so.5 => /usr/lib64/libunrar.so.5 (0x00007eff1eab9000)
        libstdc++.so.6 => /usr/lib/gcc/x86_64-pc-linux-gnu/7.3.0/libstdc++.so.6 (0x00007eff1e6b3000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007eff1e4af000)
        librt.so.1 => /lib64/librt.so.1 (0x00007eff1e2a7000)
        libgcc_s.so.1 => /usr/lib/gcc/x86_64-pc-linux-gnu/7.3.0/libgcc_s.so.1 (0x00007eff1e090000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007eff1de6f000)
        libc.so.6 => /lib64/libc.so.6 (0x00007eff1daa4000)
        libm.so.6 => /lib64/libm.so.6 (0x00007eff1d70b000)
        /lib64/ld-linux-x86-64.so.2 (0x00007eff1ef51000)

I'm guessing this is why it subtly broke for me when Gentoo unmasked the new unrar package.

hasse69 commented 6 years ago

I would try a manual compile first. If that works then Gentoo has messed up with the release and compiled towards a different version of libunrar than what is actually installed on the system. I think the version incompatibility detection works only on major versions. We could change the granularity here but then they have to match exactly which actually should not be needed. But perhaps it is safe to assume that 5.6.x stays compatible but 5.x.x might not.

tfiskgul commented 6 years ago

I use Gentoo and have been running rar2fs-1.27.0 with libunrar 5.6.6 for some time, and it has worked well. Just tried with 5.6.8 and it seems to be smooth sailing. I do remember many weeks back that I had some issue with rar2fs, where it only seemed to return zeroes on read for some files. I think I just recompiled it and restarted it, and the problem was gone. Is that the issue you're having? I did "hexdump | less" and observed zeroes if I remember correctly.

hasse69 commented 6 years ago

@tfiskgul thanks for the input. The problems observed at version mismatch is not very deterministic, that is why I am asking for a manual compile just to out rule such a case.

hasse69 commented 6 years ago

I have now tried a compile of rar2fs against unrar source v5.6.8 and my regression cannot find any issues what so ever. That said it does not mean there is not a problem lurking around. But I need to obtain a sample archive that actually fails to continue the trouble shooting.

Tested on 

3.13.0-74-generic #118~precise1-Ubuntu SMP Fri Dec 18 10:38:55 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
hasse69 commented 6 years ago

Any updates here?

KyleSanderson commented 6 years ago

Sorry I've been on the road for the past couple weeks (this week included). I'll be home next week and can likely reproduce this again as the STR seemed very reliable with certain opened archives.

I'll try what the other Gentoo-er tried and see if it happens again. Could likely be an issue with portage (or me not passing complete rebuild options). Static linking for this specific app may be the best path forward as there was likely ABI (and not API) breakage.

tfiskgul commented 6 years ago

To be clear, mine is not statically linked against librar either, since it shows up using ldd:

$ ldd /usr/bin/rar2fs linux-vdso.so.1 (0x00007ffd4cf11000) libfuse.so.2 => /usr/lib64/libfuse.so.2 (0x00007f1130560000) libunrar.so.5 => /usr/lib64/libunrar.so.5 (0x00007f1130307000) libstdc++.so.6 => /usr/lib/gcc/x86_64-pc-linux-gnu/7.3.0/libstdc++.so.6 (0x00007f112ff02000) librt.so.1 => /lib64/librt.so.1 (0x00007f112fcfa000) libgcc_s.so.1 => /usr/lib/gcc/x86_64-pc-linux-gnu/7.3.0/libgcc_s.so.1 (0x00007f112fae3000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f112f8c3000) libc.so.6 => /lib64/libc.so.6 (0x00007f112f4fb000) libdl.so.2 => /lib64/libdl.so.2 (0x00007f112f2f7000) libm.so.6 => /lib64/libm.so.6 (0x00007f112ef6e000) /lib64/ld-linux-x86-64.so.2 (0x00007f11309b6000)

$ equery b libunrar.so.5

KyleSanderson commented 6 years ago

@tfiskgul yeah excellent but you've confirmed this bug wholesale that there's a legitimate issue with the library and or code.

hasse69 commented 6 years ago

@KyleSanderson

Sorry I've been on the road for the past couple weeks (this week included).

No problem, was only interested to see if we have some progress here.

Static linking for this specific app may be the best path forward as there was likely ABI (and not API) breakage.

That is a possible cause, yes. Static linking is actually the default unless you use --disable-static-unrar when you configure the package. From what I have seen, some distros chose to use this option. This was one of the reasons why I asked for a manual compile.

@tfiskgul According to your system all looks ok but there is a small caveat here which makes this package a bit special and thus ldd and current system installation might not give you the whole picture. Since rar2fs in fact adds a few patches/extensions to the unrar source code, it is what rar2fs was built against that should match what is installed on the system, unless the two versions are ABI compatible of course. So there might be a problem if a distro chose to upgrade libunrar but does not also recompile rar2fs. It was due to this scenario I added static linking of libunrar as default.

KyleSanderson commented 5 years ago

Sorry team, still on the road (now for 3 weeks!!!). I'll try my best to find an hour but it's not looking too hot. Thought it fair to update everyone even though this is noise to the actual (very well distro-related) issue.

hasse69 commented 5 years ago

Thanks for the noise :) @tfiskgul since I am not a Gentoo user myself, would it be possible for you to contact the maintainer for rar2fs and poke around a bit and check if they disable static linking or not (I think they do) and in that case why?

tfiskgul commented 5 years ago

@hasse69 Yes, I will contact the maintainer and see how they perform the linking and the rationale behind it.

hasse69 commented 5 years ago

I am not sure but I might have found one possible source to this issue, at least I did find something that could become a potential problem unless handled properly. It might not be intentional that static linking is not performed.

In this case i checked the OpenSUSE PKGBUILD

build() {
  cd "$srcdir/$pkgname-$pkgver"
  ./configure --prefix=/usr --sbindir=/usr/bin --with-unrar=../unrar
  make
}

At first glance this looks fine, but after also checking the build log I could see this

[   55s] checking for static linking of UnRAR library... fallback
[   55s] checking for ../unrar/libunrar.a... no
[   55s] configure: WARNING: Failed to locate static UnRAR library; falling back to dynamic linking

Now the potential flaw in the build function became obvious. The recipe downloads unrarsrc but it does not build it! So what was picked up was whatever version of libunrar.so that was found in the environment. That might or might not actually be the version the PKGBUILD wanted to use, in this case

_unrarver=5.6.3

which I know for sure supports creating of both the static and the dynamic library.

For this particular distro I think what the build function should become is something like

build() {
  cd "$srcdir/unrar"
  make lib
  cd "$srcdir/$pkgname-$pkgver"
  ./configure --prefix=/usr --sbindir=/usr/bin --with-unrar=../unrar
  make
}

Perhaps Gentoo is suffering from some similar flaw?

KyleSanderson commented 5 years ago

Looking likely; https://gitweb.gentoo.org/repo/gentoo.git/tree/sys-fs/rar2fs/rar2fs-1.27.0.ebuild

# Copyright 1999-2018 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2

EAPI=6

DESCRIPTION="A FUSE based filesystem that can mount one or multiple RAR archive(s)"
HOMEPAGE="https://hasse69.github.io/rar2fs/ https://github.com/hasse69/rar2fs"
SRC_URI="https://github.com/hasse69/${PN}/releases/download/v${PV}/${P}.tar.gz"

LICENSE="GPL-3"
SLOT="0"
KEYWORDS="~amd64 ~x86"
IUSE="debug"

RDEPEND=">=app-arch/unrar-5:=
    sys-fs/fuse:0"
DEPEND="${RDEPEND}"

src_configure() {
    export USER_CFLAGS="${CFLAGS}"

    econf \
        --with-unrar=/usr/include/libunrar \
        --disable-static-unrar \
        $(use_enable debug)
}
hasse69 commented 5 years ago

Right, so here someone decided to actually disable static linkage. Would be interesting to understand the rationale behind that decision. It is perfectly fine to do to so if the RDEPEND works as intended. But does it? Are we certain that whatever is now in /usr/include/libunrar is also in fact what is installed on the running system in e.g. /usr/lib?

hasse69 commented 5 years ago

And for sure, even if --disable-static-unrar was to be removed, it would still fallback to dynamic linking for the same reason the OpenSUSE build does. I really doubt that libunrar.a can be found in /usr/include/libunrar.

KyleSanderson commented 5 years ago

I've filed https://bugs.gentoo.org/674038 but I've confirmed with dynamic linking that this is still a problem with the libunrar-5.6 series and maybe onwards.

hasse69 commented 5 years ago

Checked the bug, and maybe not that important, but it is not a matter of adding static linkage, rather it is about not using --disable-static-unrar.

For the problem with libunrar-5.6.x, you need to provide an example archive that does not work because I am unable to reproduce the issue. Data is read fine here for both compressed and uncompressed archives using 5.6.1 and later, last tried version was 5.6.8.

hasse69 commented 5 years ago

Can we close this for the same reason #103 was closed? There is a link in #103 pointing to an example archive that triggers the UnRAR source v5.6.x version incompatibility.

KyleSanderson commented 5 years ago

Yes thank you for the update. Can you please add a note saying those versions of unrar are broken?

hasse69 commented 5 years ago

Sorry, I am not completely following your request here. There is nothing wrong with these versions of unrar that I am aware of. But they are not ABI compatible with older version which means distros uplifting unrar in rar2fs recipes at compile-time must make sure also to uplift whatever is picked up in run-time. This is not a problem if unrar library is statically linked with rar2fs.

KyleSanderson commented 5 years ago

Yeah, sorry; this was a confusing problem.

In summary (correct me if I'm wrong): unrar has the problem, not rar2fs. It doesn't matter if it's statically linked or not; with certain archive types they just fail on the latest unrar versions. There's no fixing this in rar2fs and it's an upstream problem.

hasse69 commented 5 years ago

Yeah, sorry; this was a confusing problem.

Sure is :)

There is nothing as of yet indicating there is anything wrong with the versions of unrar you are referring to. You need to provide an example archive that fails for you since I (and others) are not able to reproduce.

doronbehar commented 5 years ago

I can provide an example for such an archive but it is rather large (47GB) and it'd be better to share a link to it confidentially. @hasse69 Would you be willing to test this if I'd send you privately a link to such archive?

hasse69 commented 5 years ago

@doronbehar sure, I can give it a try

doronbehar commented 5 years ago

How can I contact you?

hasse69 commented 5 years ago

Just mail me: hans.beckerus AT gmail.com

hasse69 commented 5 years ago

I think we have concluded that this archive is not showing the problem as described in this issue. So we are still missing a pointer to an archive that actually works fine on unrarsrc prior to v5.6.6+ but which does not work using some later versions.

hasse69 commented 5 years ago

Any news here?

KyleSanderson commented 5 years ago

I just tried unrar-5.7.3 and it appears to be working with a couple spot checks. I think this may be resolved (whatever this was).

hasse69 commented 5 years ago

Good to hear, I will close this issue now. Feel free to file a new issue if needed.

KyleSanderson commented 5 years ago

thanks bud :)

KyleSanderson commented 5 years ago

For completeness sake... symptoms changed but I did a manual build (./configure --with-unrar=/usr/include/libunrar5 && make && sudo install rar2fs /usr/local/bin/rar2fs ) and it all works again :) This is 100% a Gentoo build problem so I'll follow up there.

Thanks @hasse69 :-)