search

hasse69 / rar2fs

FUSE file system for reading RAR archives
https://hasse69.github.io/rar2fs/
GNU General Public License v3.0
275 stars 27 forks source link

Problems with rar2fs crashing in docker container #104

Closed zimme closed 5 years ago

zimme commented 5 years ago

Hey,

I'm having some issues with rar2fs crashing in a docker container.

I've created a docker image zimme/rar2fs which is based on alpine. I'm using it by bind-mounting my media folders into the container's /source folder and bind-mounting an arbitrary location path/to/rar2fs-media on the host into the container's /destination folder using rshared.

The container runs rar2fs with the following arguments by default

-f --seek-length=1 -o allow_other -o auto_unmount /source /destination

I can browse (cd / ls) path/to/rar2fs-media on the host without rar2fs crashing most of the time, but I've had it crash in this scenario maybe 2 times. When it usually crashes is when I run find on the directory or when I bind-mount the path into a plex container and plex is scanning the folder.

When I run rar2fs in the container via gdb using the following arguments

-d --seek-length=1 -o allow_other -o auto_unmount /source /destination

I get the following log output when it crashes

/ # gdb rar2fs
GNU gdb (GDB) 8.0.1
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-alpine-linux-musl".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from rar2fs...done.
(gdb) run -d --seek-length=1 -o allow_other -o auto_unmount /source /destination
Starting program: /usr/local/bin/rar2fs -d --seek-length=1 -o allow_other -o auto_unmount /source /destination
rar2fs[13]: mounted /destination
rar2fs[13]: mounted /destination
[New LWP 18]
[New LWP 19]
[New LWP 20]
unique: 1, opcode: INIT (26), nodeid: 0, insize: 56, pid: 0
INIT: 7.26
flags=0x001ffffb
max_readahead=0x00020000
   INIT: 7.19
   flags=0x00000010
   max_readahead=0x00020000
   max_write=0x00020000
   max_background=0
   congestion_threshold=0
   unique: 1, success, outsize: 40
unique: 2, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 0
getattr /
   unique: 2, success, outsize: 120
unique: 3, opcode: ACCESS (34), nodeid: 1, insize: 48, pid: 0
   unique: 3, error: -38 (Function not implemented), outsize: 16
unique: 4, opcode: OPENDIR (27), nodeid: 1, insize: 48, pid: 0
opendir flags: 0x8000 /
   opendir[93824994819552] flags: 0x8000 /
   unique: 4, success, outsize: 32
unique: 5, opcode: LOOKUP (1), nodeid: 1, insize: 45, pid: 0
LOOKUP /kidz
getattr /kidz
   NODEID: 2
   unique: 5, success, outsize: 144
unique: 6, opcode: OPENDIR (27), nodeid: 2, insize: 48, pid: 0
opendir flags: 0x38800 /kidz
   opendir[93824994745664] flags: 0x38800 /kidz
   unique: 6, success, outsize: 32
unique: 7, opcode: READDIR (28), nodeid: 2, insize: 80, pid: 0
readdir[93824994745664] from 0
   unique: 7, success, outsize: 144
unique: 8, opcode: READDIR (28), nodeid: 2, insize: 80, pid: 0
   unique: 8, success, outsize: 16
unique: 9, opcode: LOOKUP (1), nodeid: 2, insize: 47, pid: 0
LOOKUP /kidz/movies
getattr /kidz/movies
   NODEID: 3
   unique: 9, success, outsize: 144
unique: 10, opcode: OPENDIR (27), nodeid: 3, insize: 48, pid: 0
opendir flags: 0x38800 /kidz/movies
   opendir[140737345621952] flags: 0x38800 /kidz/movies
   unique: 10, success, outsize: 32
unique: 11, opcode: READDIR (28), nodeid: 3, insize: 80, pid: 0
readdir[140737345621952] from 0
   unique: 11, success, outsize: 2984
unique: 12, opcode: READDIR (28), nodeid: 3, insize: 80, pid: 0
   unique: 12, success, outsize: 16
unique: 13, opcode: LOOKUP (1), nodeid: 3, insize: 94, pid: 0
LOOKUP /kidz/movies/Animated.Kids.SWEDiSH.MOViE.PACK.Part1.720p.BluRay-SB
getattr /kidz/movies/Animated.Kids.SWEDiSH.MOViE.PACK.Part1.720p.BluRay-SB
   NODEID: 4
   unique: 13, success, outsize: 144
unique: 14, opcode: OPENDIR (27), nodeid: 4, insize: 48, pid: 0
opendir flags: 0x38800 /kidz/movies/Animated.Kids.SWEDiSH.MOViE.PACK.Part1.720p.BluRay-SB
   opendir[93824994745600] flags: 0x38800 /kidz/movies/Animated.Kids.SWEDiSH.MOViE.PACK.Part1.720p.BluRay-SB
   unique: 14, success, outsize: 32
unique: 15, opcode: READDIR (28), nodeid: 4, insize: 80, pid: 0
readdir[93824994745600] from 0
   unique: 15, success, outsize: 4112
unique: 16, opcode: READDIR (28), nodeid: 4, insize: 80, pid: 0
   unique: 16, success, outsize: 2728
unique: 17, opcode: READDIR (28), nodeid: 4, insize: 80, pid: 0
   unique: 17, success, outsize: 16
unique: 18, opcode: LOOKUP (1), nodeid: 4, insize: 98, pid: 0
LOOKUP /kidz/movies/Animated.Kids.SWEDiSH.MOViE.PACK.Part1.720p.BluRay-SB/A.Monster.In.Paris.2011.SWEDiSH.720p.BluRay.x264-NORDiCHD
getattr /kidz/movies/Animated.Kids.SWEDiSH.MOViE.PACK.Part1.720p.BluRay-SB/A.Monster.In.Paris.2011.SWEDiSH.720p.BluRay.x264-NORDiCHD
   NODEID: 5
   unique: 18, success, outsize: 144
unique: 19, opcode: OPENDIR (27), nodeid: 5, insize: 48, pid: 0
opendir flags: 0x38800 /kidz/movies/Animated.Kids.SWEDiSH.MOViE.PACK.Part1.720p.BluRay-SB/A.Monster.In.Paris.2011.SWEDiSH.720p.BluRay.x264-NORDiCHD
   opendir[93824994746240] flags: 0x38800 /kidz/movies/Animated.Kids.SWEDiSH.MOViE.PACK.Part1.720p.BluRay-SB/A.Monster.In.Paris.2011.SWEDiSH.720p.BluRay.x264-NORDiCHD
   unique: 19, success, outsize: 32
unique: 20, opcode: READDIR (28), nodeid: 5, insize: 80, pid: 0
readdir[93824994746240] from 0

Thread 4 "rar2fs" received signal SIGSEGV, Segmentation fault.
[Switching to LWP 20]
0x00005555555727ad in VolNameToFirstName(wchar_t const*, wchar_t*, unsigned long, bool) ()
(gdb) backtrace
#0  0x00005555555727ad in VolNameToFirstName(wchar_t const*, wchar_t*, unsigned long, bool) ()
#1  0x0000555555564196 in RARVolNameToFirstName (arch=0x7ffff7d6caa0 "/source/kidz/movies/Animated.Kids.SWEDiSH.MOViE.PACK.Part1.720p.BluRay-SB/A.Monster.In.Paris.2011.SWEDiSH.720p.BluRay.x264-NORDiCHD/nordichd-monster.se.720p.rar",
    oldstylevolume=<optimized out>) at dllext.cpp:251
#2  0x000055555556bffe in __RARVolNameToFirstName (s=0x7ffff7d6caa0 "/source/kidz/movies/Animated.Kids.SWEDiSH.MOViE.PACK.Part1.720p.BluRay-SB/A.Monster.In.Paris.2011.SWEDiSH.720p.BluRay.x264-NORDiCHD/nordichd-monster.se.720p.rar",
    vtype=1) at rar2fs.c:999
#3  0x000055555556ec1f in listrar (path=0x7ffff7d6cb80 "/kidz/movies/Animated.Kids.SWEDiSH.MOViE.PACK.Part1.720p.BluRay-SB/A.Monster.In.Paris.2011.SWEDiSH.720p.BluRay.x264-NORDiCHD", buffer=0x7ffff7fc56b0,
    arch=0x7ffff7fc5400 "/source/kidz/movies/Animated.Kids.SWEDiSH.MOViE.PACK.Part1.720p.BluRay-SB/A.Monster.In.Paris.2011.SWEDiSH.720p.BluRay.x264-NORDiCHD/nordichd-monster.se.720p.rar", first_arch=0x7ffff7fc5558, final=<optimized out>)
    at rar2fs.c:2544
#4  0x000055555556f364 in readdir_scan (dir=0x7ffff7d6cb80 "/kidz/movies/Animated.Kids.SWEDiSH.MOViE.PACK.Part1.720p.BluRay-SB/A.Monster.In.Paris.2011.SWEDiSH.720p.BluRay.x264-NORDiCHD",
    root=0x7ffff7fc55c0 "/source/kidz/movies/Animated.Kids.SWEDiSH.MOViE.PACK.Part1.720p.BluRay-SB/A.Monster.In.Paris.2011.SWEDiSH.720p.BluRay.x264-NORDiCHD", next=0x7ffff7fc56a8, next2=0x7ffff7fc56b0) at rar2fs.c:2932
#5  0x000055555556f789 in rar2_readdir (path=0x7ffff7d6cb80 "/kidz/movies/Animated.Kids.SWEDiSH.MOViE.PACK.Part1.720p.BluRay-SB/A.Monster.In.Paris.2011.SWEDiSH.720p.BluRay.x264-NORDiCHD", buffer=0x5555557b9f00, filler=0x7ffff7b3d02e,
    offset=<optimized out>, fi=<optimized out>) at rar2fs.c:3359
#6  0x00007ffff7b412dc in fuse_fs_readdir () from /usr/lib/libfuse.so.2
#7  0x00007ffff7b41455 in ?? () from /usr/lib/libfuse.so.2
#8  0x00007ffff7b46ad3 in ?? () from /usr/lib/libfuse.so.2
#9  0x00007ffff7b47cb6 in ?? () from /usr/lib/libfuse.so.2
#10 0x00007ffff7b44e25 in ?? () from /usr/lib/libfuse.so.2
#11 0x00007ffff7dbf6c2 in ?? () from /lib/ld-musl-x86_64.so.1
#12 0x0000000000000000 in ?? ()
(gdb) step
Single stepping until exit from function _Z18VolNameToFirstNamePKwPwmb,
which has no line number information.

Thread 4 "rar2fs" received signal SIGSEGV, Segmentation fault.
0x00005555555727ad in VolNameToFirstName(wchar_t const*, wchar_t*, unsigned long, bool) ()
(gdb) backtrace
#0  0x00005555555727ad in VolNameToFirstName(wchar_t const*, wchar_t*, unsigned long, bool) ()
#1  0x0000555555564196 in RARVolNameToFirstName (arch=0x7ffff7d6caa0 "/source/kidz/movies/Animated.Kids.SWEDiSH.MOViE.PACK.Part1.720p.BluRay-SB/A.Monster.In.Paris.2011.SWEDiSH.720p.BluRay.x264-NORDiCHD/nordichd-monster.se.720p.rar",
    oldstylevolume=<optimized out>) at dllext.cpp:251
#2  0x000055555556bffe in __RARVolNameToFirstName (s=0x7ffff7d6caa0 "/source/kidz/movies/Animated.Kids.SWEDiSH.MOViE.PACK.Part1.720p.BluRay-SB/A.Monster.In.Paris.2011.SWEDiSH.720p.BluRay.x264-NORDiCHD/nordichd-monster.se.720p.rar",
    vtype=1) at rar2fs.c:999
#3  0x000055555556ec1f in listrar (path=0x7ffff7d6cb80 "/kidz/movies/Animated.Kids.SWEDiSH.MOViE.PACK.Part1.720p.BluRay-SB/A.Monster.In.Paris.2011.SWEDiSH.720p.BluRay.x264-NORDiCHD", buffer=0x7ffff7fc56b0,
    arch=0x7ffff7fc5400 "/source/kidz/movies/Animated.Kids.SWEDiSH.MOViE.PACK.Part1.720p.BluRay-SB/A.Monster.In.Paris.2011.SWEDiSH.720p.BluRay.x264-NORDiCHD/nordichd-monster.se.720p.rar", first_arch=0x7ffff7fc5558, final=<optimized out>)
    at rar2fs.c:2544
#4  0x000055555556f364 in readdir_scan (dir=0x7ffff7d6cb80 "/kidz/movies/Animated.Kids.SWEDiSH.MOViE.PACK.Part1.720p.BluRay-SB/A.Monster.In.Paris.2011.SWEDiSH.720p.BluRay.x264-NORDiCHD",
    root=0x7ffff7fc55c0 "/source/kidz/movies/Animated.Kids.SWEDiSH.MOViE.PACK.Part1.720p.BluRay-SB/A.Monster.In.Paris.2011.SWEDiSH.720p.BluRay.x264-NORDiCHD", next=0x7ffff7fc56a8, next2=0x7ffff7fc56b0) at rar2fs.c:2932
#5  0x000055555556f789 in rar2_readdir (path=0x7ffff7d6cb80 "/kidz/movies/Animated.Kids.SWEDiSH.MOViE.PACK.Part1.720p.BluRay-SB/A.Monster.In.Paris.2011.SWEDiSH.720p.BluRay.x264-NORDiCHD", buffer=0x5555557b9f00, filler=0x7ffff7b3d02e,
    offset=<optimized out>, fi=<optimized out>) at rar2fs.c:3359
#6  0x00007ffff7b412dc in fuse_fs_readdir () from /usr/lib/libfuse.so.2
#7  0x00007ffff7b41455 in ?? () from /usr/lib/libfuse.so.2
#8  0x00007ffff7b46ad3 in ?? () from /usr/lib/libfuse.so.2
#9  0x00007ffff7b47cb6 in ?? () from /usr/lib/libfuse.so.2
#10 0x00007ffff7b44e25 in ?? () from /usr/lib/libfuse.so.2
#11 0x00007ffff7dbf6c2 in ?? () from /lib/ld-musl-x86_64.so.1
#12 0x0000000000000000 in ?? ()
(gdb) step
Single stepping until exit from function _Z18VolNameToFirstNamePKwPwmb,
which has no line number information.
[LWP 19 exited]
[LWP 18 exited]
[LWP 13 exited]

Program terminated with signal SIGSEGV, Segmentation fault.
The program no longer exists.
(gdb) backtrace
No stack.
(gdb)

I you want to test this yourself you can run the following

docker run -it --rm --device /dev/fuse --cap-add SYS_ADMIN --cap-add SYS_PTRACE --security-opt seccomp=unconfined -v <rar-files-folder>:/source -v /path/to/rar2fs-media:/destination:rshared --entrypoint sh zimme/rar2fs

/ # apk add gdb
/ # gdb rar2fs
(gdb) run -d --seek-length=1 -o allow_other -o auto_unmount /source /destination

Then on the host run the following and it "should" crash rar2fs.

find path/to/rar2fs-media -iname '*somethingthatmatchesafileinrar2fsmedia*'

Versions used in the docker image is rar/unrar 5.6.8 rar2fs 1.27.1

hasse69 commented 5 years ago

Thanks for the issue report. This was a new one really, never seen it before. The chance of me being able to reproduce this in an environment similar to yours is probably zero to none, but looking at the signature of the crash this should be possible to reproduce on any system. At least I hope that is the case. I will try to use what the crash dump reveals and see if this is some obvious problem. The function that crash is within libunrar

wchar* VolNameToFirstName(const wchar *VolName,wchar *FirstName,size_t MaxSize,bool NewNumbering)

My first guess is there is something fishy with the path name. But on the other hand this is a very simple function and my initial test did not reveal any issue what so ever. A few questions I need to have answered here is if you built this version of rar2fs yourself? If you did how did you build it? It is very important that whatever version of UnRAR source you are using at build-time is the same version that is used at run-time. The latter should not be a problem if you built this (both libunrar and rar2fs) yourself since by default libunrar is linked statically, but just to make sure also check this using ldd rar2fs and confirm there is no dynamically linked libunrar.so showing up. Also, if you built rar2fs yourself, please attach your config.log file.

zimme commented 5 years ago

You can look in this Dockerfile to see how I build rar2fs. It's using the latest version (1.27.1) of rar2fs and the latest version (5.6.8) of unrar and it's statically linked, verified by ldd.

hasse69 commented 5 years ago

Ok, if we can confirm rar2fs was built correctly, then you need to add some traces to the function that fails. I cannot reproduce this crash using the path name you are experiencing problems with. In fact, I could not even build libunrar (5.6.8) at all on my platform without modifying the actual makefile due to:

c++  -O2 -Wno-logical-op-parentheses -Wno-switch -Wno-dangling-else -fPIC -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -DRAR_SMP -DRARDLL -c rar.cpp
cc1plus: error: unrecognized command line option "-Wno-logical-op-parentheses"
cc1plus: error: unrecognized command line option "-Wno-dangling-else"

I still would like to see your config.log file. I will get back to you with a patch that can dump some information from the actual function that fails. It is a simple string manipulation function, and input to it looks ok judging from the stack trace you provided.

hasse69 commented 5 years ago

You can try to change the unrar makefile and replace -O2 with -O0 and add the -g flag to CXXFLAGS. Also set STRIP to the empty string. Doing that gave me the following output from gdb after having modifies the function to force a crash (SIGSEGV). Should be possible to see more exactly where it crashes in libunrar.

(gdb) where
#0  0x00000000004227bb in VolNameToFirstName (
    VolName=0x7fffffffaf50 L"/source/kidz/movies/Animated.Kids.SWEDiSH.MOViE.PACK.Part1.720p.BluRay-SB/A.Monster.In.Paris.2011.SWEDiSH.720p.BluRay.x264-NORDiCHD/nordichd-monster.se.720p.rar", 
    FirstName=0x7fffffffaf50 L"/source/kidz/movies/Animated.Kids.SWEDiSH.MOViE.PACK.Part1.720p.BluRay-SB/A.Monster.In.Paris.2011.SWEDiSH.720p.BluRay.x264-NORDiCHD/nordichd-monster.se.720p.rar", MaxSize=2048, NewNumbering=false) at pathfn.cpp:602
#1  0x00000000004141ff in RARVolNameToFirstName (
    arch=0x68e320 "/source/kidz/movies/Animated.Kids.SWEDiSH.MOViE.PACK.Part1.720p.BluRay-SB/A.Monster.In.Paris.2011.SWEDiSH.720p.BluRay.x264-NORDiCHD/nordichd-monster.se.720p.rar", oldstylevolume=false) at dllext.cpp:251
#2  0x000000000041cd06 in main (argc=<optimized out>, argv=<optimized out>) at rar2fs.c:5511

In my case line 602 in pathfn.cpp is where I added a write to address 0 :) So change makefile as below and lets see what your new stack trace will give us.

CXXFLAGS=-O0 -g -Wno-logical-op-parentheses -Wno-switch -Wno-dangling-else
STRIP=
zimme commented 5 years ago

Here are the actual build logs for my docker image https://cloud.docker.com/repository/registry-1.docker.io/zimme/rar2fs/builds/a5851ced-2e61-4e9e-89d0-bb3be61296b8

I'm also wondering if the fact that I'm running on alpine might be a contributing factor as it's not using glibc but instead using musl libc.

I'll try and build unrar with the changes you suggested and see if that will give me some different output

zimme commented 5 years ago

The following is the output when running rar2fs with unrar build using the changes you gave me.

/ # gdb rar2fs
GNU gdb (GDB) 8.0.1
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-alpine-linux-musl".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from rar2fs...done.
(gdb) run -d --seek-length=1 -o allow_other -o auto_unmount /source /destination
Starting program: /usr/local/bin/rar2fs -d --seek-length=1 -o allow_other -o auto_unmount /source /destination
rar2fs[13]: mounted /destination
rar2fs[13]: mounted /destination
[New LWP 18]
[New LWP 19]
[New LWP 20]
unique: 1, opcode: INIT (26), nodeid: 0, insize: 56, pid: 0
INIT: 7.26
flags=0x001ffffb
max_readahead=0x00020000
   INIT: 7.19
   flags=0x00000010
   max_readahead=0x00020000
   max_write=0x00020000
   max_background=0
   congestion_threshold=0
   unique: 1, success, outsize: 40
unique: 2, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 0
getattr /
   unique: 2, success, outsize: 120
unique: 3, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 0
getattr /
   unique: 3, success, outsize: 120
unique: 4, opcode: ACCESS (34), nodeid: 1, insize: 48, pid: 0
   unique: 4, error: -38 (Function not implemented), outsize: 16
unique: 5, opcode: OPENDIR (27), nodeid: 1, insize: 48, pid: 0
opendir flags: 0x18800 /
   opendir[140737345621248] flags: 0x18800 /
   unique: 5, success, outsize: 32
unique: 6, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 0
getattr /
   unique: 6, success, outsize: 120
unique: 7, opcode: READDIR (28), nodeid: 1, insize: 80, pid: 0
readdir[140737345621248] from 0
   unique: 7, success, outsize: 176
[New LWP 21]
unique: 8, opcode: READDIR (28), nodeid: 1, insize: 80, pid: 0
   unique: 8, success, outsize: 16
unique: 10, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 0
getattr /
   unique: 10, success, outsize: 120
unique: 9, opcode: RELEASEDIR (29), nodeid: 1, insize: 64, pid: 0
releasedir[140737345621248] flags: 0x0
   unique: 9, success, outsize: 16
unique: 11, opcode: LOOKUP (1), nodeid: 1, insize: 45, pid: 0
LOOKUP /kidz
getattr /kidz
   NODEID: 2
   unique: 11, success, outsize: 144
unique: 12, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 0
getattr /
   unique: 12, success, outsize: 120
unique: 13, opcode: OPENDIR (27), nodeid: 1, insize: 48, pid: 0
opendir flags: 0x8000 /
   opendir[140737345618784] flags: 0x8000 /
   unique: 13, success, outsize: 32
unique: 14, opcode: LOOKUP (1), nodeid: 1, insize: 45, pid: 0
LOOKUP /kidz
getattr /kidz
   NODEID: 2
   unique: 14, success, outsize: 144
unique: 15, opcode: OPENDIR (27), nodeid: 2, insize: 48, pid: 0
opendir flags: 0x38800 /kidz
   opendir[140737345621280] flags: 0x38800 /kidz
   unique: 15, success, outsize: 32
unique: 16, opcode: READDIR (28), nodeid: 2, insize: 80, pid: 0
readdir[140737345621280] from 0
   unique: 16, success, outsize: 144
unique: 17, opcode: READDIR (28), nodeid: 2, insize: 80, pid: 0
   unique: 17, success, outsize: 16
unique: 18, opcode: LOOKUP (1), nodeid: 2, insize: 47, pid: 0
LOOKUP /kidz/movies
getattr /kidz/movies
   NODEID: 3
   unique: 18, success, outsize: 144
unique: 19, opcode: OPENDIR (27), nodeid: 3, insize: 48, pid: 0
opendir flags: 0x38800 /kidz/movies
   opendir[140737354132160] flags: 0x38800 /kidz/movies
   unique: 19, success, outsize: 32
unique: 20, opcode: READDIR (28), nodeid: 3, insize: 80, pid: 0
readdir[140737354132160] from 0
   unique: 20, success, outsize: 2984
unique: 21, opcode: READDIR (28), nodeid: 3, insize: 80, pid: 0
   unique: 21, success, outsize: 16
unique: 22, opcode: LOOKUP (1), nodeid: 3, insize: 94, pid: 0
LOOKUP /kidz/movies/Animated.Kids.SWEDiSH.MOViE.PACK.Part1.720p.BluRay-SB
getattr /kidz/movies/Animated.Kids.SWEDiSH.MOViE.PACK.Part1.720p.BluRay-SB
   NODEID: 4
   unique: 22, success, outsize: 144
unique: 23, opcode: OPENDIR (27), nodeid: 4, insize: 48, pid: 0
opendir flags: 0x38800 /kidz/movies/Animated.Kids.SWEDiSH.MOViE.PACK.Part1.720p.BluRay-SB
   opendir[93824995504224] flags: 0x38800 /kidz/movies/Animated.Kids.SWEDiSH.MOViE.PACK.Part1.720p.BluRay-SB
   unique: 23, success, outsize: 32
unique: 24, opcode: READDIR (28), nodeid: 4, insize: 80, pid: 0
readdir[93824995504224] from 0
   unique: 24, success, outsize: 4112
unique: 25, opcode: READDIR (28), nodeid: 4, insize: 80, pid: 0
   unique: 25, success, outsize: 2728
unique: 26, opcode: READDIR (28), nodeid: 4, insize: 80, pid: 0
   unique: 26, success, outsize: 16
unique: 27, opcode: LOOKUP (1), nodeid: 4, insize: 98, pid: 0
LOOKUP /kidz/movies/Animated.Kids.SWEDiSH.MOViE.PACK.Part1.720p.BluRay-SB/A.Monster.In.Paris.2011.SWEDiSH.720p.BluRay.x264-NORDiCHD
getattr /kidz/movies/Animated.Kids.SWEDiSH.MOViE.PACK.Part1.720p.BluRay-SB/A.Monster.In.Paris.2011.SWEDiSH.720p.BluRay.x264-NORDiCHD
   NODEID: 5
   unique: 27, success, outsize: 144
unique: 28, opcode: OPENDIR (27), nodeid: 5, insize: 48, pid: 0
opendir flags: 0x38800 /kidz/movies/Animated.Kids.SWEDiSH.MOViE.PACK.Part1.720p.BluRay-SB/A.Monster.In.Paris.2011.SWEDiSH.720p.BluRay.x264-NORDiCHD
   opendir[140737351437184] flags: 0x38800 /kidz/movies/Animated.Kids.SWEDiSH.MOViE.PACK.Part1.720p.BluRay-SB/A.Monster.In.Paris.2011.SWEDiSH.720p.BluRay.x264-NORDiCHD
   unique: 28, success, outsize: 32
unique: 29, opcode: READDIR (28), nodeid: 5, insize: 80, pid: 0
readdir[140737351437184] from 0

Thread 5 "rar2fs" received signal SIGSEGV, Segmentation fault.
[Switching to LWP 21]
0x0000555555575a52 in VolNameToFirstName (VolName=<error reading variable: Cannot access memory at address 0x7ffff7f91a18>, FirstName=<error reading variable: Cannot access memory at address 0x7ffff7f91a10>,
    MaxSize=<error reading variable: Cannot access memory at address 0x7ffff7f91a08>, NewNumbering=<error reading variable: Cannot access memory at address 0x7ffff7f91a04>) at pathfn.cpp:600
600 pathfn.cpp: No such file or directory.
(gdb) backtrace
#0  0x0000555555575a52 in VolNameToFirstName (VolName=<error reading variable: Cannot access memory at address 0x7ffff7f91a18>, FirstName=<error reading variable: Cannot access memory at address 0x7ffff7f91a10>,
    MaxSize=<error reading variable: Cannot access memory at address 0x7ffff7f91a08>, NewNumbering=<error reading variable: Cannot access memory at address 0x7ffff7f91a04>) at pathfn.cpp:600
#1  0x0000555555567476 in RARVolNameToFirstName (arch=0x7ffff7d6cc60 "/source/kidz/movies/Animated.Kids.SWEDiSH.MOViE.PACK.Part1.720p.BluRay-SB/A.Monster.In.Paris.2011.SWEDiSH.720p.BluRay.x264-NORDiCHD/nordichd-monster.se.720p.rar",
    oldstylevolume=<optimized out>) at dllext.cpp:251
#2  0x000055555556f2de in __RARVolNameToFirstName (s=0x7ffff7d6cc60 "/source/kidz/movies/Animated.Kids.SWEDiSH.MOViE.PACK.Part1.720p.BluRay-SB/A.Monster.In.Paris.2011.SWEDiSH.720p.BluRay.x264-NORDiCHD/nordichd-monster.se.720p.rar",
    vtype=1) at rar2fs.c:999
#3  0x0000555555571eff in listrar (path=0x7ffff7d6cbc0 "/kidz/movies/Animated.Kids.SWEDiSH.MOViE.PACK.Part1.720p.BluRay-SB/A.Monster.In.Paris.2011.SWEDiSH.720p.BluRay.x264-NORDiCHD", buffer=0x7ffff7fae6b0,
    arch=0x7ffff7fae400 "/source/kidz/movies/Animated.Kids.SWEDiSH.MOViE.PACK.Part1.720p.BluRay-SB/A.Monster.In.Paris.2011.SWEDiSH.720p.BluRay.x264-NORDiCHD/nordichd-monster.se.720p.rar", first_arch=0x7ffff7fae558, final=<optimized out>)
    at rar2fs.c:2544
#4  0x0000555555572644 in readdir_scan (dir=0x7ffff7d6cbc0 "/kidz/movies/Animated.Kids.SWEDiSH.MOViE.PACK.Part1.720p.BluRay-SB/A.Monster.In.Paris.2011.SWEDiSH.720p.BluRay.x264-NORDiCHD",
    root=0x7ffff7fae5c0 "/source/kidz/movies/Animated.Kids.SWEDiSH.MOViE.PACK.Part1.720p.BluRay-SB/A.Monster.In.Paris.2011.SWEDiSH.720p.BluRay.x264-NORDiCHD", next=0x7ffff7fae6a8, next2=0x7ffff7fae6b0) at rar2fs.c:2932
#5  0x0000555555572a69 in rar2_readdir (path=0x7ffff7d6cbc0 "/kidz/movies/Animated.Kids.SWEDiSH.MOViE.PACK.Part1.720p.BluRay-SB/A.Monster.In.Paris.2011.SWEDiSH.720p.BluRay.x264-NORDiCHD", buffer=0x7ffff77e0e80, filler=0x7ffff7b3d02e,
    offset=<optimized out>, fi=<optimized out>) at rar2fs.c:3359
#6  0x00007ffff7b412dc in fuse_fs_readdir () from /usr/lib/libfuse.so.2
#7  0x00007ffff7b41455 in ?? () from /usr/lib/libfuse.so.2
#8  0x00007ffff7b46ad3 in ?? () from /usr/lib/libfuse.so.2
#9  0x00007ffff7b47cb6 in ?? () from /usr/lib/libfuse.so.2
#10 0x00007ffff7b44e25 in ?? () from /usr/lib/libfuse.so.2
#11 0x00007ffff7dbf6c2 in ?? () from /lib/ld-musl-x86_64.so.1
#12 0x0000000000000000 in ?? ()
(gdb) step

Thread 5 "rar2fs" received signal SIGSEGV, Segmentation fault.
0x0000555555575a52 in VolNameToFirstName (VolName=<error reading variable: Cannot access memory at address 0x7ffff7f91a18>, FirstName=<error reading variable: Cannot access memory at address 0x7ffff7f91a10>,
    MaxSize=<error reading variable: Cannot access memory at address 0x7ffff7f91a08>, NewNumbering=<error reading variable: Cannot access memory at address 0x7ffff7f91a04>) at pathfn.cpp:600
600 in pathfn.cpp
(gdb) backtrace
#0  0x0000555555575a52 in VolNameToFirstName (VolName=<error reading variable: Cannot access memory at address 0x7ffff7f91a18>, FirstName=<error reading variable: Cannot access memory at address 0x7ffff7f91a10>,
    MaxSize=<error reading variable: Cannot access memory at address 0x7ffff7f91a08>, NewNumbering=<error reading variable: Cannot access memory at address 0x7ffff7f91a04>) at pathfn.cpp:600
#1  0x0000555555567476 in RARVolNameToFirstName (arch=0x7ffff7d6cc60 "/source/kidz/movies/Animated.Kids.SWEDiSH.MOViE.PACK.Part1.720p.BluRay-SB/A.Monster.In.Paris.2011.SWEDiSH.720p.BluRay.x264-NORDiCHD/nordichd-monster.se.720p.rar",
    oldstylevolume=<optimized out>) at dllext.cpp:251
#2  0x000055555556f2de in __RARVolNameToFirstName (s=0x7ffff7d6cc60 "/source/kidz/movies/Animated.Kids.SWEDiSH.MOViE.PACK.Part1.720p.BluRay-SB/A.Monster.In.Paris.2011.SWEDiSH.720p.BluRay.x264-NORDiCHD/nordichd-monster.se.720p.rar",
    vtype=1) at rar2fs.c:999
#3  0x0000555555571eff in listrar (path=0x7ffff7d6cbc0 "/kidz/movies/Animated.Kids.SWEDiSH.MOViE.PACK.Part1.720p.BluRay-SB/A.Monster.In.Paris.2011.SWEDiSH.720p.BluRay.x264-NORDiCHD", buffer=0x7ffff7fae6b0,
    arch=0x7ffff7fae400 "/source/kidz/movies/Animated.Kids.SWEDiSH.MOViE.PACK.Part1.720p.BluRay-SB/A.Monster.In.Paris.2011.SWEDiSH.720p.BluRay.x264-NORDiCHD/nordichd-monster.se.720p.rar", first_arch=0x7ffff7fae558, final=<optimized out>)
    at rar2fs.c:2544
#4  0x0000555555572644 in readdir_scan (dir=0x7ffff7d6cbc0 "/kidz/movies/Animated.Kids.SWEDiSH.MOViE.PACK.Part1.720p.BluRay-SB/A.Monster.In.Paris.2011.SWEDiSH.720p.BluRay.x264-NORDiCHD",
    root=0x7ffff7fae5c0 "/source/kidz/movies/Animated.Kids.SWEDiSH.MOViE.PACK.Part1.720p.BluRay-SB/A.Monster.In.Paris.2011.SWEDiSH.720p.BluRay.x264-NORDiCHD", next=0x7ffff7fae6a8, next2=0x7ffff7fae6b0) at rar2fs.c:2932
#5  0x0000555555572a69 in rar2_readdir (path=0x7ffff7d6cbc0 "/kidz/movies/Animated.Kids.SWEDiSH.MOViE.PACK.Part1.720p.BluRay-SB/A.Monster.In.Paris.2011.SWEDiSH.720p.BluRay.x264-NORDiCHD", buffer=0x7ffff77e0e80, filler=0x7ffff7b3d02e,
    offset=<optimized out>, fi=<optimized out>) at rar2fs.c:3359
#6  0x00007ffff7b412dc in fuse_fs_readdir () from /usr/lib/libfuse.so.2
#7  0x00007ffff7b41455 in ?? () from /usr/lib/libfuse.so.2
#8  0x00007ffff7b46ad3 in ?? () from /usr/lib/libfuse.so.2
#9  0x00007ffff7b47cb6 in ?? () from /usr/lib/libfuse.so.2
#10 0x00007ffff7b44e25 in ?? () from /usr/lib/libfuse.so.2
#11 0x00007ffff7dbf6c2 in ?? () from /lib/ld-musl-x86_64.so.1
#12 0x0000000000000000 in ?? ()
(gdb) step
[LWP 21 exited]
[LWP 19 exited]
[LWP 18 exited]
[LWP 13 exited]

Program terminated with signal SIGSEGV, Segmentation fault.
The program no longer exists.
(gdb) backtrace
No stack.
(gdb)
hasse69 commented 5 years ago

From what I can tell you have a stack issue here. Is it possible your default stack size in the container is too small? Can you increase the stack size temporarily to see if that might be the case?

If we look at the stack walk you attached frame 1 looks ok

#1  0x0000555555567476 in RARVolNameToFirstName (arch=0x7ffff7d6cc60 "/source/kidz/movies/Animated.Kids.SWEDiSH.MOViE.PACK.Part1.720p.BluRay-SB/A.Monster.In.Paris.2011.SWEDiSH.720p.BluRay.x264-NORDiCHD/nordichd-monster.se.720p.rar",
    oldstylevolume=<optimized out>) at dllext.cpp:251

The only action we make in this function is to allocate a wide character array on the stack (4k) before calling VolNameToFirstName in libunrar. But gdb fails to pick up the arguments and points to the first line in the function (line 600) which often indicates some issue in the function prologue.

#0  0x0000555555575a52 in VolNameToFirstName (VolName=<error reading variable: Cannot access memory at address 0x7ffff7f91a18>, FirstName=<error reading variable: Cannot access memory at address 0x7ffff7f91a10>,
    MaxSize=<error reading variable: Cannot access memory at address 0x7ffff7f91a08>, NewNumbering=<error reading variable: Cannot access memory at address 0x7ffff7f91a04>) at pathfn.cpp:600

You could also try to patch dllext.cpp and instead of allocating space on the stack, use malloc() to allocate space from the heap. Remember wchar is 8 bytes so you need to multiply NM by sizeof(wchar). But this would only isolate the problem, not fix it. It is just a matter of time before it will crash somewhere else. And oh. if you take the malloc route, do not forget to free the memory after it has been used.

hasse69 commented 5 years ago

I think you can let gcc handle the dynamic allocation for you.

If neither STACK_CHECK_BUILTIN nor STACK_CHECK_STATIC_BUILTIN is defined, GCC will change its allocation strategy for large objects if the option -fstack-check is specified: they will always be allocated dynamically if their size exceeds STACK_CHECK_MAX_VAR_SIZE bytes.

I believe you can play with the -fstack-check compiler option to see if your problem is related to stack overflow or not. I assume you need to add the option to both the unrar makefile as well as adding it to Makefile.am and the AM_CFLAGS + AM_CXXFLAGS. I also see now that stack protection is forcibly turned off by rar2fs, I honestly have no idea why, but you can also remove -fno-stack-protector from AM_CFLAGS and give it another shot. You most likely need to re-generate the Makefile after this change using autoreconf -fi and run the configure script again.

hasse69 commented 5 years ago

Using -fstack-check seems like a bit of a blunt tool to me. And -fstack-protect is more for protecting the stack from attacks, nothing we should be worried about here really and it results in increased footprint and overhead which would explain why I chose to not turn it on (it is off by default on most systems but might be enabled on some gcc distributions).

I think the easiest way to determine if this is stack related or not is to use limit or ulimit to temporarily change maximum stack size and see if the problem persists or not. If that helps, we can use -fstack-usage to check if there are any functions in rar2fs that could be made less stack consuming but the unrar library is out of reach I am afraid so it would only do so much to limit the stack pressure. This is more about setting up the OS in a way that it can handle the requests thrown at it which in turn is very system specific and might require tweaking of the stack size limit on some, or not at all on others. For a file system like rar2fs it is also very depending on use case and layout.

hasse69 commented 5 years ago

Another thing you should try if stack size is a dead end is to use dynamic linking. This can easily be forced by using the --disable-static-unrar configure option. Simply to see if it is related in anyway.

zimme commented 5 years ago

I've set the stack size to 32000, was 8192, using the --ulimit flag of docker run and still get the same error, I'll try and use dynamic linking of unrar and see if that changes anything.

hasse69 commented 5 years ago

Ok, that was a rather huge setback I am afraid :( I know very little about docker containers and how to change their maximum stack size. The signature of the crash certainly points to a stack problem. I cannot see anything actually wrong in the code, and even less reproduce what you see. If dynamic linking does not help the malloc approach should be tested as well.

zimme commented 5 years ago

I modified my docker file to build and run on debian:stable-slim instead of alpine and with that it builds and runs without a problem so there seems to be a problem with using alpine (at least in a container) with rar2fs.

The ldd output when running on debian seems to tell that it's dynamically linking some more libraries than when on alpine, and I'm guessing it's glibc vs musl related.

In both cases unrar is statically linked.

I'll try and play around some more with this and I'll try another alpine based that has glibc and see if that makes any difference.

hasse69 commented 5 years ago

Did you read this? https://wiki.musl-libc.org/functional-differences-from-glibc.html

I still believe this is stack size related but my guess is that is the default stack size per thread that is the culprit here rather than the ulimit. The default process stack size of 8192k should more than enough. But it is likely that musl default thread stack size is smaller than what glibc provides and then that would explain what you see. It should also hint that moving to a glibc based container would also work. Lets continue the investigation, we might end up in having to update rar2fs to increase stack size on some systems.

EDIT: On my system setting stack size through ulimit seems to reflect fine to maximum thread size as well. Question is if it works the same on musl? Can you apply the below patch just to see what the stack size becomes on musl, before and after ulimit is changed?

diff --git a/rar2fs.c b/rar2fs.c
index a16daf9..96ef0fe 100644
--- a/rar2fs.c
+++ b/rar2fs.c
@@ -5022,6 +5022,12 @@ struct work_task_data {
  ****************************************************************************/
 static void *work_task(void *data)
 {
+pthread_attr_t foo;
+size_t stacksize;
+        pthread_getattr_np(pthread_self(), &foo);
+pthread_attr_getstacksize(&foo, &stacksize);
+printf("stack size for thread is %zu bytes \n", stacksize);
+
         struct work_task_data *wdt = (struct work_task_data *)data;
         wdt->status = wdt->mt ? fuse_loop_mt(wdt->fuse) : fuse_loop(wdt->fuse);
         wdt->work_task_exited = 1;
zimme commented 5 years ago

I'll have a look at this tonight :+1:

hasse69 commented 5 years ago

This seems related https://github.com/libuv/libuv/issues/1507

zimme commented 5 years ago

Nice find!

This is a bit out of my depth but I'll be happy to test any changes you suggest in the source or system config stuff

hasse69 commented 5 years ago

I still think you should stick to the plan and test the outcome of the patch I posted.

I am not fully convinced about how this was solved in libuv, I would like to avoid playing libc here and instead fix the potential problems where we can actually see they in fact exist. As far as I know, currently musl is where we have an issue. But I would like to get it confirmed that the stack size on musl is not fully adhering to RLIMIT_STACK which seems to be the case for Native POSIX Thread Library (NPTL) that is used on Linux. I have not seen any reports on rar2fs not working on OS X, so until then I would rather keep it as is. There is an implementation problem here as well. It is not rar2fs that creates the threads in question. It is the FUSE framework. In single threaded mode (-s) I believe it should work to set the stack size explicitly on musl from the .init callback function since that is the one and only thread that will serve your file system requests. But in multi-threaded mode it is a completely different story. Single threaded mode is not enabled by default. Can you also check if gcc sets some internal definition to identify this being a musl based platform? The simplest way to do that is

gcc -dM -E - < /dev/null

In any case, if we can determine that the thread stack size on musl based platforms must be increased, then a proper fix must be applied to FUSE, not rar2fs.

hasse69 commented 5 years ago

Please note that I changed the patch above to use a non-portable function in glibc to read the current stack attributes. If this function is not implemented by musl there is no simple way to check what the stack size is set to in current executing thread.

zimme commented 5 years ago

I'll try the patch and see what the stack size is for the different scenarios.

I found this just now in the 2008-02-08 release.

  • If the "FUSE_THREAD_STACK" environment is set, initialize the stack size of threads by this value. Patch by Florin Malita

https://github.com/libfuse/libfuse/blob/fuse_2_9_3/ChangeLog

zimme commented 5 years ago

With the patch and the default thread stack size, the code seems to be crashing before the logging the patch added, or something's not working with the logging.

With a "normal" rar2fs and FUSE_THREAD_STACK=2048000 exported everything seems to work as expected though.

When running the patched version with FUSE_THREAD_STACK=2048000 it works but I don't seem to get any of the new logging.

hasse69 commented 5 years ago

Nice work! You need to run rar2fs in the foreground using -f (or -d) to see the log (stdout), which should appear only once right in the beginning. But I think we can leave the patch behind since you have found the solution already. Funny, it was sitting there in front of us all the time, my bad really. Should have checked the fuse commit history as soon as I suspected a stack issue. Anyway, I think we can close this issue, agree?

zimme commented 5 years ago

I'm ok with closing this.

However, maybe adding to the docs/readme about using the env var on systems that are using musl. We should maybe also check if uclibc has the same quirk

hasse69 commented 5 years ago

I have invited you to become a project collaborator, as a collaborator you have permission to update the wiki. The FAQ section seems like a perfect spot for adding information about musl and possibly also uclibc. I used uclibc a long time back on a small embedded system and did not hit any major issues back then,

hasse69 commented 5 years ago

There is also an alternative option we can consider here. We can add a mount option to rar2fs that can be used to force the stack size by letting is call e.g.

        setenv("FUSE_THREAD_STACK", stacksize, 1);

In that way you can chose to set the stack size / instance rather than globally using an environment variable. It is also possible to set it temporarily using the following syntax:

$ FUSE_THREAD_STACK=2048000 rar2fs ...

Setting the last argument to setenv to 1 means it will override whatever is set in _FUSE_THREADSTACK currently. Cannot decide whether to add this feature or not :)

zimme commented 5 years ago

TBH, it seems like a thread-stacksize option would belong as a fuse option just like -o allow_other.

I believe the env var approach is good enough for now.

hasse69 commented 5 years ago

I agree. There was an attempt to add an option like that already back in 2005 but it was dropped for some reason. https://sourceforge.net/p/fuse/mailman/message/7408097/

zimme commented 5 years ago

Maybe it's worth trying to breathe life in that now that we seem to have a use case for it other than what was put forth back then.

zimme commented 5 years ago

but for me personally it doesn't really matter as I'm gonna run rar2fs inside a docker container that doesn't do anything else.

Also working on making it a docker volume plugin once I get a normal rar2fs container working :+1:

hasse69 commented 5 years ago

Maybe it's worth trying to breathe life in that now that we seem to have a use case for it other than what was put forth back then.

I am not too sure about that. The FUSE_THREAD_STACK patch was added back in 2008. For some reason that was favored over a mount option. You can argue if that was a good choice or not but they must have had their reasons behind it unless the thread from 2005 was completely forgotten about,

zimme commented 5 years ago

Found this.

2) thread_stack (-o thread_stack=N): minimum stack allocated for each thread. Library option, previously system default (8MB starts being an issue with a couple hundred concurrent requests).

Yes, this has been brought up a number of times. I remember suggesting the use of environment variables to control thread attributes, but I can't find the message any more.

Adding an option is more or less OK, but I'd prefer an env variable approach, because it seems very unlikely, that the user would want to tweak the stack size of threads.

Whole thread here, http://fuse.996288.n3.nabble.com/PATCH-0-4-FUSE-tunables-td4524.html.

But all in all, let's let this one go as we have a solution that works good enough for now and I'll update the FAQ section of the Wiki with some info regarding this tonight or this weekend. I'll leave the issue open until the Wiki has been updated.

zimme commented 5 years ago

Thanks for all the help, it's been really interesting to dig into this :+1:

hasse69 commented 5 years ago

Glad I could help, but you found the solution, not me ;)

zimme commented 5 years ago

Wiki has been updated. Feel free to change wording etc.

https://github.com/hasse69/rar2fs/wiki#q-210-why-is-rar2fs-crashing-when-running-on-musl-libc-based-systems-such-as-alpine-linux

Closing this.

hasse69 commented 5 years ago

Looks good. Thank you!