Closed Nischi85 closed 2 years ago
Unfortunately there is very little to work on here. Never seen anything similar before. That the file system seems to be gracefully unmounted is really strange and should only happen if the FUSE main loop exit for some reason. What version of FUSE is this btw? The INTERRUPT operations look unfamiliar to me. So there are no other traces of problems in e.g. the syslog?
rar2fs v1.29.5 (DLL version 8) FUSE library version: 2.9.9 using FUSE kernel interface version 7.19 Using unrar 6.12
Okay so this is from the syslog(unraid) from the first attempt that errored out:
Sep 28 12:20:10 unraid rar2fs[21005]: Got signal 11, faulty address is 0x152e12f4660e, from 0x411879 Sep 28 12:20:10 unraid rar2fs[21005]: rar2fs() [0x411879] Sep 28 12:20:10 unraid rar2fs[21005]: /lib64/libc.so.6(+0x3a6a0) [0x152f4aec46a0] Sep 28 12:20:10 unraid rar2fs[21005]: /lib64/libc.so.6(+0x1664ab) [0x152f4aff04ab] Sep 28 12:20:10 unraid rar2fs[21005]: rar2fs() [0x4119e7] Sep 28 12:20:10 unraid rar2fs[21005]: rar2fs() [0x411b80] Sep 28 12:20:10 unraid rar2fs[21005]: rar2fs() [0x411b6f] Sep 28 12:20:10 unraid rar2fs[21005]: rar2fs() [0x411b6f] Sep 28 12:20:10 unraid rar2fs[21005]: rar2fs() [0x411b6f] Sep 28 12:20:10 unraid rar2fs[21005]: rar2fs() [0x411b6f] Sep 28 12:20:10 unraid rar2fs[21005]: rar2fs() [0x411b6f] Sep 28 12:20:10 unraid rar2fs[21005]: rar2fs() [0x411b6f] Sep 28 12:20:10 unraid rar2fs[21005]: rar2fs() [0x411b6f] Sep 28 12:20:10 unraid rar2fs[21005]: rar2fs() [0x411b6f] Sep 28 12:20:10 unraid rar2fs[21005]: rar2fs() [0x411b6f] Sep 28 12:20:10 unraid rar2fs[21005]: rar2fs() [0x411b6f] Sep 28 12:20:10 unraid rar2fs[21005]: rar2fs() [0x411b6f] Sep 28 12:20:10 unraid rar2fs[21005]: rar2fs() [0x411b6f] Sep 28 12:20:10 unraid rar2fs[21005]: rar2fs() [0x411b6f] Sep 28 12:20:10 unraid rar2fs[21005]: rar2fs() [0x411b6f] Sep 28 12:20:10 unraid rar2fs[21005]: rar2fs() [0x411b6f] Sep 28 12:20:10 unraid rar2fs[21005]: rar2fs() [0x411b6f] Sep 28 12:20:10 unraid rar2fs[21005]: rar2fs() [0x411b6f] Sep 28 12:20:10 unraid rar2fs[21005]: rar2fs() [0x411b6f] Sep 28 12:20:10 unraid rar2fs[21005]: rar2fs() [0x411b6f] Sep 28 12:20:10 unraid rar2fs[21005]: rar2fs() [0x411b6f] Sep 28 12:20:10 unraid rar2fs[21005]: rar2fs() [0x411b6f] Sep 28 12:20:10 unraid rar2fs[21005]: rar2fs() [0x411b6f] Sep 28 12:20:10 unraid rar2fs[21005]: rar2fs() [0x411b6f] Sep 28 12:20:10 unraid rar2fs[21005]: rar2fs() [0x411b6f] Sep 28 12:20:10 unraid rar2fs[20998]: unmounted /mnt/user/rar2fs_level1
And this is from the second attempt which just flat out quit without any error(dont mind the eth0 renaming, its just me stoping/pausing the emby docker container):
Sep 28 12:26:54 unraid rar2fs[6743]: mounted /mnt/user/rar2fs_level1 Sep 28 12:27:01 unraid emhttpd: cmd: /usr/local/emhttp/plugins/user.scripts/backgroundScript.sh /tmp/user.scripts/tmpScripts/rar2fs mount/script Sep 28 12:27:01 unraid rar2fs[6879]: mounted /mnt/user/rar2fs Sep 28 12:27:01 unraid rofs-filtered[6892]: rofs-filtered 1.7: Starting up. Using source: /mnt/user/rar2fs and config: /etc/filter1.rc Sep 28 12:27:01 unraid rofs-filtered[6897]: rofs-filtered 1.7: Starting up. Using source: /mnt/user/share/ful and config: /etc/filter1.rc Sep 28 12:27:20 unraid kernel: eth0: renamed from veth5943514 Sep 28 12:29:51 unraid rar2fs[6743]: unmounted /mnt/user/rar2fs_level1 Sep 28 12:31:16 unraid kernel: veth5943514: renamed from eth0
The last one is a mystery. I have no answer to why the file system is actively unmounted. That is nothing rar2fs would ever attempt by itself as I have already explained.
The first one is a crash though. Please reconfigure/recompile rar2fs with --enable-debug flag set and run it through gdb to get a proper trap when the crash (signal 11) occurs so that we can dump the stack trace with symbol information.
Okay, I'll read up on how to do that. As of now I'm using the following script to create my binaries.
WORKDIR=mktemp -d
&& cd $WORKDIR
wget https://www.rarlab.com/rar/unrarsrc-6.1.4.tar.gz tar zxvf unrarsrc-6.1.4.tar.gz cd unrar make && sudo make install make lib && sudo make install-lib cd ..
wget https://github.com/hasse69/rar2fs/releases/download/v1.29.5/rar2fs-1.29.5.tar.gz tar zxvf rar2fs-1.29.5.tar.gz cd rar2fs-1.29.5 ./configure --with-unrar=../unrar --with-unrar-lib=/usr/lib/ make && sudo make install
rm -rf $WORKDIR
Try to not use the --with-unrar-lib option so that you are guaranteed to compile and link statically to the correct version of unrarsrc.
What does the output of
ldd <path to your built rar2fs>
look like?
Also note that if you run from gdb, run the compiled binary (from the src directory) and not the installed version of rar2fs since the latter will be completely stripped from any symbols.
linux-vdso.so.1 (0x00007fff0d5b4000)
libfuse.so.2 => /lib64/libfuse.so.2 (0x000014bd21a48000)
libdl.so.2 => /lib64/libdl.so.2 (0x000014bd21a43000)
librt.so.1 => /lib64/librt.so.1 (0x000014bd21a3e000)
libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x000014bd2181b000)
libm.so.6 => /lib64/libm.so.6 (0x000014bd21735000)
libgcc_s.so.1 => /usr/lib64/libgcc_s.so.1 (0x000014bd21719000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x000014bd21714000)
libc.so.6 => /lib64/libc.so.6 (0x000014bd2151d000)
/lib64/ld-linux-x86-64.so.2 (0x000014bd21a97000)
Rest will come tomorrow.
Okay sorry for the delay.
I just ran my usual test but with new binaries compiled without the --with-unrar-lib options as you said. I'm not too well versed in gdb tho. But I will make an effort to learn if that is what it takes.
It crashed somewhat gracefully again. When starting the scan df is as this: (note that rar2fs is not mounted in emby docker, only the 2 rofs-filtered instances)
rar2fs 22690549372 17955481232 4733171076 80% /mnt/user/rar2fs_level1 rar2fs 22690549372 17955481232 4733171076 80% /mnt/user/rar2fs rofs-filtered 22690549372 17955481232 4733171076 80% /mnt/user/rofs rofs-filtered 22456118340 17858346360 4597771980 80% /mnt/user/rofs_ful
When it crashes df ends up like this:
df: /mnt/user/rar2fs: Transport endpoint is not connected df: /mnt/user/rofs: Transport endpoint is not connected rofs-filtered 22456118340 17858346360 4597771980 80% /mnt/user/rofs_ful
syslog:
Oct 1 11:56:22 unraid rar2fs[3617]: Got signal 11, faulty address is 0x1481b80cf6f1, from 0x411879 Oct 1 11:56:22 unraid rar2fs[3617]: rar2fs() [0x411879] Oct 1 11:56:22 unraid rar2fs[3617]: /lib64/libc.so.6(+0x3a6a0) [0x1480fd7236a0] Oct 1 11:56:22 unraid rar2fs[3617]: /lib64/libc.so.6(+0x1664ab) [0x1480fd84f4ab] Oct 1 11:56:22 unraid rar2fs[3617]: rar2fs() [0x4119e7] Oct 1 11:56:22 unraid rar2fs[3617]: rar2fs() [0x411b80] Oct 1 11:56:22 unraid rar2fs[3617]: rar2fs() [0x411b6f] Oct 1 11:56:22 unraid rar2fs[3617]: rar2fs() [0x411b6f] Oct 1 11:56:22 unraid rar2fs[3617]: rar2fs() [0x411b6f] Oct 1 11:56:22 unraid rar2fs[3617]: rar2fs() [0x411b6f] Oct 1 11:56:22 unraid rar2fs[3617]: rar2fs() [0x411b6f] Oct 1 11:56:22 unraid rar2fs[3617]: rar2fs() [0x411b6f] Oct 1 11:56:22 unraid rar2fs[3617]: rar2fs() [0x411b6f] Oct 1 11:56:22 unraid rar2fs[3617]: rar2fs() [0x411b6f] Oct 1 11:56:22 unraid rar2fs[3617]: rar2fs() [0x411b6f] Oct 1 11:56:22 unraid rar2fs[3617]: rar2fs() [0x411b6f] Oct 1 11:56:22 unraid rar2fs[3617]: rar2fs() [0x411b6f] Oct 1 11:56:22 unraid rar2fs[3617]: rar2fs() [0x411b6f] Oct 1 11:56:22 unraid rar2fs[3617]: rar2fs() [0x411b6f] Oct 1 11:56:22 unraid rar2fs[3617]: rar2fs() [0x411b6f] Oct 1 11:56:22 unraid rar2fs[3617]: rar2fs() [0x411b6f] Oct 1 11:56:22 unraid rar2fs[3617]: rar2fs() [0x411b6f] Oct 1 11:56:22 unraid rar2fs[3617]: rar2fs() [0x411b6f] Oct 1 11:56:22 unraid rar2fs[3617]: rar2fs() [0x411b6f] Oct 1 11:56:22 unraid rar2fs[3617]: rar2fs() [0x411b6f] Oct 1 11:56:22 unraid rar2fs[3617]: rar2fs() [0x411b6f] Oct 1 11:56:22 unraid rar2fs[3617]: rar2fs() [0x411b6f] Oct 1 11:56:22 unraid rar2fs[3617]: rar2fs() [0x411b6f] Oct 1 11:56:22 unraid rar2fs[3617]: rar2fs() [0x411b6f] Oct 1 11:56:22 unraid rar2fs[3617]: rar2fs() [0x411b6f] Oct 1 11:56:22 unraid rar2fs[3321]: unmounted /mnt/user/rar2fs_level1
rar2fs debug:
read[22699508220432] 32739 bytes from 1015083 flags: 0x8000 read[22699508220432] 32739 bytes from 1015083 unique: 380044, success, outsize: 32755 unique: 380046, opcode: LOOKUP (1), nodeid: 762, insize: 45, pid: 4590 LOOKUP /Movies/Saint.Laurent.2014.LIMITED.1080p.BluRay.x264-USURY/Subs getattr /Movies/Saint.Laurent.2014.LIMITED.1080p.BluRay.x264-USURY/Subs NODEID: 7659 unique: 380046, success, outsize: 144 unique: 380048, opcode: FLUSH (25), nodeid: 7664, insize: 64, pid: 15020 flush[22699508220432] unique: 380049, opcode: INTERRUPT (36), nodeid: 0, insize: 48, pid: 0 INTERRUPT: 380048 unique: 380048, success, outsize: 16 unique: 380050, opcode: RELEASE (18), nodeid: 7664, insize: 64, pid: 0 release[22699508220432] flags: 0x8000 unique: 380050, success, outsize: 16 rar2fs[3321]: unmounted /mnt/user/rar2fs_level1
Yes signature looks the same. But looking at it from the bright side is that it seems pretty easy to reproduce. Simply google how to run a binary through gdb and you should be good to go. There are not that many steps involved to get a nice stack trace.
thank you for bearing with me.
I ran the gdb. got some more info I guess?
flush[23455891494592] unique: 376828, success, outsize: 16 unique: 376829, opcode: INTERRUPT (36), nodeid: 0, insize: 48, pid: 0 INTERRUPT: 376828 unique: 376830, opcode: RELEASE (18), nodeid: 4395, insize: 64, pid: 0 unique: 376829, error: -11 (Resource temporarily unavailable), outsize: 16 release[23455891494592] flags: 0x8000 unique: 376830, success, outsize: 16 [Thread 0x1555548db6c0 (LWP 23122) exited] [Thread 0x155554d216c0 (LWP 21993) exited] [Thread 0x155554afe6c0 (LWP 21994) exited] [Thread 0x155554f446c0 (LWP 21991) exited] rar2fs[21987]: unmounted /mnt/user/rar2fs_level1 [Inferior 1 (process 21987) exited normally] (gdb) bt No stack.
Not really what we need. You need to make sure to run rar2fs through gdb and mount with the -f flag (-d should work as well) so that process is forced to execute in the foreground. That will allow for gdb to catch the signal 11 and you would be able to manually print the stack trace.
I think that is what you did but in this case you did not capture the crash, this was something else.
Okay sorry. I ran this command:
gdb --args /usr/local/bin/rar2fs -d --exclude=.lock --seek-length=1 --relatime -d -o auto_cache -o direct_io -o max_write=131072 -o splice_read -o splice_write -o splice_move -o allow_other -o nonempty -o umask=0000 -o uid=0 -o gid=0 /mnt/user/share/fin /mnt/user/rar2fs_level1
and got this:
flush[23455958719024] unique: 438628, success, outsize: 16 unique: 438629, opcode: INTERRUPT (36), nodeid: 0, insize: 48, pid: 0 INTERRUPT: 438628 unique: 438630, opcode: RELEASE (18), nodeid: 5193, insize: 64, pid: 0 unique: 438629, error: -11 (Resource temporarily unavailable), outsize: 16 release[23455958719024] flags: 0x8000 unique: 438630, success, outsize: 16 [Thread 0x1555544006c0 (LWP 6710) exited] [Thread 0x155554afe6c0 (LWP 27522) exited] [Thread 0x155554d216c0 (LWP 27521) exited] [Thread 0x15553ffff6c0 (LWP 6711) exited] [Thread 0x1555546b86c0 (LWP 4331) exited] [Thread 0x1555548db6c0 (LWP 30238) exited] [Thread 0x155554f446c0 (LWP 27519) exited] rar2fs[27513]: unmounted /mnt/user/rar2fs_level1 [Inferior 1 (process 27513) exited normally] (gdb) bt No stack.
Hmm maybe something else is crashing first? I'm not sure...
Yes, it is not this process that crash since it simply terminates without catching any signals.
I will try to test some more but I'm traveling for a few days and won't be able to post something new until the weekend.
No worries. Take your time. I have not seen any reports yet of similar problems which hopefully means that this is not a problem affecting a whole lot of users.
What you need to do is to run both instances of rar2fs through gdb so that you trap the crash in the correct instance since it seems only one of them is in fact crashing. Let's try to find the root cause of the crash first, then we can check if the abnormal unmount still happens. It is certainly a bit strange but you are also running a version of FUSE that I personally have never tried before. Maybe some other user(s) can report if they also use this version and then without problems.
Okay so I tried again, by running both instances of rar2fs in gdb as described before. I enter the --args command, and follow it by typing "run" in gdb, I hope that is correct.
I fixed another error I had in some of my rar-archives by setting --seek-length=0 . It seems to have cleared out some of the errors I've gotten for files in the emby log, and also made stacking fuse systems hold for longer. Instead of randomly unmounting rar2fs in 1-5 minutes as before. It seems to be more stable and hold for 30-60 minutes.
Here is an excerpt of the gdb instance of rar2fs_level (the first of the fuse-stack). (The second stack never stopped working)
unique: 464526, opcode: RELEASE (18), nodeid: 10200, insize: 64, pid: 0 unique: 464525, error: -11 (Resource temporarily unavailable), outsize: 16 release[23455688824624] flags: 0x8000 unique: 464526, success, outsize: 16 [Thread 0x15553fe786c0 (LWP 15755) exited] [Thread 0x15553fc776c0 (LWP 15754) exited] [Thread 0x15553fa766c0 (LWP 15745) exited] [Thread 0x1555542b46c0 (LWP 14321) exited] [Thread 0x1555548fd6c0 (LWP 28228) exited] [Thread 0x155554afe6c0 (LWP 23254) exited] [Thread 0x155554d216c0 (LWP 23253) exited] [Thread 0x155554f446c0 (LWP 23251) exited] rar2fs[23247]: unmounted /mnt/user/rar2fs_level1 [Inferior 1 (process 23247) exited normally] (gdb) bt No stack.
Syslog reports as before:
Oct 9 15:14:00 unraid rar2fs[23247]: unmounted /mnt/user/rar2fs_level1
I have a theory. When I follow the debug of all 4 stacked fuse-systems, the first one, rar2fs_level1 is the one only spitting out random errors during the library scan.
And it mostly seems to be like this:
`LOOKUP /Movies/Rogue.One.2016.1080p.BluRay.x264-SPARKS/Subs/rogue.one.2016.1080p.bluray.x264-sparks.sub getattr /Movies/Rogue.One.2016.1080p.BluRay.x264-SPARKS/Subs/rogue.one.2016.1080p.bluray.x264-sparks.sub unique: 79756, error: -2 (No such file or directory), outsize: 16 unique: 79758, opcode: LOOKUP (1), nodeid: 7, insize: 92, pid: 27158
/Movies/Dirty.Grandpa.2016.1080p.BluRay.x264-GECKOS/Subs/ /Movies/This.Is.Where.I.Leave.You.2014.1080p.BluRay.x264-BLOW/Subs/blow-this.is.where.i.leave.you.2014.1080p.bluray.x264.sub
getattr /Movies/Let.Me.Eat.Your.Pancreas.2017.1080p.BluRay.x264-REGRET/Subs/let.me.eat.your.pancreas.2017.1080p.bluray.x264-regret.sub unique: 62760, error: -2 (No such file or directory), outsize: 16
getattr /Movies/Project.Almanac.2014.1080p.BluRay.x264-SPARKS/Subs/Project.Almanac.2014.1080p.BluRay.x264-SPARKS.sub unique: 77196, error: -2 (No such file or directory), outsize: 16`
In other words, always .sub giving errors.
ls printout of the different levels at the same directory below.
root@unraid:/mnt/user/rar2fs_level1/Movies/Let.Me.Eat.Your.Pancreas.2017.1080p.BluRay.x264-REGRET/Subs# ll total 5016 drwxrwxrwx 1 root root 160 Jul 2 2018 ./ drwxrwxrwx 1 root root 83 Feb 3 2022 ../ -rwxrwxrwx 1 root root 139021 May 10 2018 let.me.eat.your.pancreas.2017.1080p.bluray.x264-regret.idx* -rwxrwxrwx 1 root root 4989182 May 10 2018 let.me.eat.your.pancreas.2017.1080p.bluray.x264-regret.rar* -rwxrwxrwx 1 root root 74 Jul 2 2018 let.me.eat.your.pancreas.2017.1080p.bluray.x264-regret.subs.sfv*
/mnt/user/rar2fs/Movies/Let.Me.Eat.Your.Pancreas.2017.1080p.BluRay.x264-REGRET/Subs# ll total 21188 drwxrwxrwx 1 root root 160 Jul 2 2018 ./ drwxrwxrwx 1 root root 83 Feb 3 2022 ../ -rwxrwxrwx 1 root root 139021 May 10 2018 let.me.eat.your.pancreas.2017.1080p.bluray.x264-regret.idx* -rwxrwxrwx 1 root root 4989182 May 10 2018 let.me.eat.your.pancreas.2017.1080p.bluray.x264-regret.rar* -rwxrwxrwx 1 root root 16556032 May 10 2018 let.me.eat.your.pancreas.2017.1080p.bluray.x264-regret.sub* -rwxrwxrwx 1 root root 74 Jul 2 2018 let.me.eat.your.pancreas.2017.1080p.bluray.x264-regret.subs.sfv*
Is it possible that the errors somehow cumulatively causes the process to automatically unmount? _Since it's a refresh of the library in Emby I'm guessing that's why it's looking for the .sub files in those locations, and on rar2fslevel1 they do not exist, but on the second level they do. Could it cause some kind of dissonance?
CPU is fairly low whilst scanning, around 6-13% usage, and about 20GB of ram still available on the server.
On the last run the second level of rar2fs crashed... `LOOKUP /Movies/Blast.From.The.Past.1999.1080p.BluRay.x264-SiNNERS getattr /Movies/Blast.From.The.Past.1999.1080p.BluRay.x264-SiNNERS unique: 225228, error: -2 (No such file or directory), outsize: 16 unique: 225232, opcode: LOOKUP (1), nodeid: 7, insize: 91, pid: 1738 LOOKUP /Movies/Blast.From.The.Past.1999.1080p.BluRay.x264-SiNNERS getattr /Movies/Blast.From.The.Past.1999.1080p.BluRay.x264-SiNNERS unique: 225232, error: -2 (No such file or directory), outsize: 16 unique: 225234, opcode: LOOKUP (1), nodeid: 1, insize: 47, pid: 27436 LOOKUP /Movies getattr /Movies
Thread 5 "rar2fs" received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x1555548ba6c0 (LWP 30034)] 0x00001555551064ab in __strcmp_avx2 () from /lib64/libc.so.6 (gdb) bt
key=0x15541d56893b <error: Cannot access memory at address 0x15541d56893b>, hash=<optimized out>) at hashtable.c:149
hash=177546, level=32, level@entry=30) at hashtable.c:225
hash=177546, level=30, level@entry=29) at hashtable.c:222
hash=177546, level=29, level@entry=28) at hashtable.c:222
hash=177546, level=28, level@entry=27) at hashtable.c:222
hash=177546, level=27, level@entry=26) at hashtable.c:222
hash=177546, level=26, level@entry=25) at hashtable.c:222
hash=177546, level=25, level@entry=24) at hashtable.c:222
hash=177546, level=24, level@entry=23) at hashtable.c:222
hash=177546, level=23, level@entry=22) at hashtable.c:222
hash=177546, level=22, level@entry=21) at hashtable.c:222
hash=177546, level=21, level@entry=20) at hashtable.c:222
hash=177546, level=20, level@entry=19) at hashtable.c:222
hash=177546, level=19, level@entry=18) at hashtable.c:222
--Type
Can you please make sure to pull master/LATEST from git repo. It looks like the crash is something that has already been addressed and was first spotted in v1.29.5.
I did the following: git clone https://github.com/hasse69/rar2fs.git cd rar2fs wget https://www.rarlab.com/rar/unrarsrc-6.1.2.tar.gz tar -zxvf unrarsrc-6.1.2.tar.gz cd unrar make lib make install-lib cd .. autoreconf -f -i ./configure && make make install rar2fs -V rar2fs v1.29.5-gitf2e7893 (DLL version 8) Copyright (C) 2009 Hans Beckerus This program comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it under certain conditions; see http://www.gnu.org/licenses/ for details. FUSE library version: 2.9.9 using FUSE kernel interface version 7.19
flush[23455891258592] unique: 515976, success, outsize: 16 unique: 515977, opcode: INTERRUPT (36), nodeid: 0, insize: 48, pid: 0 INTERRUPT: 515976 unique: 515978, opcode: RELEASE (18), nodeid: 11631, insize: 64, pid: 0 unique: 515977, error: -11 (Resource temporarily unavailable), outsize: 16 release[23455891258592] flags: 0x8000 unique: 515978, success, outsize: 16 [Thread 0x15555427f6c0 (LWP 5546) exited] [Thread 0x1555544806c0 (LWP 5545) exited] [Thread 0x1555546b86c0 (LWP 4055) exited] [Thread 0x1555548db6c0 (LWP 4054) exited] [Thread 0x155554afe6c0 (LWP 27971) exited] [Thread 0x155554d216c0 (LWP 27970) exited] [Thread 0x155554f446c0 (LWP 27969) exited] rar2fs[27964]: unmounted /mnt/user/rar2fs_level1 [Inferior 1 (process 27964) exited normally]
Sound to me that you have corrupt .rar files like myself in some folders. You can start rar2fs but it kills it self after 10-30min. Works fine if i skip this folder on my side tho.
Please build with symbols and continue to run through gdb. We need to make sure there is no crash on any level. Again, rar2fs would never unmount itself. The only thing that could result in a controlled unmount is if the FUSE main loop exit. The error seem to indicate second level still crashes with the latest version.
Just to confirm; I assume you point Emby to the second- and never the first- level mount, right?
Sound to me that you have corrupt .rar files like myself in some folders. You can start rar2fs but it kills it self after 10-30min. Works fine if i skip this folder on my side tho.
Thanks for your feedback. However, no matter how corrupt things are, the file system should never unmount itself. There simply is no such logic in rar2fs to accomplish that.
Sound to me that you have corrupt .rar files like myself in some folders. You can start rar2fs but it kills it self after 10-30min. Works fine if i skip this folder on my side tho.
I thought so too actually, but I've run sfv-checks on some of the errored files, and they are all correct. However applying --seek-length=0 definitely helped Emby find more files and improved playback of video files in some cases.
I will look into how to build with symbols. Meanwhile I will try to describe the system. The host is unRAID which combines different drives into 1 share using SHFS and perhaps some voodoo I don't know about.
unraid SHFS - /mnt/user/ RAR2FS mounts /mnt/user/fin/ at /mnt/user/rar2fs_level1 RAR2FS mounts /mnt/user/rar2fs_level1 at /mnt/user/rar2fs rofs-filtered mounts /mnt/user/rar2fs at /mnt/user/rofs
Docker container for Emby is mapped to /mnt/user/rofs.
I will add for clarity that if I only mount 1 RAR2FS layer, and then rofs-filtered as described above. I never have any issues like this at all. The crash/unmount only happens when I stack 2 RAR2FS on top of each other with rofs-filtered on top.
To build with symbols just repeat what you did last time. Configure with --enable-debug and make sure to run through gdb and use the built binary directly from the src directory since the installed binary is automatically stripped from any symbols.
Any particular reason to why you need this rofs mount? Can you skip that to reduce the use-case somewhat? The rar2fs file system is mounted read-only by default anyway.
What I first wish to conclude is that the crash is no longer observed. Then we can take it from there. But I am bit worried this could be related to the version of FUSE you are using and specifically these weird INTERRUPT ops logged by it. That rar2fs appear to just unmount itself is more likely that FUSE decides to do that.
You should never point configure to /usr/lib! There is a great chance you compile against a source tree that does not match the library linked to. You should not need to use the —with-unrar-lib since build should by default link statically to the libunrar.a that was built in your unrarsrc directory.
I'm currently compiling from inside a VM running slackware (which unraid is based on). I think more than a few packages and dependencies are missing from unraid to compile directly on target. Is this something that could introduce problems?
Oh and about the rofs-mount. I don't care for the read only part, In fact I'd prefer if it was writeable. But I need to filter out a lot of files that are unnecessary for Emby.
Oh and about the rofs-mount. I don't care for the read only part, In fact I'd prefer if it was writeable. But I need to filter out a lot of files that are unnecessary for Emby.
That is perfectly alright, my ask was only about the possibility for you to temporarily skip that extra layer for the simple reason to reduce the scope of your use-case.
I'm currently compiling from inside a VM running slackware (which unraid is based on). I think more than a few packages and dependencies are missing from unraid to compile directly on target.
Is this something that could introduce problems?
Here I am not sure I fully understand. Then how did you compile last time since you obviously got a better stack trace with full symbol information?
Anyhow, let's skip that for now. All I wish to conclude is if the crash you experienced earlier is now gone. I think the syslog was pretty obvious here, showing that something crashed. Let's check that again.
I'm currently compiling from inside a VM running slackware (which unraid is based on). I think more than a few packages and dependencies are missing from unraid to compile directly on target. Is this something that could introduce problems?
Here I am not sure I fully understand. Then how did you compile last time since you obviously got a better stack trace with full symbol information?
Anyhow, let's skip that for now. All I wish to conclude is if the crash you experienced earlier is now gone. I think the syslog was pretty obvious here, showing that something crashed. Let's check that again.
I compile in the VM and move the binaries out to unraid in appropriate locations.
I'm trying to compile with the --enable-debug option but I get an error. Perhaps I'm not doing it correct?
root@slack:~/rar2fs# make --enable-debug install make: unrecognized option '--enable-debug'
No, --enable-debug is an option/switch to the configure script you normally run before you build and invoke make.
But as I proposed earlier, never mind this step. We only need this if you still observe the crash or we wish to instrument the code in order to get a better idea to what is going on when file system is magically unmounted for no apparent reason.
No, --enable-debug is an option/switch to the configure script you normally run before you build and invoke make.
thank you for your' patience. This is new territory for me. I think I successfully ran ./configure --enable-debug && make
The crashes do still occur. Since I started this thread I went from a pre-compiled binary supplied through "nerdpack" on unraid, to compiling my own binaries per your instructions. I guess that's why perhaps it looks a bit different.
Will try to "crash it again with gdb running" for both instances of RAR2FS.
Lets make sure we agree on the difference between the crash and the unmount. They might be related or they might also not be. But did you also pull in the latest version from master/HEAD before you built it? The last stack trace I saw pointed into the direction of an old version still.
Lets make sure we agree on the difference between the crash and the unmount. They might be related or they might also not be. But did you also pull in the latest version from master/HEAD before you built it? The last stack trace I saw pointed into the direction of an old version still.
git clone https://github.com/hasse69/rar2fs.git won't do?
I just got an unmount again
Invalidating hash key /TV.Series/The.Last.Man.On.Earth/Season.3/The.Last.Man.On.Earth.S03E01.720p.HDTV.x264-AVS/The.Last.Man .On.Earth.S03E01.720p.HDTV.x264-AVS.mkv in 0x155548000db0 Invalidating hash key /TV.Series/The.Last.Man.On.Earth/Season.3/The.Last.Man.On.Earth.S03E01.720p.HDTV.x264-AVS in 0x1555480 00db0 Invalidating all hash keys in 0x155548000db0 rar2fs[28754]: unmounted /mnt/user/rar2fs_level1 [Inferior 1 (process 28754) exited normally]
Nothing special in syslog.
Will do a second run just to be sure.
Second run done, and it just unmounted again without crashing.
Yes, seems like the crash is gone at least. But the unmount remains a mystery. Can you run outside of gdb now and also mount using the -d flag to collect some debug output from FUSE itself. If possible also skip the rofs mount.
Getting really confused right now. I skipped adding the last layer with rofs-filtered. Ie only running 2 layers of rar2fs, and I can't get it to unmount like before. Really annoying seeming as rofs-filtered didn't crash earlier, but somehow rar2fs unmounted. So I still don't know the cause. And I do need the filtering capabilities...got too much files which emby is being confused by and cluttering the library with.
The only plan left I have is that I will set up a log of all the processes running, and see if any process at all is being changed when the unmount happens.
Try the opposite then? Let rofs become your first level, then mount rar2fs as level 2 and 3.
Seems to be working. Still none the wiser why the reverse wont work though.
I do loose some functionality though. Mainly:
There are some files inside the RAR-archives I'd like to filter out, like .ifo files which Emby picks up even though I'd rather not.
Second, I loose write-capability for the docker containers that do need it, like bazaar. I suppose I could just mount 1-2 more instances of rar2fs specifically for that, which brings me up to 5-6 FUSE, seems awfully complicated :)
I have looked at cmdfs as an alternative to rofs-filtered, but havn't got it working yet, not sure of the state of it since it's been free of updates since 2018.
My dream scenario would of course be if rar2fs had an option to point to a config file with regex based filters much like rofs-filtered does ;-)
Seems to be working. Still none the wiser why the reverse wont work though.
That makes two of us I am afraid. The only thing that comes to mind is if rofs-filtered for some reason is getting punished by accessing rar2fs level 2 and 1, possibly caused by some low level timeout in FUSE? Something that ripples down to render some unexpected tear down on kernel level. I simply cannot seem to ignore the fact that those INTERRUPT ops logged by FUSE look very suspicious. I guess the only way to get some answers here would be to raise a question to the FUSE support mailing list. Something you could try is to create three levels of rar2fs mounts, and skip rofs-filtered completely. Then point Emby to your third level. That would then basically become a dummy mount but would also mimic something rather similar to what rofs-filtered does. Just to see if this in some way is related to rofs or in fact something with FUSE as such. That this would boil down to some issue with rar2fs sounds a bit far fetched, but cannot be out-ruled of course. But then we need more facts in order to pursue this issue further.
My dream scenario would of course be if rar2fs had an option to point to a config file with regex based filters much like rofs-filtered does ;-)
I can only give you the same answer I have given other users with similar requests in the past. Having a feature to filter out files would be outside scope of what rar2fs was designed to do. Not only does it make the design more complex but also there should be other file system or tools available that would do a much better job at it specifically designed for that particular task. I am a bit surprised that Emby itself does not provide such a feature. It cannot be unique to your use case that files might cause unwanted "noise" in the media libraries. Maybe there is some plug-in that actually would deliver what you want? Have you looked into that possibility?
Have you checked this?
https://emby.media/community/index.php?/topic/109245-auto-organize-20-public-beta/
Okay so first. I will look into stacking 3 layers of RAR2FS like you suggested, I will also try to log ps -e for any differences during the unmount.
Second. I did search yesterday night after seeing your suggestion about there possibly being some plugin or emby function for this. And I did not find anything, now I come back and see you have potentially found something. Thanks for that.
I will try this out, but also report back about the unmounting part.
Any progress here?
Yes a little bit. I stacked 4 layers of RAR2FS, no errors. I havn't had the chance yet to log for processes during the unmount. I will get to it tho.
I looked into filtering-capability plugins for emby, but the consensus when reading the forum is that it's something the filesystem should deliver. I also looked into the plugin you found as well, but it does not seem to deliver such functionality .
Yes a little bit. I stacked 4 layers of RAR2FS, no errors.
Sounds to me rofs is involved somehow, could it be too slow when filtering out content? There are no mount options that could be used for rofs that can be related to performance?
I looked into filtering-capability plugins for emby, but the consensus when reading the forum is that it's something the filesystem should deliver. I also looked into the plugin you found as well, but it does not seem to deliver such functionality .
Sounds strange, there are no standard file systems delivering this functionality so how can consensus be to rely on such? Also the plug-in when I looked at it seemed to deliver filtering on wildcard level?
The plugin is more about sorting unsorted files into the correct libraries. For example, dumping everything into downloads, and then letting the plugin sort it out to whatever appropiate library, like tv or movies. I asked in the thread about the use of filtering, and the creator confirmed that the plugin won't do what I'm looking for.
rofs do have some mount options, and I've tried a few over the last 1-2 years. What i've tried to change is the more standard fuse-options like splice_read. I havn't tried to change the various read options tho. like:
-o max_readahead=N set maximum readahead
-o max_background=N set number of maximum background requests
-o congestion_threshold=N set kernel's congestion threshold
-o async_read perform reads asynchronously (default)
-o sync_read perform reads synchronously
Anyway, I ran some more tests and logged the output of ps -e and pstree -a into seperate files and compared them with winmerge. I'm not that fluent in linux processes, but there is definitly some difference between the outputs of ps-e. Do note that the system is not used in any other way during the time than re-scanning the library inside emby (well I have a VM running pfsense also). Most of the difference seems to be related to kworker. Although that might not be the problem. I saw some lines about btrfs as well. The docker container is running on disks with btrfs as well as emby's database, the media is not tho.
In the middle of a house move. Will get back to this a bit later. Is there anything else you can think of that I can log or look into given the running processes?
Not at this moment, no. Since this seems related to rofs rather than rar2fs I am not sure how much further we will get here. I would try to push for some active media server/database filter. To claim this to be the responsibility of a file system is wrong in my opinion. It is the media server that dictates what files it can serve/support, then it should also be able to ignore files that is bloating the libraries.
I'm a having difficulties lately to find the time to further investigate this issue. I had an idea of trying to somehow make cmdfs work, and try it together with rar2fs. But haven't had they time. Probably fine to close this issue IMO.
Seems like cmdfs might cover your needs, despite the fact that it does not seem to be a very active project.
I will close this issue since there are nothing so far pointing to it being a problem with rar2fs. Feel free to re-open if there are any new findings.
So I've been having issues with my first stack of rar2fs getting unmounted when I scan my library in Emby together with rofs-filtered.
Emby is mounted on the 2 rofs-filtered systems.
This is my setup (although I've tried various combinations, such as removing cache options, and without splice options)
rar2fs --exclude=.lock --seek-length=1 --relatime -d -o auto_cache -o direct_io -o max_write=131072 -o splice_read -o splice_write -o splice_move -o allow_other -o nonempty -o umask=0000 -o uid=0 -o gid=0 /mnt/user/share/fin /mnt/user/rar2fs_level1
rar2fs --exclude=.lock --seek-length=1 --relatime -o auto_cache -o direct_io -o max_write=131072 -o splice_read -o splice_write -o splice_move -o allow_other -o nonempty -o umask=0000 -o uid=0 -o gid=0 /mnt/user/rar2fs_level1 /mnt/user/rar2fs
rofs-filtered /mnt/user/rofs -o source=/mnt/user/rar2fs -o config=/etc/filter1.rc -o kernel_cache -o auto_cache -o direct_io -o max_write=131072 -o splice_read -o splice_write -o splice_move -o allow_other -o nonempty -o umask=0000 -o uid=0 -o gid=0
rofs-filtered /mnt/user/rofs_ful -o source=/mnt/user/share/ful -o config=/etc/filter1.rc -o kernel_cache -o auto_cache -o direct_io -o max_write=131072 -o splice_read -o splice_write -o splice_move -o allow_other -o nonempty -o umask=0000 -o uid=0 -o gid=0
The scan in emby runs for a few minutes, then comes to a halt. The 2 last attempts with debug enabled on the rar2fs_level1 resulted in the output below. I'm not sure what I am looking at here, the first attempt clearly has errors, but the second seems to unmount without errors. Any pointers on what I can read out from this, or look further into?
read[23159808100864] 32740 bytes from 1440712 flags: 0x8000 read[23159808100864] 32740 bytes from 1440712 unique: 377498, success, outsize: 32756 unique: 377500, opcode: FLUSH (25), nodeid: 4390, insize: 64, pid: 12911 flush[23159808100864] unique: 377500, success, outsize: 16 unique: 377501, opcode: INTERRUPT (36), nodeid: 0, insize: 48, pid: 0 unique: 377502, opcode: RELEASE (18), nodeid: 4390, insize: 64, pid: 0 release[23159808100864] flags: 0x8000 INTERRUPT: 377500 unique: 377502, success, outsize: 16 unique: 377504, opcode: OPEN (14), nodeid: 4390, insize: 48, pid: 12920 unique: 377501, error: -11 (Resource temporarily unavailable), outsize: 16 unique: 377505, opcode: INTERRUPT (36), nodeid: 0, insize: 48, pid: 0 INTERRUPT: 377504 open flags: 0x8000 /Movies/Extremely.Loud.and.Incredibly.Close.2011.720p.BluRay.X264-AMIABLE/Subs/Extremely.Loud.and.Incredibly.Close.2011.720p.BluRay.X264-AMIABLE.rar open[23159807905184] flags: 0x8000 /Movies/Extremely.Loud.and.Incredibly.Close.2011.720p.BluRay.X264-AMIABLE/Subs/Extremely.Loud.and.Incredibly.Close.2011.720p.BluRay.X264-AMIABLE.rar unique: 377504, success, outsize: 32 unique: 377506, opcode: FLUSH (25), nodeid: 4390, insize: 64, pid: 12920 flush[23159807905184] unique: 377507, opcode: INTERRUPT (36), nodeid: 0, insize: 48, pid: 0 INTERRUPT: 377506 unique: 377506, success, outsize: 16 unique: 377508, opcode: RELEASE (18), nodeid: 4390, insize: 64, pid: 0 release[23159807905184] flags: 0x8000 unique: 377508, success, outsize: 16 rar2fs[6743]: unmounted /mnt/user/rar2fs_level1
And
read[22712525407936] 32740 bytes from 1571641 flags: 0x8000 read[22712525407936] 32740 bytes from 1571641 unique: 374466, success, outsize: 32756 unique: 374468, opcode: LOOKUP (1), nodeid: 1, insize: 47, pid: 18693 LOOKUP /Movies getattr /Movies NODEID: 2 unique: 374468, success, outsize: 144 unique: 374470, opcode: FLUSH (25), nodeid: 1818, insize: 64, pid: 22959 unique: 374471, opcode: INTERRUPT (36), nodeid: 0, insize: 48, pid: 0 INTERRUPT: 374470 flush[22712525407936] unique: 374470, success, outsize: 16 unique: 374472, opcode: RELEASE (18), nodeid: 1818, insize: 64, pid: 0 release[22712525407936] flags: 0x8000 unique: 374472, success, outsize: 16 rar2fs[17365]: unmounted /mnt/user/rar2fs_level1