Closed zeeter82 closed 2 years ago
Oh my cmd line options are: rar2fs -owarmup -o allow_other --seek-length=1
After some searching I see now some others possibly ran into similar issues, but those were patched. Had another drop a few min ago...About a week ago this issue was non-existent....no idea what changed.
I was able to find this in /var/log/messages:
Nov 13 01:56:37 host123 rar2fs[1263]: Got signal 11, faulty address is (nil), from 0x5601b8c7e310
Nov 13 01:56:37 host123 rar2fs[1263]: rar2fs(+0x13310) [0x5601b8c7e310]
Nov 13 01:56:37 host123 rar2fs[1263]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7f1e0835d140]
Nov 13 01:56:37 host123 rar2fs[1263]: /lib/x86_64-linux-gnu/libc.so.6(cfree+0x1c) [0x7f1e0820e73c]
Nov 13 01:56:37 host123 rar2fs[1263]: rar2fs(+0x1d420) [0x5601b8c88420]
Nov 13 01:56:37 host123 rar2fs[1263]: rar2fs(+0x1db7e) [0x5601b8c88b7e]
Nov 13 01:56:37 host123 rar2fs[1263]: /lib/x86_64-linux-gnu/libfuse.so.2(+0xfb29) [0x7f1e08561b29]
Nov 13 01:56:37 host123 rar2fs[1263]: /lib/x86_64-linux-gnu/libfuse.so.2(+0x17252) [0x7f1e08569252]
Nov 13 01:56:37 host123 rar2fs[1263]: /lib/x86_64-linux-gnu/libfuse.so.2(+0x18350) [0x7f1e0856a350]
Nov 13 01:56:37 host123 rar2fs[1263]: /lib/x86_64-linux-gnu/libfuse.so.2(+0x15039) [0x7f1e08567039]
Nov 13 01:56:37 host123 rar2fs[1263]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8ea7) [0x7f1e08351ea7]
Nov 13 01:56:37 host123 rar2fs[1263]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f1e08281def]
Update: I reinstalled using latest git clone and changed my unrarsrc to 6.0.3....I was using 6.0.7. I'll see what happens.
If this problem suddenly just started to appear I can only guess it is some new archive addition to your library that triggered this issue.
That would make sense, but the main mount that keeps failing hasn't had any new content added in months. So still kinda lost here. And it's still randomly crashing after making my changes from above.
Then you need to rebuild and configure using --enable-debug so we can get some symbols from the crash signature. Also since it seems fairly easy to reproduce run rar2fs from within gdb and it will trap the crash.
Just an FYI: I'm running this inside Debian 11 VM as well which I update fairly regularly.....maybe a kernel/smb issue going on here too?
I'll see what I can do to get some debug output. Thanks!
I'm not sure how to use the gdb method, everything I tried wasn't working. I'll try to configure using --enable-debug.
I had a couple mounts crash again:
Nov 13 19:06:31 host123 rar2fs[1964]: Got signal 11, faulty address is (nil), from 0x5594c5f4acd7
Nov 13 19:06:31 host123 rar2fs[1964]: rar2fs(+0x13cd7) [0x5594c5f4acd7]
Nov 13 19:06:31 host123 rar2fs[1964]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7f3561e35140]
Nov 13 19:06:31 host123 rar2fs[1964]: /lib/x86_64-linux-gnu/libc.so.6(cfree+0x1c) [0x7f3561ce673c]
Nov 13 19:06:31 host123 rar2fs[1964]: rar2fs(+0x1d579) [0x5594c5f54579]
Nov 13 19:06:31 host123 rar2fs[1964]: rar2fs(+0x1d71f) [0x5594c5f5471f]
Nov 13 19:06:31 host123 rar2fs[1964]: rar2fs(+0x1e77f) [0x5594c5f5577f]
Nov 13 19:06:31 host123 rar2fs[1964]: /lib/x86_64-linux-gnu/libfuse.so.2(+0xfb29) [0x7f3562039b29]
Nov 13 19:06:31 host123 rar2fs[1964]: /lib/x86_64-linux-gnu/libfuse.so.2(+0x17252) [0x7f3562041252]
Nov 13 19:06:31 host123 rar2fs[1964]: /lib/x86_64-linux-gnu/libfuse.so.2(+0x18350) [0x7f3562042350]
Nov 13 19:06:31 host123 rar2fs[1964]: /lib/x86_64-linux-gnu/libfuse.so.2(+0x15039) [0x7f356203f039]
Nov 13 19:06:31 host123 rar2fs[1964]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8ea7) [0x7f3561e29ea7]
Nov 13 19:06:31 host123 rar2fs[1964]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f3561d59def]
Nov 13 19:06:47 host123 rar2fs[1470]: Got signal 11, faulty address is (nil), from 0x55b3754d5cd7
Nov 13 19:06:47 host123 rar2fs[1470]: rar2fs(+0x13cd7) [0x55b3754d5cd7]
Nov 13 19:06:47 host123 rar2fs[1470]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7faf0d161140]
Nov 13 19:06:47 host123 rar2fs[1470]: /lib/x86_64-linux-gnu/libc.so.6(cfree+0x1c) [0x7faf0d01273c]
Nov 13 19:06:47 host123 rar2fs[1470]: rar2fs(+0x1d579) [0x55b3754df579]
Nov 13 19:06:47 host123 rar2fs[1470]: rar2fs(+0x1d71f) [0x55b3754df71f]
Nov 13 19:06:47 host123 rar2fs[1470]: rar2fs(+0x1e77f) [0x55b3754e077f]
Nov 13 19:06:47 host123 rar2fs[1470]: /lib/x86_64-linux-gnu/libfuse.so.2(+0xfb29) [0x7faf0d365b29]
Nov 13 19:06:47 host123 rar2fs[1470]: /lib/x86_64-linux-gnu/libfuse.so.2(+0x17252) [0x7faf0d36d252]
Nov 13 19:06:47 host123 rar2fs[1470]: /lib/x86_64-linux-gnu/libfuse.so.2(+0x18350) [0x7faf0d36e350]
Nov 13 19:06:47 host123 rar2fs[1470]: /lib/x86_64-linux-gnu/libfuse.so.2(+0x15039) [0x7faf0d36b039]
Nov 13 19:06:47 host123 rar2fs[1470]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8ea7) [0x7faf0d155ea7]
Nov 13 19:06:47 host123 rar2fs[1470]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7faf0d085def]
How do I get you debugging logs?
Is this now a version running your own build configured with --enable-debug? I think the best way to get a better crash signature is to run the program through gdb, such as:
gdb --args src/rar2fs -owarmup -o allow_other --seek-length=1
with any additional options you need like source and destination path etc. Note that if you built your own version and did not also install it using 'make install' it is very likely you still run the old version and not the one you compiled. The resulting binary is located in the src directory in the root of the rar2fs source directory.
From the gdb prompt just type 'run' and rar2fs should mount as expected. Note that this means the program will run in the foreground so you need to keep that terminal window open to see when it crashes and halts. When it does, you simply type 'where' and gdb should provide a bit more detailed stack trace.
Yep I rebuilt from a new cloned git and reconfigured with --enable-debug and did make install and rebooted. I can also try the gdb method from your instructions above. Seems like I get 1-3 crashes every 2-6 hours now.
So how exactly does this work? I'm confused and can't tell that I ran it correctly.
gdb --args /stuff/rar2fs/src/rar2fs -owarmup -o allow_other --seek-length=1 /source/path/ /mount/path/
gdb now displays: Type "apropos word" to search for commands related to "word"... Reading symbols from /stuff/rar2fs/src/rar2fs...
But I check my mounts and it didn't actually mount anything. Is this normal? Will I not be able to actually use this as a rar2fs mount while it debugs or reads symbols?
UPDATE:
Haha nevermind. I missed your step above to "run" after getting to gdb prompt. Seems to be all good now.
So how do I see any output or logs? I think one of these gdb sessions had a crash, but the window still looks normal. It didn't change. I thought the putty buffer was going to fill up with a bunch of output.
So I think I messed up and probably lost the data we needed. I unmounted the two crashed mounts, but have them mounted again inside gdb sessions. It seems to be random which ones crash, so I might eventually have to load all of them in gdb.
Yes if you have multiple mounts you of course need to monitor them all.
After I get another crash how do I get you logs/symbols?
I got a crash. I left the window open and I haven't manually performed the unmount yet. This is what I get:
(gdb) where
No stack.
Is it because maybe it hasn't fully crashed? I think I can still see one related process ID running, but a df -h shows:
df: /mount/path123: Transport endpoint is not connected
Maybe that's related to the gdb session itself. Do I need to run fusermount -u on the crashed mount before I can do the "where" from gdb? Not sure what my next steps should be.
Sorry, I was not clear. When you run through gdb you must also add the -f flag to the rar2fs arguments or else gdb will not properly intercept the crashed process.
After it has crashed you always need to unmount before trying to start a new mount. But that has nothing to do with gdb, that is always the case.
Ahh gotcha no worries. Makes sense haha. I'll add the -f flag and get all my sessions going again.
I don't know if this is a crash or not, but my box and VM became unstable this morning. So maybe that's part of the problem too. Maybe the last Plex update or the last VM update had an issue. But I had 12 gdb sessions running for all my mounts and noticed this in one of the windows:
`Read 131072 bytes from vol=0, base=0
rar2_read() size=131072, offset=626688, fh=140736818959936
PID 207955 calling lread_raw(), seq = 10, offset=626688
size = 131072, chunk = 99373179
Read 131072 bytes from vol=0, base=0
rar2_flush()
(207955) FLUSH [0x7fffe4128ae0 ][called from 215972]
lflush()
rar2_release()
(207955) RELEASE [0x7fffb4031a10 ]
Closing file handle 0x7fffe421c070
(207955) FREE [0x7fffb4031a10 ]
rar2_flush()
(207955) FLUSH [0x7fffe4128c50 ][called from 215972]
lflush()
rar2_release()
(207955) RELEASE [0x7fffe4177640 ]
Closing file handle 0x7fffc4023cc0
(207955) FREE [0x7fffe4177640 ]
rar2_flush()
(207955) FLUSH [0x7fffb8000910 ][called from 215972]
lflush()
rar2_release()
(207955) RELEASE [0x7fffe41a0e20 ]
--Type
Thread 3274 "rar2fs" received signal SIG32, Real-time event 32. [Switching to Thread 0x7fffdcdf7700 (LWP 345063)] futex_abstimed_wait_cancelable (private=0, abstime=0x7fffdcdf6ed0, clockid=-589337008, expected=0, futex_word=0x7fffb80009c4) at ../sysdeps/nptl/futex-internal.h:323 323 ../sysdeps/nptl/futex-internal.h: No such file or directory. (gdb) where
at ../sysdeps/nptl/futex-internal.h:323
I might have to rebuild my VM. Not sure what's going on but it doesn't like me starting up all these gdb sessions. It's almost like the NIC crashes under vmware. There might be an underlying issue with VM anyways, but I don't understand how it would just appear out of nowhere like this.
Nothing just happens, there is always a reason to why it does.
There is almost nothing, irrespective how broken it is, that should result in a crash in rar2fs. If it does it is not robust enough. Still the underlying problem might make things behave differently but crashing, unacceptable.
I think you need to trap SIG32 in gdb to make it ignore it and not stop. That is not the signal we are interested in. Thus this was not a crash you saw but some interrupt that causes gdb to halt the process. I don't have it in my head so please google how to ignore signals in gdb.
A simple search gave me this:
https://gnobal.net/591/what-is-sig32-during-a-gdb-debug-session
Btw, how many mounts do you have exactly? Was just thinking that it might be a good idea to reduce your configuration to a minimum for which you can still reproduce the issue.
I have 12 mounts total. I guess I can try to lower this, but I might just work on creating a new VM. I would hate to keep troubleshooting the crashes when it's something that has nothing to do with rar2fs.
Nothing just happens, there is always a reason to why it does.
There is almost nothing, irrespective how broken it is, that should result in a crash in rar2fs. If it does it is not robust enough.
I've always beefed up this VM to have 4 cores and 4GB ram. I would think that would be plenty and it's never been a problem until the last couple weeks or so.
As said already, if rar2fs crash there is a problem with it unless it crashes somewhere in which rar2fs has no control of it, such as system calls or third party libraries. Only a stack trace would tell. By trying to work around the crash by changing the VM does not change the fact that there is some issue lurking around here. Even if you would be helped by a new VM hiding the issue, others would not. Do you see where I am trying to get here?
Yep I gotcha. I'll try to help get this figured out. I updated my host, updated vmware, and rebooted. Got a few gdb sessions started and started up Plex. Let's see what happens.
So I don't know how much more help I can be with this. I think I'm going to have to try rebuilding my VM now because I can't even do a simple Plex library scan. As soon as I kick off a scan the VM kinda just goes into lala land. I run df -h on a new shell and it just sits there with no output and the UNC connection to the shared mount dies (I can no longer access from host Win10 box). I'm not sure if this is because of running some of my mounts with gdb or if there are other issues, but something has changed within the last 2 weeks to cause my setup to go crazy. I've been running this for a few years now with no major issues like this. Although the last time I had issues with my Plex scanning I did rebuild my rar2fs VM. So this would be the second time rebuilding if I go that route.
If you changed PLEX media server recently first thing I would try is to go back to a version that was confirmed working on your system. I get more and more reports about recent versions of PMS causing all kinds of issues. I have no clue why that is.
So I reverted back to an older VM snapshot that's on Debian 10 (the one I was on I had upgraded to 11 a couple months back). I'm also running an older rar2fs build in this snapshot:
rar2fs v1.29.1 (DLL version 8) Copyright (C) 2009 Hans Beckerus
This program comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it under
certain conditions; see <http://www.gnu.org/licenses/> for details.
FUSE library version: 2.9.9
fusermount version: 2.9.9
using FUSE kernel interface version 7.19
So far everything is stable and Plex seems to be fine with scanning etc. as well. I'll keep it like this for a few days then upgrade my rar2fs to latest and see what happens.
Also haveing this problem, last 2 weeks reverted plex back to 1.18.9.2578 from 1.24.. still, the same error. Guess i need to reinstall rar2fs with --enable-debug . Running Debian 11, ran apt update a whileago don't know if its related. But this error is new for me to.
Also haveing this problem, last 2 weeks reverted plex back to 1.18.9.2578 from 1.24.. still, the same error. Guess i need to reinstall rar2fs with --enable-debug . Running Debian 11, ran apt update a whileago don't know if its related. But this error is new for me to.
I really don't know enough about it, but from what I'm seeing in my environment, rolling back my VM to Debian 10 has been 100% stable so far. I'm just curious since that you are also on Debian 11 and seeing similar issues if something changed with a recent kernel/package update.
I'm also running an older version of rar2fs since I reverted to snapshot, but I plan on upgrading to latest git clone soon if I don't see anymore issues.
So I definitely found an interesting issue, but I really need to keep my Plex system running and just can't continue to experience downtime while assisting with troubleshooting.
The snapshot that I reverted back to running Debian 10 and rar2fs v1.29.1 has been 100% stable over the last few days. So I decided to upgrade once again to latest rar2fs while still being on Debian 10:
rar2fs v1.29.5-gitc87cd02 (DLL version 8) Copyright (C) 2009 Hans Beckerus
This program comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it under
certain conditions; see <http://www.gnu.org/licenses/> for details.
FUSE library version: 2.9.9
fusermount version: 2.9.9
using FUSE kernel interface version 7.19
also using unrarsrc-6.1.2
My mounts immediately started crashing again while Plex does it's periodic rescans and/or metadata refreshes. So I would think the issue is lying somewhere with the newer rar2fs and/or unrarsrc (or something with newer Plex builds interacting with rar2fs). I'm hoping some others can help troubleshoot, because I just don't know how much longer I can keep making my libraries go up and down while trying to debug.
For now I'm reverting back to my snapshot again where everything is working and stable. I'll try to help where I can, but do I like to aim for 100% uptime for my Plex server.
Did you use same unrarsrc when swapping between 1.29.1 and 1.29.5 I think you at least need to narrow it down to for what version of rar2fs your problems starts, eg what happens if you try 1.29.3?
No when I updated to 1.29.5 I also updated to latest unrarsrc 6.1.2.
So when I reverted to working snapshot again, I'm using 1.29.1 and I believe unrarsrc-5.9.4. Is there a way to confirm my unrarsrc version?
Should I try to upgrade to 1.29.5 again and not touch unrarsrc (keep the version the same)?
I havent had a crash since last time i wrote. What i was running when i first wrote. Changed plex version from 1.24 to 1.18 and was using rar2fs 1.29.5 with unrarsrc 6.0.7 and it still crashed everyday. After my last reply i reinstalled rar2fs 1.29.5 with unrarsrc 6.1.2 and still used plex 1.18. Everything works as it shuld (so far) Had similar problem where a rar2fs mount just hangs with a plex alternative and had had to force kill the rar2fs process and remount it. cant give you more on this error yet.
You should avoid using any distribution versions of rar2fs since they tend to use dynamic linking of libunrar which is very error prone. You should download and build your own version of libunrar and rar2fs will link statically to that version by default. When you configure your rar2fs package you need to make sure your version of libunrar is picked up properly by looking at output from configure or the config.log. That is how you make sure proper version is used. There is no other way. If you build against unrarsrc but another version is picked by the dynamic linker it is very likely it will crash due to ABI not being compatible.
checking rar.hpp usability... yes
checking rar.hpp presence... yes
checking for rar.hpp... yes
checking for static linking of UnRAR library... fallback
checking if unrar library needs -DRAR_SMP... yes
I would also avoid using 1.29.5 since that has a known issue. Better then to use 1.29.4 or build master/HEAD.
I've never used the distribution versions for these. I've always built from source from here (or latest git clone) and grabbed rarlabs unrarsrc and built that. So I should try running 1.29.4 and with what version of unrarsrc?
Make sure you build unrarsrc (make lib) before you configure the rar2fs package. If you are unsure just post your config.log and we can double check. I just tried latest version of unrarsrc and see no issue with using that.
I just built 1.29.4 with unrarsrc 6.1.2. I'll let this run a bit and see what happens.
checking rar.hpp usability... yes
checking rar.hpp presence... yes
checking for rar.hpp... yes
checking for static linking of UnRAR library... fallback
checking if unrar library needs -DRAR_SMP... yes
Ok, it is a bit unfortunate choice of the word "fallback". It basically means it will try static linkage if it can (if library archive can be found in selected source folder) otherwise dynamic linking will be used. To the only way to be absolutely sure static linkage is applied and nothing else is by using --enable-static-unrar configure option. But no worries, you can easily check what it ended up to become using 'ldd path-to-your-built-version-of-rar2fs'
I see this:
rar2fs-1.29.4# ldd src/rar2fs
linux-vdso.so.1 (0x00007ffd8ff2c000)
libfuse.so.2 => /lib/x86_64-linux-gnu/libfuse.so.2 (0x00007f0afdfa9000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f0afdfa4000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f0afdf9a000)
libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f0afde16000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f0afdc93000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f0afdc79000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f0afdc56000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f0afda95000)
/lib64/ld-linux-x86-64.so.2 (0x00007f0afe05d000)
Is that what you are looking for? I don't see anything about the unrarsrc there.
I've pretty much always followed these instructions:
https://github.com/hasse69/rar2fs/wiki#q-13-how-do-i-build-this-package
I see this:
rar2fs-1.29.4# ldd src/rar2fs linux-vdso.so.1 (0x00007ffd8ff2c000) libfuse.so.2 => /lib/x86_64-linux-gnu/libfuse.so.2 (0x00007f0afdfa9000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f0afdfa4000) librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f0afdf9a000) libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f0afde16000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f0afdc93000) libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f0afdc79000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f0afdc56000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f0afda95000) /lib64/ld-linux-x86-64.so.2 (0x00007f0afe05d000)
Is that what you are looking for? I don't see anything about the unrarsrc there.
That is actually what to expect for static linkage, so all is good.
Got a couple more crashes while running 1.29.4 and unrarsrc 6.1.2.
For now I'm gonna revert back to 1.29.1 and unrarsrc 5.9.4....this has still proven to be stable in my environment/config.
Not adding anything new to the discussion but would just like to say that I have experienced the same problem as described. I have sort of worked around it and have just gathered all mounts into a single one in rar2fs and that seems to work better. Im not helping but I just want to reassure zeeter82 that there certainly is a problem, somewhere.
Fantastic program btw.
Got a couple more crashes while running 1.29.4 and unrarsrc 6.1.2.
For now I'm gonna revert back to 1.29.1 and unrarsrc 5.9.4....this has still proven to be stable in my environment/config.
Can we avoid changing too many parameters at the same time? Stay with same unrarsrc and only bump rar2fs version or vice versa. Trying to narrow down a problem is a lot more tricky otherwise. My bet would be to use latest unrarsrc and then slowly move backwards until things starts to work again. If you cannot make it work by going back all the way to rar2fs v1.29.1 then problem is probably with unrarsrc and we need to take it from there.
I am also having this issue. Running version v1.29.5-gitc87cd02 with unrarsrc-6.0.7.tar.gz, statically linked.
I am also having this issue. Running version v1.29.5-gitc87cd02 with unrarsrc-6.0.7.tar.gz, statically linked.
Then may I ask for the same assistance in trying to find at what version of rar2fs this problem started and also a proper crash dump with symbols?
I am also having this issue. Running version v1.29.5-gitc87cd02 with unrarsrc-6.0.7.tar.gz, statically linked.
Then may I ask for the same assistance in trying to find at what version of rar2fs this problem started and also a proper crash dump with symbols?
Sure, I have recompiled the same versions with --enable-debug and running through gdb now. Now waiting for it to crash.
I'm having random mounts drop on me and I'm not sure why:
running df -h shows this: df: /home/xyz123/plex_mounts/fs02-rar2fs-x264-1080p: Transport endpoint is not connected
This is the second time this has happened in the last couple days and I've done nothing to change configuration. I can resolve it by unmounting share and mounting again. The file server (source of files) doesn't seem to be having any issues.
Any ideas?