Malfunction with unrar>=6.0 (or unrar>=6.1)?

[tsjk]

I run the git-version of rar2fs on my gentoo system (mostly because the latest release seemingly leaks resources and I get daily OOM-kills). The git-version seems to work better, but with unrar-6.1.7 many archives are rar2fs-d strangely, where in-archive files show up as directories. Downgrading to unrar-5.9.4 and recompiling rar2fs makes the issue go away.

[hasse69]

Thanks for the issue report. I believe that latest version of unrarsrc confirmed to work is v6.0.3, can you please confirm if that works ok also in your environment?

Regarding the resource leaks you experience with the officially released version of rar2fs. That seems a bit strange since I cannot recall any other reports of similar issues and the delta compared with the master/HEAD should not really address any major leaks as well.

[hasse69]

Also maybe if you could check if this is the patch that makes a differences with respect to the resource leaks.

https://github.com/hasse69/rar2fs/commit/11819a872b0db016c4d9abacf226c9351105da95

Otherwise I would suggest you write a separate issue report for this particular problem since it is not related to the unrarsrc compatibility from what I can tell by your description.

[tsjk]

I'll check with v6.0.3 tomorrow. I realize that this is a rather informal bug report, and I apologize. Anyway, with the official release I had like a daily of eg

Apr 11 07:20:24 nil kernel: [1267397.553887][ T9715] CPU: 3 PID: 9715 Comm: rar2fs Tainted: P        W  O      5.15.26-gentoo-20220309_1 #1
Apr 11 07:20:24 nil kernel: [1267397.553889][ T9715] Hardware name: Gigabyte Technology Co., Ltd. Z370 HD3P/Z370 HD3P-CF, BIOS F6 03/01/2018
Apr 11 07:20:24 nil kernel: [1267397.553890][ T9715] Call Trace:
Apr 11 07:20:24 nil kernel: [1267397.553891][ T9715]  <TASK>
Apr 11 07:20:24 nil kernel: [1267397.553892][ T9715]  dump_stack_lvl+0x34/0x44
Apr 11 07:20:24 nil kernel: [1267397.553895][ T9715]  dump_header+0x45/0x1eb
Apr 11 07:20:24 nil kernel: [1267397.553898][ T9715]  oom_kill_process.cold+0xb/0x10
Apr 11 07:20:24 nil kernel: [1267397.553899][ T9715]  out_of_memory+0x1f2/0x290
Apr 11 07:20:24 nil kernel: [1267397.553902][ T9715]  mem_cgroup_out_of_memory+0x131/0x150
Apr 11 07:20:24 nil kernel: [1267397.553905][ T9715]  try_charge_memcg+0x6ce/0x790
Apr 11 07:20:24 nil kernel: [1267397.553906][ T9715]  charge_memcg+0x3a/0x90
Apr 11 07:20:24 nil kernel: [1267397.553908][ T9715]  __mem_cgroup_charge+0x27/0x80
Apr 11 07:20:24 nil kernel: [1267397.553910][ T9715]  wp_page_copy+0xfa/0x890
Apr 11 07:20:24 nil kernel: [1267397.553912][ T9715]  ? __schedule+0x28d/0x1190
Apr 11 07:20:24 nil kernel: [1267397.553914][ T9715]  __handle_mm_fault+0xabd/0x13a0
Apr 11 07:20:24 nil kernel: [1267397.553916][ T9715]  handle_mm_fault+0xba/0x280
Apr 11 07:20:24 nil kernel: [1267397.553917][ T9715]  do_user_addr_fault+0x1a4/0x5c0
Apr 11 07:20:24 nil kernel: [1267397.553919][ T9715]  ? schedule+0x3f/0xa0
Apr 11 07:20:24 nil kernel: [1267397.553920][ T9715]  exc_page_fault+0x5b/0x80
Apr 11 07:20:24 nil kernel: [1267397.553922][ T9715]  ? asm_exc_page_fault+0x8/0x30
Apr 11 07:20:24 nil kernel: [1267397.553924][ T9715]  asm_exc_page_fault+0x1e/0x30
Apr 11 07:20:24 nil kernel: [1267397.553925][ T9715] RIP: 0033:0x7f5a0f4e60f2
Apr 11 07:20:24 nil kernel: [1267397.553926][ T9715] Code: e9 fa ff 41 89 c0 4c 8b 54 24 18 8b 54 24 14 48 8b 74 24 08 8b 7c 24 10 b8 3d 00 00 00 0f 05 48 3d 00 f0 ff ff 77 31 44 89 c7 <89> 44 24 10 e8 25 ea fa ff 8b 44 24 10 48 83 c4 28 c3 0f 1f 40 00
Apr 11 07:20:24 nil kernel: [1267397.553927][ T9715] RSP: 002b:00007f59f5ffcae0 EFLAGS: 00010207
Apr 11 07:20:24 nil kernel: [1267397.553929][ T9715] RAX: 0000000000000000 RBX: 0000000000006581 RCX: 00007f5a0f4e60e7
Apr 11 07:20:24 nil kernel: [1267397.553930][ T9715] RDX: 0000000000000003 RSI: 00007f59f5ffcb1c RDI: 0000000000000000
Apr 11 07:20:24 nil kernel: [1267397.553931][ T9715] RBP: 00007f59f5ffcb30 R08: 0000000000000000 R09: 00007f59f0125940
Apr 11 07:20:24 nil kernel: [1267397.553931][ T9715] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f59f5ffcb1c
Apr 11 07:20:24 nil kernel: [1267397.553932][ T9715] R13: 0000000000006581 R14: 000056534e056ae8 R15: 0000000000000000

I think changing the HEAD made that go away. The only thing I can think of is that the HEAD version previously being compiled against a non-working unrarsrc, and that this made the ooms go away (since nothing's working)... I guess time will tell. rar2fs is "only" for my leisure time, of which I don't have much...

[hasse69]

Well when it comes to OOM it is a hint only and does not always necessarily mean or guarantee that the leak is in rar2fs, but it is triggered by something it does or tries to do but there are for some reason insufficient memory to do so. So it is a potential source of the problem but it might as well be something else that hogs the necessary resources. Only way to tell would be to monitor the memory usage per process to see which one(s) in fact seem to constantly grow. Valgrind was used rather recently to verify that there are no obvious leaks, but on the other hand there are use-cases that might not have been covered by those tests.

[tsjk]

Well, the oom was caused not because of system's resources running low, but because the memory in the cgroup for rar2fs (which is allowed some 20GB of memory) was consumed. So, I'm pretty sure it's rar2fs... But, one thing at a time now. :)

[hasse69]

Sure, but I am also pretty sure rar2fs does not just "leak" such a huge amount of resources unless something is in fact still in need of it or does not properly close resources like open file descriptors etc. But if it is related to the issue with thread termination then maybe..but that would only apply if you have plenty of compressed archives on the other hand.

[tsjk]

rar2fs seems to work with unrarsrc v6.0.3 as well.

[U] app-arch/unrar
     Available versions:  (~)5.9.4(0/5)^m[1] (~)6.0.3(0/6)^m[1] 6.1.7(0/6)
     Installed versions:  6.0.3(0/6)^m[1](12:07:29 2022-07-02)
     Homepage:            https://www.rarlab.com/rar_add.htm
     Description:         Uncompress rar files

[I] sys-fs/rar2fs
     Available versions:  (~)1.26.0^m[1] (~)1.29.5 (**)9999*l[1] {debug}
     Installed versions:  9999*l[1](12:07:53 2022-07-02)(-debug)
     Homepage:            https://hasse69.github.io/rar2fs/ https://github.com/hasse69/rar2fs
     Description:         A FUSE based filesystem that can mount one or multiple RAR archive(s)

And about the OOMs, I think you can definitely say that I have plenty of compressed archives.

[hasse69]

I have tried to reproduce the issue as reported and files unexpectedly showing up as folders but have not been able to reproduce. I assume when you tried this version of unrarsrc you did a scratch build and install of both libunrar.so and rar2fs?

[lazi3b0y]

I'm unable to build with unrar >= 6.0.1 (tried 6.0.3 and 6.1.7 as well) on my VM running Ubuntu Server 22.04, I tried with unrar 5.9.4 as OP suggested and that seems to build without issues for me.

The error I'm getting with unrar >= 6.0.1:

configure: creating ./config.status
config.status: creating Makefile
config.status: creating src/Makefile
config.status: creating man/Makefile
config.status: creating src/config.h
config.status: executing depfiles commands
Making all in src
make[1]: Entering directory '/tmp/tmp.n33dgvxPZb/rar2fs-1.29.0/src'
make  all-am
make[2]: Entering directory '/tmp/tmp.n33dgvxPZb/rar2fs-1.29.0/src'
g++ -DHAVE_CONFIG_H -I.  -I/tmp/tmp.n33dgvxPZb/unrar -D_FILE_OFFSET_BITS=64 -DFUSE_USE_VERSION=26 -DNDEBUG -DRARDLL -DRAR_SMP -Wall -Wno-reorder -g -O2  -MT dllext.o -MD -MP -MF .deps/dllext.Tpo -c -o dllext.o dllext.cpp
dllext.cpp: In function ‘size_t ListFileHeader(wchar*, Archive&)’:
dllext.cpp:384:9: error: no matching function for call to ‘itoa(int64&, wchar [20])’
  384 |     itoa(hd.UnpSize,UnpSizeText);
      |     ~~~~^~~~~~~~~~~~~~~~~~~~~~~~
In file included from /tmp/tmp.n33dgvxPZb/unrar/rar.hpp:30,
                 from dllext.cpp:31:
/tmp/tmp.n33dgvxPZb/unrar/strfn.hpp:43:6: note: candidate: ‘void itoa(int64, char*, size_t)’
   43 | void itoa(int64 n,char *Str,size_t MaxSize);
      |      ^~~~
/tmp/tmp.n33dgvxPZb/unrar/strfn.hpp:43:6: note:   candidate expects 3 arguments, 2 provided
/tmp/tmp.n33dgvxPZb/unrar/strfn.hpp:44:6: note: candidate: ‘void itoa(int64, wchar*, size_t)’
   44 | void itoa(int64 n,wchar *Str,size_t MaxSize);
      |      ^~~~
/tmp/tmp.n33dgvxPZb/unrar/strfn.hpp:44:6: note:   candidate expects 3 arguments, 2 provided
dllext.cpp:385:7: error: no matching function for call to ‘itoa(int64&, wchar [20])’
  385 |   itoa(hd.PackSize,PackSizeText);
      |   ~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /tmp/tmp.n33dgvxPZb/unrar/rar.hpp:30,
                 from dllext.cpp:31:
/tmp/tmp.n33dgvxPZb/unrar/strfn.hpp:43:6: note: candidate: ‘void itoa(int64, char*, size_t)’
   43 | void itoa(int64 n,char *Str,size_t MaxSize);
      |      ^~~~
/tmp/tmp.n33dgvxPZb/unrar/strfn.hpp:43:6: note:   candidate expects 3 arguments, 2 provided
/tmp/tmp.n33dgvxPZb/unrar/strfn.hpp:44:6: note: candidate: ‘void itoa(int64, wchar*, size_t)’
   44 | void itoa(int64 n,wchar *Str,size_t MaxSize);
      |      ^~~~
/tmp/tmp.n33dgvxPZb/unrar/strfn.hpp:44:6: note:   candidate expects 3 arguments, 2 provided
dllext.cpp:408:19: error: no matching function for call to ‘RarTime::GetText(wchar [50], long unsigned int, bool, bool)’
  408 |   hd.mtime.GetText(DateStr,ASIZE(DateStr),true,true);
      |   ~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /tmp/tmp.n33dgvxPZb/unrar/rar.hpp:19,
                 from dllext.cpp:31:
/tmp/tmp.n33dgvxPZb/unrar/timefn.hpp:53:10: note: candidate: ‘void RarTime::GetText(wchar*, size_t, bool)’
   53 |     void GetText(wchar *DateStr,size_t MaxSize,bool FullMS);
      |          ^~~~~~~
/tmp/tmp.n33dgvxPZb/unrar/timefn.hpp:53:10: note:   candidate expects 3 arguments, 4 provided
dllext.cpp:470:21: error: no matching function for call to ‘RarTime::GetText(wchar [50], long unsigned int, bool, bool)’
  470 |     hd.ctime.GetText(DateStr,ASIZE(DateStr),true,true);
      |     ~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /tmp/tmp.n33dgvxPZb/unrar/rar.hpp:19,
                 from dllext.cpp:31:
/tmp/tmp.n33dgvxPZb/unrar/timefn.hpp:53:10: note: candidate: ‘void RarTime::GetText(wchar*, size_t, bool)’
   53 |     void GetText(wchar *DateStr,size_t MaxSize,bool FullMS);
      |          ^~~~~~~
/tmp/tmp.n33dgvxPZb/unrar/timefn.hpp:53:10: note:   candidate expects 3 arguments, 4 provided
dllext.cpp:479:21: error: no matching function for call to ‘RarTime::GetText(wchar [50], long unsigned int, bool, bool)’
  479 |     hd.atime.GetText(DateStr,ASIZE(DateStr),true,true);
      |     ~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /tmp/tmp.n33dgvxPZb/unrar/rar.hpp:19,
                 from dllext.cpp:31:
/tmp/tmp.n33dgvxPZb/unrar/timefn.hpp:53:10: note: candidate: ‘void RarTime::GetText(wchar*, size_t, bool)’
   53 |     void GetText(wchar *DateStr,size_t MaxSize,bool FullMS);
      |          ^~~~~~~
/tmp/tmp.n33dgvxPZb/unrar/timefn.hpp:53:10: note:   candidate expects 3 arguments, 4 provided
make[2]: *** [Makefile:503: dllext.o] Error 1
make[2]: Leaving directory '/tmp/tmp.n33dgvxPZb/rar2fs-1.29.0/src'
make[1]: *** [Makefile:364: all] Error 2
make[1]: Leaving directory '/tmp/tmp.n33dgvxPZb/rar2fs-1.29.0/src'
make: *** [Makefile:386: all-recursive] Error 1

[tsjk]

I have tried to reproduce the issue as reported and files unexpectedly showing up as folders but have not been able to reproduce. I assume when you tried this version of unrarsrc you did a scratch build and install of both libunrar.so and rar2fs?

Well. You may have something there. When I noticed the error, it may be that unrarsrc was upgraded without rar2fs being recompiled. I just assumed downgrading would solve the issue when I observed it...

[hasse69]

@lazi3b0y To compile with unrarsrc 6.0.1 and later you need to be on at least this commit in the tree

https://github.com/hasse69/rar2fs/commit/6942480e202fa57592024506d3de09457abc4eb4

@tsjk You cannot safely just upgrade nor downgrade unrarsrc without doing a recompile of rar2fs. That is why I added static linking to become the default in order to in some extent mitigate some obvious/common scenarios. However, many distros seems to stick with the classic dynamic linking approach and that is even more error prone unfortunately since unrarsrc does not handle revisioning properly. Naming all versions, even if they introduce non-backward compatible changes as libunrar.so is asking for trouble. Unfortunately that is not within my control how unrarsrc build/packaging works. Anyhow, not rebuilding rar2fs might lead to all sorts of problems, wrong behavior, memory leaks or even crashes. Please report back after doing a full recompile towards unrarsrc and if problem persists, lets have another look at it.

[tfiskgul]

I also recently experienced the files-as-directories issue briefly on Gentoo, and solved it by a recompile of rar2fs.

[tsjk]

Yes, I know that I'd need to recompile. I think the problem is that ABI compatibility is not well defined for unrarsrc (as you point out), and so the build recipe for rar2fs cannot really assert that recompilation is needed after unrarsrc upgrades. I'll try to rebuild against the latest unrarsrc and see how it goes.

[lazi3b0y]

@lazi3b0y To compile with unrarsrc 6.0.1 and later you need to be on at least this commit in the tree

6942480

Ah, yup seems to build just fine when I actually use the latest version of rar2fs (I thought I was, but nope)

[hasse69]

Yes, I know that I'd need to recompile. I think the problem is that ABI compatibility is not well defined for unrarsrc (as you point out), and so the build recipe for rar2fs cannot really assert that recompilation is needed after unrarsrc upgrades. I'll try to rebuild against the latest unrarsrc and see how it goes.

Yes, and that is why I am careful with blaming the distros or maintainers for that matter. A non backward compatible change should normally be dealt with using the library version itself, such as libfoo.so.1.x.y vs e.g. libfoo.so.2.x.y. This is not how unrarsrc is delivered out from the package. This is, I would say, a significant part of the problem. Not saying it would be impossible to handle more seamlessly from the distro(s) but it is a somewhat black sheep since it is unaligned with common well defined praxis.

[hasse69]

Can we close this issue?

[tsjk]

Yes. Sorry for wasting time here.