.rarconfig not working

[lazuratus]

Starting at Patch release v1.29.1 .rarconfig files seem to no longer work.

Neither does alias = "/...", "/..." nor seek-length = 0

Went back to Stable release v1.29.0 and both of those features inside .rarconfig are working, now I just have issues with the Fix a regression with .NNN style volumes from v1.29.1

[hasse69]

I need some more info about how your mount command line looks like etc. Just tested using v1.29.4 and my regression tests pass. There are some details missing here that we need to be able to find the reason why it does not work for you any more.

[lazuratus]

System is running Debian GNU/Linux 9.13 (stretch)

Here is the installation script:

#!/bin/bash
#UNRARVER=6.0.3
UNRARVER=5.9.4
RAR2FSVER=1.29.0
WORKDIR=`mktemp -d` && cd $WORKDIR

# Get deps
sudo apt-get -q update
sudo apt-get -qy install make libfuse-dev g++

# Get, make and install unrar
wget https://www.rarlab.com/rar/unrarsrc-$UNRARVER.tar.gz
tar zxvf unrarsrc-$UNRARVER.tar.gz
cd unrar
make && sudo make install
make lib && sudo make install-lib
cd ..

# Get, make and install rar2fs
wget https://github.com/hasse69/rar2fs/releases/download/v$RAR2FSVER/rar2fs-$RAR2FSVER.tar.gz
tar zxvf rar2fs-$RAR2FSVER.tar.gz
cd rar2fs-$RAR2FSVER
./configure --with-unrar=../unrar --with-unrar-lib=/usr/lib/
make && sudo make install

# Cleanup
rm -rf $WORKDIR

Changing RAR2FSVER=1.29.0 to any version above 1.29.0 breaks the .rarconfig feature for me.

What else can I provide you with?

[hasse69]

We need the mount command line and preferably the .rarconfig (or a sample of it) that does not work, Feel free to "mask" any sensitive information in the file. For what it is worth I have tested all version from 1.29.0 to 1.29.4 and all of them pass regression. So something is "different" in your use-case that regression does not cover.

[lazuratus]

Everything is mounted using autofs

SMB shares are mounted via /etc/auto.cifs

Sample line:

/mnt/smb/srv1/plex  -fstype=cifs,ro,vers=3.0,noserverino,uid=1000,iocharset=utf8,credentials=/home/lazuratus/.smb_srv1  ://192.168.1.102/plex\$

Rar2FS shares are mounted via /etc/auto.rar2fs

Sample line:

/mnt/temp/rar2fs-1st-pass/srv1/plex  -fstype=fuse,ro,allow_other,noatime,umask=222,--seek-length=1  :rar2fs#/mnt/smb/srv1/plex
/mnt/rar2fs/srv1/plex                -fstype=fuse,ro,allow_other,noatime,umask=222,--seek-length=1  :rar2fs#/mnt/temp/rar2fs-1st-pass/srv1/plex

Sample .rarconfig lines:

[show.name.s01e00.special.episode.1.title.720p.web-ind.rar]
    alias = "/show.name.s01e00.special.episode.1.title.720p.web-ind.mkv", "/show.name.s00e01.special.episode.1.title.720p.web-ind.mkv"

[show.name.s01e00.special.episode.2.title.720p.web-ind.rar]
    alias = "/show.name.s01e00.special.episode.2.title.720p.web-ind.mkv", "/show.name.s00e02.special.episode.2.title.720p.web-ind.mkv"

After installing different versions of rar2fs I tried rebooting and restarting the autofs service but I get the same result for either method.

[hasse69]

There is nothing very "odd" with your use-case from what I can tell. It looks like a standard folder tree mount using --seek-length=1. I cannot really explain why this would not work. I can see that you point out the archive using

[show.name.s01e00.special.episode.1.title.720p.web-ind.rar]

So that archive is located in the root of the mount point? I would have guessed that it was residing under some sub-folder?

[lazuratus]

That archive is located inside a structure similar to this one:

/mnt/rar2fs/srv1/plex/TV Shows/Latest {Special}/Show Name (YEAR) {Special}/Season 0 {S}/Show.Name.S01E00.Special.Episode.1.Title.720p.web-ind/

Which doesn't seem to cause any issues using 1.29.0

[hasse69]

Sure, but I guess this is the "source" directory and not the actual mount point? Can you please reduce the test scope somewhat here and skip anything that has to do with autofs and manually mount what you believe is a representative use-case? The .rarconfig file must be located in the root of the mount point so that would be the start of the folder tree. At this moment I cannot explain why this suddenly has stopped working for you, Nothing has changed from 1.29.0 to 1.29.4 with respect to the .rarconfig function so I am bit baffled here. From the comment in rarconfig.example

# [<archive>|</some/dir/archive>]
#       seek-length = <n>
#       password = "<password>"
#       save-eof = [true|false]
#       alias = <"filename","alias">
#
# The optional path format of the archive specifier is an absolute path
# relative to the mount point root and not the soutce folder. For volumes
# the archive name must be that of the first file in the set

If this archive is not located in the root of the mount point rar2fs would not be able to find it? Using the first format might result in name collisions with other archives under a different sub-folder and thus the second syntax has been provided to work around such cases. But again, if this worked in v1.29.0 it was either due to plain luck or simply due to a bug in rar2fs. But as I just said, I cannot recall any changes what so ever in this area since v1.29.0.

Anyway, reduce the test case and lets try to figure out what is going on.

[hasse69]

Actually did a test of my own using a setup like

source/x.rar
source/vol/y.rar
source/vol/y.r01
...

And then mount source on a folder called d. It works fine without specifying [/vol/y.rar] and only [y.rar] in source/.rarconfig so I guess rar2fs is more clever than I remember. The second syntax is only needed if you have a confirmed name collisions I guess.

[lazuratus]

That was the actual mount point. Source in in /mnt/smb/[similar path] (mount root being /mnt/rar2fs/srv1/plex/)

I just tried mounting with rar2fs directly, and it seems to work :confused:

sudo mkdir /mnt/rar2fs/srv1/plex2
sudo chown lazuratus:lazuratus /mnt/rar2fs/srv1/plex2
rar2fs /mnt/smb/srv1/plex /mnt/rar2fs/srv1/plex2 --seek-length=1

I moved the show into the root of the mount point /mnt/rar2fs/srv1/plex2 for easier testing.

If I go to /mnt/rar2fs/srv1/plex2/Show.Name.S01E00.Special.Episode.1.Title.720p.web-ind/ .rarconfig works fine. If I go back to /mnt/rar2fs/srv1/plex/Show.Name.S01E00.Special.Episode.1.Title.720p.web-ind/ (plex not plex2) it again doesn't work.

Decided to do more tests with autofs, and found a strange behavior... commented most lines and added those 2:

/mnt/temp/rar2fs-1st-pass/srv1/plex  -fstype=fuse,ro,allow_other,noatime,umask=222,--seek-length=1  :rar2fs#/mnt/smb/srv1/plex
/mnt/t1stpass/srv1/plex              -fstype=fuse,ro,allow_other,noatime,umask=222,--seek-length=1  :rar2fs#/mnt/smb/srv1/plex

So far, .rarconfig seems to work inside /mnt/t1stpass/srv1/plex but not inside /mnt/temp/rar2fs-1st-pass/srv1/plex

Both rar2fs mount points are using /mnt/smb/srv1/plex as the source, but only one of them works with .rarconfig...

edit:

after a few sudo service autofs restart the mount with the working .rarconfig got switched :confused:

[hasse69]

Ok, Can you provide a manual mount case that does not work? Then contact me in private mail (you got it since before) and let me try to test locally using your exact archive.

[lazuratus]

Yeah, I'll do more tests later/tomorrow. As soon as I pinpoint a 100% working scenario, I'll get in touch.

It's just very weird that in those tests, after going back to 1.29.0, both mount points work 100% of the time after restarting the service MANY times...

[hasse69]

Yes I admit it is strange but many times strange things have a simple explanation.

[lazuratus]

Yeah, and because so far, all manual mount tests seem to work, I think I'll have to do a lot more tests with autofs tomorrow. Seems to be related to 1.29.1+ and autofs. It's not normal that sometimes it works after restarting the autofs service a couple of times.

[hasse69]

Since you have a rather clear case here that v1.29.0 works but v1.29.1 does not I believe it is rather safe to assume that one or more commits made in the delta causes something to break. After going through them all I cannot spot anything obvious. But it would be good if you could try them one by one, starting at the one preceding some of the more "suspicious" ones. I would suggest you start with checking out c3fd64f983630f4e3988313399522222efa88776.

# git checkout c3fd64f983630f4e3988313399522222efa88776

and build it from scratch. If that one works, start checking out commits one by one. You should be able to find which one it is that is relevant to this issue. It is not that many commits made after the one I mention here until v1.29.1 was released so it should be a fairly quick task.

[hasse69]

I went through all the changes again and I still see nothing that should really cause the problem you are experiencing. My guess now is that this is not about just a simple flaw in rar2fs but something more related to e.g. timing. The new version of rar2fs might be faster (or slower for that matter) causing something to break due to some currently unknown behaviour or limitation in autofs. That e.g. the alias and seek length options should stop working for no apparent reason at all is a bit far fetched to be honest. The .rarconfig file is simply opened and parsed at startup, even before the actual mount is performed. A bug in this area should be seen also for manual mounts and not only indirectly through e.g. autofs, So I have one more thing you can try. There is an option you can use to point rar2fs to another location for the .rarconfig file. Can you please try that (e.g. /tmp/rarconfig) and see if it has any effect on this issue? The only reason right now that I can think of is that something goes wrong with the access to your source folder and that rar2fs fails to read the .rarconfig file properly.

If this still fails then we might have to instrument rar2fs code a bit and throw some traces to the syslog in an attempt to understand more about what is going on here.

[lazuratus]

Ok, finaly had some time to do the tests. I created this new install script:

rm -rf rar2fs
git clone https://github.com/hasse69/rar2fs.git
cd rar2fs

#git checkout 011c77dbbde57a20869c442cb8c253a7ac8c3ba6
#git checkout cc5b2eabfeeccad113518d21e8f856d2629896e9
#git checkout 1401cf093baa289b334230d829688d00630c078c
#git checkout f2c83d1c4f0e0d103fcc2fb00ee9e3a66ec34951
#git checkout ea4c422b9858bd9e2358850651185aa4c2ff3138
#git checkout 611c987ef0e31fe0e48bd1bdb4204cfa3f9c173b
#git checkout 7f1db908ca66c771fac89d4c785d9389daa9a6ae
#git checkout 19285d1b45e70a82e9685097623fc6ebfef2e584
#git checkout c3fd64f983630f4e3988313399522222efa88776
#git checkout 3c723ce042a2cad9156fee5a86254530ba9a7fd4
#git checkout 3c4cea4c8b38da63aa4a8d4463c22d60cb42dea5
#git checkout 717b550e4f41e326902cfb22cf6214c313b52643
#git checkout 24bd1c31496cddc2240464e8a1d3275017069389
git checkout d56a2276583628ef95d6ca744bc5b15e28f9e009
#git checkout 41470f39d99e1e2db6ac35f11f7016a145f3cc51
#git checkout 1091bd7be8a5d4e3c7f162a9350894c88452b0ae

wget http://www.rarlab.com/rar/unrarsrc-5.9.4.tar.gz
tar -zxvf unrarsrc-5.9.4.tar.gz

cd unrar
make lib
sudo make install-lib
cd ..

autoreconf -f -i
./configure && make
sudo make install

I uncommented one git checkout line at a time, and did a few sudo service autofs restart after running the script.

Once I got to commit d56a2276583628ef95d6ca744bc5b15e28f9e009 the issue happened right after the first sudo service autofs restart so this is the commit that's causing my issue with the .rarconfig files.

As soon as I went back to previous commit (24bd1c31496cddc2240464e8a1d3275017069389) issue was gone.

Also, for the tests, I had 4 rar2fs mounts, plex_test_a, plex_test_b, plex_test_c & plex_test_d. All 4 of them are mounting the same location: (/mnt/smb/srv1/plex/), and the issue was appearing randomly. Sometimes just one of the 4, sometime 2, but always random as to which rar2fs mount didn't read/process the .rarconfig file.

[hasse69]

Did you try to point rar2fs to a common location for the config file?

[lazuratus]

Ok, so I added --config=/mnt/smb/srv1/plex/.rarconfig, did a few sudo service autofs restart and for a while it seemed to work, but when I tried other locations, problem was still there. So I tried adding --config=/mnt/smb/srv2/movies/.rarconfig to the mount line, but it didn't seem to work. And as always, going back to commit 24bd1c31496cddc2240464e8a1d3275017069389 all mounts work perfectly with the .rarconfig file (and no need to use --config=file option). I'm at a loss as to what else I can do/try/test....

Going to stick with commit 24bd1c31496cddc2240464e8a1d3275017069389 in the meantime, since it has the Fix a regression with .NNN style volumes fix, and .rarconfig files work. Will try to do more tests during the week/next weekend...

[hasse69]

Can you also try to send the rar2fs process a HUP signal after mount? It will re-read the configuration file.

Also, did you actually try to place the configuration file outside of the actual source tree that you intend to mount, like in /tmp? I still think that something is preventing rar2fs from accessing your source folder when trying to access the config file.

[hasse69]

Any news here? I still do not see how this issue can be related to how and when the config file is processed by rar2fs.

[lazuratus]

Was very busy and eventually ended forgetting about this issue because commit 24bd1c3 works great.

No idea how to send the rar2fs process a HUP signal after mount so I'll have to look into this over the weekend. As for place the configuration file outside of the actual source I'll give this a try asap.

[hasse69]

kill -HUP <pid>

The pid to use would be the lower of the ones ps will display for rar2fs since multiple threads are started.

[lazuratus]

Ok, had some time to do some tests.

Configuration file outside of the actual source tree: .rarconfig was in /home/user/.rarconfig and used --config=/home/user/.rarconfig in the autofs mount command.

... didn't seem to do anything, same behavior as before.

As for kill -HUP <pid>: I commented almost everything except the 4 test mounts (plex_test_a, plex_test_b, plex_test_c & plex_test_d)

I did: ps -C rar2fs and got a list of processes.

Using kill -HUP <lowest pid> didn't work, but when I ran kill -HUP on all of the IDs, every single test folder was showing to correct filenames from the .rarconfig file!!!

[hasse69]

Using kill -HUP <lowest pid> didn't work, but when I ran kill -HUP on all of the IDs, every single test folder was showing to correct filenames from the .rarconfig file!!!

Not sure I understand this. But I read it as it works if you ask rar2fs to reload the configuration?

This must be some access permission issue caused by autofs. I fail to reproduce this myself when not using it. I will try to add some more trace points in the code and make a patch you can try. It might be that I have to give up on this one unless it is a silly mistake somewhere that I have overlooked. Doubt it though.

[lazuratus]

Yeah, that's what it seems like... here's an example of what I did:

ps -C rar2fs

   PID TTY          TIME CMD
 37300 ?        00:00:00 rar2fs
 37318 ?        00:00:00 rar2fs
 37327 ?        00:00:00 rar2fs
 37337 ?        00:00:00 rar2fs
 37346 ?        00:00:00 rar2fs
 37356 ?        00:00:00 rar2fs

sudo kill -HUP 37300
sudo kill -HUP 37318
sudo kill -HUP 37327
sudo kill -HUP 37337
sudo kill -HUP 37346
sudo kill -HUP 37356

After this, all mount points are using the .rarconfig file correctly.

[hasse69]

I will try to find some time to add additional traces in the code. I guess another workaround would be to simply reload the config file after mount is created, basically loading it twice. Not something I really wish to do until root cause has been found though.

[hasse69]

No news here? Were you able to test the patch I sent you?

[hasse69]

After some feedback provided by the extra debug information I think it is rather clear that the issue is related to order of first- and second level stacked mount points. Also the fact that autofs cannot know there are dependencies between different mount points.

I do no really regard this as a bug in rar2fs but what we can do is to make sure directory style mounts delay read of the configuration until file system is confirmed to be mounted. For archive style mounts this is however not an option. That is why I do not really like such an approach.

Also I still do not fully understand how this could have worked previously since mounting second- before first level would always imply a race condition irrespective of when configuration is read.

[hasse69]

I would like to close this issue. As I see it this is a "user" problem rather than a bug. Using the --config option should make sure the configuration file is accessible also for stacked mounts subject to race conditions.

Any updates from your testing?

[lazuratus]

Also I still do not fully understand how this could have worked previously since mounting second- before first level would always imply a race condition irrespective of when configuration is read.

That is what's bugging me the most. I'll close this for now and continue using commit 24bd1c3 for now. When I have time to do more tests, if I discover anything new, I'll re-open this issue.

[hasse69]

I do not think you need to lock yourself to a specific version if you use the --config option. Seems like the perfect use-case for it.