Issue with folder structure

[karibertils]

Using rar2fs v1.29.4-gitcad225 (DLL version 8)

Here's an example archive where only a folder with no contents shows up through rar2fs.

https://drive.google.com/file/d/1osy5JNvdbgl40Gi19UmRPfMBIJ_xE2Ig/view

Probably something weird in that rar archive, but it does extract properly so I think it should work.

I have few archives that are like that. Oddly I am using the same rar2fs version on another server where some of these same archive work. So pretty weird behaviour.

[hasse69]

Are you using the same libunrar version on these servers? Is the libunrar statically- or dynamically linked with rar2fs? Usually these kind of problems are caused by mismatching libunrar versions, i.e. what was used at build-time contra run-time. If libunrar is statically linked then that would not be an issue though.

Try mounting using -d and only mount the archive itself to reduce the output. You can also try to re-build rar2fs using --enable-debug=3 to get more details about what is going on.

I cannot reproduce the issue myself.

Also, any other differences between these servers of yours might be of interest.

[karibertils]

linunrar is statically linked. And it's the same executable on both servers.

I tested mounting a folder with only this archive now and it works fine on both servers. Then i remounted the original folder where the issue was occurring (affecting different archives on both servers, and only archives with folder structure inside). And it's also working for that mount on both. So it's a weird one. Maybe some issue with caching ? Rar2fs had been running and used heavily for 2 days before opening those folders where the issue occurred. But opening them after fresh mount everything seems fine in all cases.

I think we can just close this for now and I will reopen if it happens again or I find a way to reproduce it.

[hasse69]

Never seen or had reports of entries falling out of the cache like that. It does not sound likely since if the file cache entry got dropped it should be repopulated and also if it is not found you should see some error, not just an empty directory. Did you check if the . and .. entries were there? Might be if the directory cache for some reason got inconsistent. Let's see if you can reproduce and then we might need to instrument the code a bit to get some more info.

I think we can leave the issue open unless you fail to reproduce it.

[karibertils]

I did run find when it happend. Output was something like this:

./ARC1
./ARC1/media
./ARC2
./ARC2/media
./ARC3
./ARC3/media
./ARC4
./ARC4/media
./ARC5
./ARC5/media

It was showing the first level folder structure from inside the archives and after that empty. When listing inside the folders with find it was super fast like when the cache has been populated. When listing inside the folders the first time it was slow, so I did see when the cache was populated and the structure was same like above. I did not check specifically for . and .. but I was able to enter into the empty media folders and cd .. out from them.

After remounting all archives showed files like so:

./ARC1/media/etc/etc/etc/etc/etc/file.mkv

I have very few archives which contain folders, so might be very long time until I see this again. But I will notify if it happens.

[hasse69]

So in short you saw that even immediately after cache was populated only a small part of the directory structure was visible? So question is if the problem is at population itself or if it is lost after some time?

Also, the other archives you mention do they share the exact same directory structure? In that case it might be related.

[hasse69]

I see that archive path starts with an entry named "media", if the other archives does as well then it might be some cache collision problem that has passed unnoticed.

[hasse69]

Can you provide a link to one or more of the other archives as well?

[karibertils]

Yes I believe I saw it immediately after cache population since listing paused first time I noticed it. It's likely those archives had already been populated and dropped from the cache earlier.

Indeed all archives where I saw this have similar folder structure.

Here's link to the others. https://drive.google.com/file/d/1iqhlK2NPL5pVWIz9Tn6kULjS4txU475o/view

[hasse69]

I cannot seem to find any obvious problem here, thus I have not been able to reproduce. I have tried two different things; using original folder names (5 of them) and put them all in one source folder and also putting all archive files (.rar and .rNN) into the same source folder.

Can you try to mount a simple source folder only holding those five (5) archives on both your servers? I am starting to wonder if there is some differences in the environment that can cause this.

[karibertils]

Yeah, I have tried that as well, and so far unable to reproduce the issue on both servers.

[hasse69]

For what it's worth I have also tried to reduce the hash table to 1, thus all entries will collide. Still issue fails to reveal itself.

[hasse69]

Reproduced. I had my suspicions it might be related to invalidation rather than population of the cache. I will look into it and report back as soon as I have a solution.

[hasse69]

I have a solution now that seems to work ok with minimal impact on existing logic. I will merge the patch to master as soon as I have done some regression on it. Thanks for this issue report, turned out to be a rather severe bug 🤨

[karibertils]

No problem. IMO this bug is not that serious since those archives are real edge case and I have not noticed this happening otherwise. And I think your open source project has better support than most commercially available projects. So thanks for your work and kudos. :smile:

[hasse69]

Right. But the bug extended a bit further than only this use case. That is why I classify it as rather serious potentially leaving cache entries in limbo (memory leak).

[karibertils]

Ahh okey I see! That's Interesting. Would that happen only to cache entries where the issue presented itself, or potentially other entries as well ?

[hasse69]

Any entries in the file cache associated with an entry in the directory cache that is being invalidated due to e.g. renaming of parent folder.

[hasse69]

This issue should be fixed now.