search

hasse69 / rar2fs

FUSE file system for reading RAR archives
https://hasse69.github.io/rar2fs/
GNU General Public License v3.0
275 stars 27 forks source link

Some files show up as 8 exabyte (same as #142) #163

Closed jvwdev closed 3 years ago

jvwdev commented 3 years ago

OS: FreeBSD 13.0-RELEASE-p1 unrar: 6.02,6 fusefs-rar2fs: 1.29.5

I have this issue for a while with certain archives, the same issue as described in #142 Not sure what debug information could be helpful.

rar2fs -o allow_other,direct_io --seek-length=1 --exclude=.temp/

I'm running with direct_io otherwise my logs spams an error but everything stays functional without it, not sure if this is related.

kernel: fuse_vnop_lookup: cache incoherent on /mnt/rar2fs! Buggy FUSE server detected. To prevent data corruption, disable the data cache by mounting with -o direct_io, or as directed otherwise by your FUSE server's documentation

$ unrar lt some.rar

UNRAR 6.02 freeware      Copyright (c) 1993-2021 Alexander Roshal

Archive: some.rar
Details: RAR 4, volume

        Name: some.iso
        Type: File
        Size: ?
 Packed size: 249999898
       Ratio: -->
       mtime: 2021-05-18 06:03:14,964000000
  Attributes: .......
  Pack-CRC32: 470CCD17
     Host OS: Windows
 Compression: RAR 1.5(v29) -m1 -md=4M
hasse69 commented 3 years ago

That you are using direct i/o should not be relevant. That you are forced to use it however is. Have you contacted FreeBSD about this? I have never seen nor heard reports of this message before. Can you try some other FUSE file system like sshfs and compare? Can you try some old version of rar2fs to see if it behaves the same?

Regarding the size, the simplest way would be for you to provide me with a link to the archive.

jvwdev commented 3 years ago

Here is the link to the archive: https://drive.google.com/file/d/1p0pKho0QWWyEl2b8gS0Ev_6TuZy0jn6N/view?usp=sharing

I tried sshfs (which uses fusefs-libs3) and couldn't reproduce the error. I tried rar2fs without direct_io and also couldn't reproduce the error message. My logs say it started on Nov 13 2020, that was 1.29.1. The errors don't appear as often as I thought, 13 times with fuse_internal_do_getattr/fuse_vnop_lookup. It might but something weird that serviio, my dlna server, is doing as it scans/monitors the rar2fs mounted folder. It's not really a problem for me as everything works perfectly, I was just wondering if it was related.

I will remove the direct_io mount option and see if it happens again. If so I can try a version older then 1.29.1.

Thanks for the help!

hasse69 commented 3 years ago

Thanks for the link. I will try it out in a day or two. Not having access to my regular severs right now.

Regarding the issue reported by FreeBSD. Some searching on this issue reveals it is a known problem that started to appear with FreeBSD 12.2. I have not been able to fully understand if this is a problem with the FUSE implementation on FreeBSD or not but the two most frequent recommendations seems to suggest either the use of direct io (slow) or ask file system maintainers to migrate to FUSE3 that has an improved cache algorithm. The latter is currently not an option for rar2fs. I would suggest you continue this discussion towards FreeBSD. No one can really expect file systems to migrate due to an issue in the underlying OS. I would ask for a back port of FUSE3 logic to FUSE2 so that legacy FUSE2 file systems can operate seamlessly using both implementations also on FreeBSD 12.2 and later.

hasse69 commented 3 years ago

So, I have checked the file now and there is not much that can be done I am afraid. A properly generated compressed volume file (first) should look like this:

Details: RAR 4, volume

        Name: name_of_file
        Type: File
        Size: 722264064
 Packed size: 14999903
       Ratio: -->
       mtime: 2003-06-24 16:33,000
  Attributes: ..A....
  Pack-CRC32: AB6FB0DF
     Host OS: Windows
 Compression: RAR 3.0(v29) -m3 -md=4M

Note the Size attribute. It is a ? in your case which means it is not set properly. There is no way to know the size of the file unless we would first extract it since the packed size can not be accumulated in this case. That would only be possible for uncompressed archives (-m0) which is what the workaround relies on, I think your choice in this case is to extract the file and either keep it extracted or re-archive it using a tool confirming to the RAR specification.

jvwdev commented 3 years ago

Okay I wasn't aware it doesn't follow the RAR specification.

Regarding the errors in the log, I did still get them after mounting without direct_io. For now I will just use that mount option. I did find the patch which added this behavior: https://reviews.freebsd.org/D24012

fusefs: avoid cache corruption with buggy fuse servers

The FUSE protocol allows the client (kernel) to cache a file's size, if the server (userspace daemon) allows it. A well-behaved daemon obviously should not change a file's size while a client has it cached. But a buggy daemon might. If the kernel ever detects that that has happened, then it should invalidate the entire cache for that file. Previously, we would not only cache stale data, but in the case of a file extension while we had the size cached, we accidentally extended the cache with zeros.

And a bug report of fusefs-ntfs: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=246577

Not without a very detailed analysis of the problem. It's most likely a bug in the FUSE server, not the kernel. Since you said you did "du" while a copy was in progress, the problem might be a thread race in the server, but that's just a guess. To definitively answer the question would require a trace of all FUSE operations during your session.

Thank you for looking into both problems!