search

hasse69 / rar2fs

FUSE file system for reading RAR archives
https://hasse69.github.io/rar2fs/
GNU General Public License v3.0
279 stars 27 forks source link

Build rar2fs with WinFSP layer? #75

Closed grimmthetallest closed 7 years ago

grimmthetallest commented 7 years ago

Is it possible to run rar2fs on top of the WinFSP FUSE API? This would allow use of rar2fs on Windows without additional abstraction through WSL BASH, etc.

Link to @billziss-gh 's WinFSP: https://github.com/billziss-gh/winfsp

Ideally this would allow rar2fs to access standard NTFS volumes on the host Windows PC and then add them back (decompressed) to the host system as a standard network mount.

billziss-gh commented 7 years ago

From a quick perusal of the rarfs code, it looks like a FUSE file system that should build on top of a POSIX system. I would expect that a port to Cygwin and WinFsp would be relatively straightforward.

I am happy to support this if someone decides to go ahead.

hasse69 commented 7 years ago

From what I can tell by looking at sshfs, if rar2fs would build under Cygwin it is all about writing the wrapper for WinFSP around it? But I might be wrong here of course. Getting rar2fs to build under Cygwin has been tried before, do no recall exactly why right now, but it was abandoned for some reason.

(btw fixed the link above since it was broken)

billziss-gh commented 7 years ago

@hasse69 Assuming that you have installed WinFsp and cygfuse [link] the tasks usually involve fixing the configure script for Cygwin and some occasional changes in the file system code (the usual one is disabling the stat call for the mountpoint since mountpoints do not exist prior to mounting on Windows).

I have done this exercise with sshfs, fuse-nfs, mp3fs and others and that is usually all I have to do.

hasse69 commented 7 years ago

Feel free to submit any patches for review needed in the rar2fs source tree to make this happen :) The source should be POSIX compliant, but be aware that you also need to make the UnRAR source tree (libunrar) to compile under cygwin which I think was a bit of a disappointment last time I tried. Or I was not stubborn enough to follow it up.

hasse69 commented 7 years ago

Btw, time for a stupid question... Why would WinFSP be needed if rar2fs builds under Cygwin and fuse for Cygwin?

billziss-gh commented 7 years ago

@hasse69 because FUSE for Cygwin is just a simple wrapper around WinFsp. The real FUSE functionality (including the necessary kernel mode driver, etc.) is provided by WinFsp.

hasse69 commented 7 years ago

Got it. So fuse for Cygwin simply abstracts the real API towards WinFSP... I noticed that WinFSP comes with a dual license, both open source and commercial. Will that cause an issue and what does the commercial license limit me from doing?

hasse69 commented 7 years ago

@grimmthetallest I will look into this, hopefully with some help from billziss-gh? But do not hold your breath on this one even though it looks/sounds promising. Now I need to do first aid on my old 64-bit windows machine if I can find it under all the dust :)

hasse69 commented 7 years ago

Hmm, MSI bummer... Module winfsp-x64.dll failed to register :(

billziss-gh commented 7 years ago

That is weird. Which OS are you running? Which version of WinFsp did you try to install?

If for some reason WinFsp 1.0 did not work, you can also try the latest build from here. This build is stable for our purposes, I only mark it "pre-release" because it contains a new layer for .NET.

billziss-gh commented 7 years ago

I noticed that WinFSP comes with a dual license, both open source and commercial. Will that cause an issue and what does the commercial license limit me from doing?

The WinFsp license is GPLv3 with a FLOSS exception. Since rarfs is also GPLv3, I do not see any issue.

@grimmthetallest I will look into this, hopefully with some help from billziss-gh?

You can certainly count on my support.

hasse69 commented 7 years ago

@billziss-gh Windows 10 64-bit

billziss-gh commented 7 years ago

Please try the latest WinFsp 1.1 and let me know if the problem persists.

I have had only one other report from someone telling me that DLL registration failed. Based on some of his/her feedback I have fixed a potential problem in 1.0. Unfortunately he/she never came back to confirm if the fix was adequate. And of course I was never able to reproduce this registration failure myself.

hasse69 commented 7 years ago

I got problems on both Windows10 32- and 64-bit, on the 64-bit machine it still fails with the same error but on the 32-bit machine it worked... but I guess I need to get the 64-bit version to work!

billziss-gh commented 7 years ago

@hasse69 do you get problems with both WinFsp 1.0 and 1.1?

In general you should not have any problems unless you have identified a bug with the WinFsp installer.

BTW, if you have DbgView on the machine that the installation fails you might see some additional information on the failure.

hasse69 commented 7 years ago

Problem solved, sandbox issues... Next problem I stumble into now is how to get GNU 'make' to work properly in cygwin :( The one that comes with the installation seems broken (no output at all) and the mingw one simply aborts with some Access Denied error when accessing /cygdrive/c/mingw/bin/, it worked for unrar however so it seems related to the auto generated Makefile

hasse69 commented 7 years ago

And make now works too...had to install it using aptcyg, and not using the setup file. So here it is....some warnings, probably can live with most of those but the linking fails...

$ make
make  all-am
make[1]: Entering directory '/home/Hugo/project/rar2fs.git'
g++ -DHAVE_CONFIG_H -I.  -I./unrar -I/usr/include/fuse -D_FILE_OFFSET_BITS=64 -DFUSE_USE_VERSION=26 -DNDEBUG -DRARDLL -DRAR_SMP -Wall -Wno-reorder -g -O2  -MT dllext.o -MD -MP -MF .deps/dllext.Tpo -c -o dllext.o dllext.cpp
mv -f .deps/dllext.Tpo .deps/dllext.Po
gcc -DHAVE_CONFIG_H -I.  -I./unrar -I/usr/include/fuse -D_FILE_OFFSET_BITS=64 -DFUSE_USE_VERSION=26 -DNDEBUG -Wall -fno-stack-protector -g -O2 -DGITREV_=0xd622dc1 -fno-omit-frame-pointer -Wextra -Wno-unused-result  -MT optdb.o -MD -MP -MF .deps/optdb.Tpo -c -o optdb.o optdb.c
mv -f .deps/optdb.Tpo .deps/optdb.Po
gcc -DHAVE_CONFIG_H -I.  -I./unrar -I/usr/include/fuse -D_FILE_OFFSET_BITS=64 -DFUSE_USE_VERSION=26 -DNDEBUG -Wall -fno-stack-protector -g -O2 -DGITREV_=0xd622dc1 -fno-omit-frame-pointer -Wextra -Wno-unused-result  -MT filecache.o -MD -MP -MF .deps/filecache.Tpo -c -o filecache.o filecache.c
mv -f .deps/filecache.Tpo .deps/filecache.Po
gcc -DHAVE_CONFIG_H -I.  -I./unrar -I/usr/include/fuse -D_FILE_OFFSET_BITS=64 -DFUSE_USE_VERSION=26 -DNDEBUG -Wall -fno-stack-protector -g -O2 -DGITREV_=0xd622dc1 -fno-omit-frame-pointer -Wextra -Wno-unused-result  -MT iobuffer.o -MD -MP -MF .deps/iobuffer.Tpo -c -o iobuffer.o iobuffer.c
mv -f .deps/iobuffer.Tpo .deps/iobuffer.Po
gcc -DHAVE_CONFIG_H -I.  -I./unrar -I/usr/include/fuse -D_FILE_OFFSET_BITS=64 -DFUSE_USE_VERSION=26 -DNDEBUG -Wall -fno-stack-protector -g -O2 -DGITREV_=0xd622dc1 -fno-omit-frame-pointer -Wextra -Wno-unused-result  -MT sighandler.o -MD -MP -MF .deps/sighandler.Tpo -c -o sighandler.o sighandler.c
mv -f .deps/sighandler.Tpo .deps/sighandler.Po
gcc -DHAVE_CONFIG_H -I.  -I./unrar -I/usr/include/fuse -D_FILE_OFFSET_BITS=64 -DFUSE_USE_VERSION=26 -DNDEBUG -Wall -fno-stack-protector -g -O2 -DGITREV_=0xd622dc1 -fno-omit-frame-pointer -Wextra -Wno-unused-result  -MT hashtable.o -MD -MP -MF .deps/hashtable.Tpo -c -o hashtable.o hashtable.c
mv -f .deps/hashtable.Tpo .deps/hashtable.Po
gcc -DHAVE_CONFIG_H -I.  -I./unrar -I/usr/include/fuse -D_FILE_OFFSET_BITS=64 -DFUSE_USE_VERSION=26 -DNDEBUG -Wall -fno-stack-protector -g -O2 -DGITREV_=0xd622dc1 -fno-omit-frame-pointer -Wextra -Wno-unused-result  -MT dircache.o -MD -MP -MF .deps/dircache.Tpo -c -o dircache.o dircache.c
mv -f .deps/dircache.Tpo .deps/dircache.Po
gcc -DHAVE_CONFIG_H -I.  -I./unrar -I/usr/include/fuse -D_FILE_OFFSET_BITS=64 -DFUSE_USE_VERSION=26 -DNDEBUG -Wall -fno-stack-protector -g -O2 -DGITREV_=0xd622dc1 -fno-omit-frame-pointer -Wextra -Wno-unused-result  -MT dirlist.o -MD -MP -MF .deps/dirlist.Tpo -c -o dirlist.o dirlist.c
mv -f .deps/dirlist.Tpo .deps/dirlist.Po
gcc -DHAVE_CONFIG_H -I.  -I./unrar -I/usr/include/fuse -D_FILE_OFFSET_BITS=64 -DFUSE_USE_VERSION=26 -DNDEBUG -Wall -fno-stack-protector -g -O2 -DGITREV_=0xd622dc1 -fno-omit-frame-pointer -Wextra -Wno-unused-result  -MT rarconfig.o -MD -MP -MF .deps/rarconfig.Tpo -c -o rarconfig.o rarconfig.c
mv -f .deps/rarconfig.Tpo .deps/rarconfig.Po
gcc -DHAVE_CONFIG_H -I.  -I./unrar -I/usr/include/fuse -D_FILE_OFFSET_BITS=64 -DFUSE_USE_VERSION=26 -DNDEBUG -Wall -fno-stack-protector -g -O2 -DGITREV_=0xd622dc1 -fno-omit-frame-pointer -Wextra -Wno-unused-result  -MT rar2fs.o -MD -MP -MF .deps/rar2fs.Tpo -c -o rar2fs.o rar2fs.c
In file included from /usr/include/fuse/fuse_common.h:26:0,
                 from /usr/include/fuse/fuse.h:26,
                 from rar2fs.c:39:
/usr/include/fuse/winfsp_fuse.h: In function ‘fsp_fuse_env’:
/usr/include/fuse/winfsp_fuse.h:367:19: warning: missing initializer for field ‘reserved’ of ‘struct fsp_fuse_env’ [-Wmissing-field-initializers]
     static struct fsp_fuse_env env = FSP_FUSE_ENV_INIT;
                   ^
/usr/include/fuse/winfsp_fuse.h:251:12: note: ‘reserved’ declared here
     void (*reserved[3])();
            ^
rar2fs.c: In function ‘work’:
rar2fs.c:5012:17: warning: implicit declaration of function ‘fuse_exited’ [-Wimplicit-function-declaration]
         while (!fuse_exited(f) && !wdt.work_task_exited) {
                 ^
rar2fs.c: In function ‘__handle_sigusr1’:
rar2fs.c:330:1: warning: visibility attribute not supported in this configuration; ignored [-Wattributes]
 }
 ^
rar2fs.c: In function ‘__handle_sighup’:
rar2fs.c:344:1: warning: visibility attribute not supported in this configuration; ignored [-Wattributes]
 }
 ^
mv -f .deps/rar2fs.Tpo .deps/rar2fs.Po
gcc -o rar2fs.exe dllext.o optdb.o filecache.o iobuffer.o sighandler.o hashtable.o dircache.o dirlist.o rarconfig.o rar2fs.o  -rdynamic    -lfuse ./unrar/libunrar.a  -lstdc++ -ldl -lrt
rar2fs.o: In function `fsp_fuse_signal_thread':
/usr/include/fuse/winfsp_fuse.h:293: undefined reference to `__imp_fsp_fuse_signal_handler'
/usr/include/fuse/winfsp_fuse.h:293:(.text+0x9a): relocation truncated to fit: R_X86_64_PC32 against undefined symbol `__imp_fsp_fuse_signal_handler'
rar2fs.o: In function `fuse_opt_add_arg':
/usr/include/fuse/fuse_opt.h:83: undefined reference to `__imp_fsp_fuse_opt_add_arg'
/usr/include/fuse/fuse_opt.h:83:(.text+0x721): relocation truncated to fit: R_X86_64_PC32 against undefined symbol `__imp_fsp_fuse_opt_add_arg'
rar2fs.o: In function `fuse_main_real':
/usr/include/fuse/fuse.h:123: undefined reference to `__imp_fsp_fuse_main_real'
/usr/include/fuse/fuse.h:123:(.text+0x74d): relocation truncated to fit: R_X86_64_PC32 against undefined symbol `__imp_fsp_fuse_main_real'
rar2fs.o: In function `fuse_opt_add_arg':
/usr/include/fuse/fuse_opt.h:83: undefined reference to `__imp_fsp_fuse_opt_add_arg'
/usr/include/fuse/fuse_opt.h:83:(.text+0x78e): relocation truncated to fit: R_X86_64_PC32 against undefined symbol `__imp_fsp_fuse_opt_add_arg'
rar2fs.o: In function `fuse_main_real':
/usr/include/fuse/fuse.h:123: undefined reference to `__imp_fsp_fuse_main_real'
/usr/include/fuse/fuse.h:123:(.text+0x7ba): relocation truncated to fit: R_X86_64_PC32 against undefined symbol `__imp_fsp_fuse_main_real'
rar2fs.o: In function `fuse_loop':
/usr/include/fuse/fuse.h:153: undefined reference to `__imp_fsp_fuse_loop'
/usr/include/fuse/fuse.h:153:(.text+0x9a4): relocation truncated to fit: R_X86_64_PC32 against undefined symbol `__imp_fsp_fuse_loop'
rar2fs.o: In function `fuse_loop_mt':
/usr/include/fuse/fuse.h:160: undefined reference to `__imp_fsp_fuse_loop_mt'
/usr/include/fuse/fuse.h:160:(.text+0x9c2): relocation truncated to fit: R_X86_64_PC32 against undefined symbol `__imp_fsp_fuse_loop_mt'
rar2fs.o: In function `fuse_opt_parse':
/usr/include/fuse/fuse_opt.h:75: undefined reference to `__imp_fsp_fuse_opt_parse'
/usr/include/fuse/fuse_opt.h:75:(.text.startup+0xa1): relocation truncated to fit: R_X86_64_PC32 against undefined symbol `__imp_fsp_fuse_opt_parse'
rar2fs.o: In function `fuse_version':
/usr/include/fuse/fuse_common.h:100: undefined reference to `__imp_fsp_fuse_version'
/usr/include/fuse/fuse_common.h:100:(.text.startup+0x1ce): relocation truncated to fit: R_X86_64_PC32 against undefined symbol `__imp_fsp_fuse_version'
rar2fs.o: In function `fuse_opt_add_arg':
/usr/include/fuse/fuse_opt.h:83: undefined reference to `__imp_fsp_fuse_opt_add_arg'
/usr/include/fuse/fuse_opt.h:83:(.text.startup+0x207): relocation truncated to fit: R_X86_64_PC32 against undefined symbol `__imp_fsp_fuse_opt_add_arg'
rar2fs.o: In function `fuse_parse_cmdline':
/usr/include/fuse/fuse_common.h:121: undefined reference to `__imp_fsp_fuse_parse_cmdline'
/usr/include/fuse/fuse_common.h:121:(.text.startup+0x302): additional relocation overflows omitted from the output
rar2fs.o: In function `fuse_opt_free_args':
/usr/include/fuse/fuse_opt.h:97: undefined reference to `__imp_fsp_fuse_opt_free_args'
rar2fs.o: In function `fuse_mount':
/usr/include/fuse/fuse_common.h:107: undefined reference to `__imp_fsp_fuse_mount'
rar2fs.o: In function `fuse_new':
/usr/include/fuse/fuse.h:138: undefined reference to `__imp_fsp_fuse_new'
rar2fs.o: In function `work':
/home/Hugo/project/rar2fs.git/rar2fs.c:5012: undefined reference to `fuse_exited'
rar2fs.o: In function `fuse_unmount':
/usr/include/fuse/fuse_common.h:114: undefined reference to `__imp_fsp_fuse_unmount'
rar2fs.o: In function `fuse_destroy':
/usr/include/fuse/fuse.h:146: undefined reference to `__imp_fsp_fuse_destroy'
rar2fs.o: In function `fuse_unmount':
/usr/include/fuse/fuse_common.h:114: undefined reference to `__imp_fsp_fuse_unmount'
collect2: error: ld returned 1 exit status
make[1]: *** [Makefile:506: rar2fs.exe] Error 1
make[1]: Leaving directory '/home/Hugo/project/rar2fs.git'
make: *** [Makefile:404: all] Error 2
hasse69 commented 7 years ago

So what do I need to link towards to resolve the symbols?

This one needs some investigation:

rar2fs.c: In function ‘work’:
rar2fs.c:5012:17: warning: implicit declaration of function ‘fuse_exited’ [-Wimplicit-function-declaration]
         while (!fuse_exited(f) && !wdt.work_task_exited) {

The fuse_exited function seems to be missing in the cygwin port?

The attribute warning I have fixed...

The rest seems related to WinFSP?

billziss-gh commented 7 years ago

The fuse_exited function seems to be missing in the cygwin port?

Indeed that seems to be a real problem. I have no idea how this was missed from the port. I will look into adding it.

Perhaps you can fake it for now with something along the lines of

#define fuse_exit(f) (0)
/usr/include/fuse/winfsp_fuse.h:293: undefined reference to `__imp_fsp_fuse_signal_handler'
...

Ok, it looks like you are linking directly with WinFsp rather than through cygfuse. I recommend installing cygfuse first. You can install on your local Cygwin using this command line:

$ tar -C/ -xaf fuse-2.8-*.tar.xz

Once you have cygfuse installed, configure should pick up the correct headers/libraries using pkg-config. If not you can also run pkg-config fuse --cflags --libs to pick them up.

hasse69 commented 7 years ago

Yea, I can stub the fuse_exited function for now. But the linking part, I am not too sure...I have already installed cygfuse, but it might be that I need to point rar2fs to the library under /usr/lib? But I have checks for that in configure.ac, it found -lfuse and then that would not be WinFSP, would it? Will check using pkg-config...

pkg-config seems to point me in the direction of -lfuse-2.8? I cannot really understand how it then could find -lfuse during configure? Also the -DCYGFUSE is missing in my build.

billziss-gh commented 7 years ago

pkg-config seems to point me in the direction of -lfuse-2.8? I cannot really understand how it then could find -lfuse during configure? Also the -DCYGFUSE is missing in my build.

I think the missing -DCYGFUSE is the reason this is trying to link directly against WinFsp rather than through cygfuse. Also WinFsp emulates FUSE 2.8, hence the -lfuse-2.8.

It sounds like this may be a problem with the configure script.

hasse69 commented 7 years ago

Well, it did find -lfuse, no magic there...I do not have that many libraries to chose from... Let's see what -DCYGFUSE will do for us...

billziss-gh commented 7 years ago

BTW, after cloning the repo and trying to configure myself I got:

configure: error:
        Cannot find fuse.h - add alternative include search path using
        --with-fuse and rerun configure.

So it does look like a problem with configure.

[Leaving for family dinner, but should be able to help with this more later.]

hasse69 commented 7 years ago

Yea, there is a flaw in the cygfuse package, fuse installs a wrapper fuse.h include file (for backwards compatibility) in /usr/include, linking to /usr/include/fuse/fuse.h. Since that is missing you need to configure using --with-fuse=/usr/include/fuse

billziss-gh commented 7 years ago

It appears that the configure script does not use pkg-config. [I could be wrong as I am not an expert in autoconf.]

I am wondering if a command line like this could work:

$ CFLAGS=-DCYGFUSE ./configure --with-fuse=/usr/include/fuse --with-fuse-lib=/usr/lib
hasse69 commented 7 years ago

I do not think you need the --with-fuse-lib

billziss-gh commented 7 years ago

In that case passing the -DCYGFUSE to the C compiler should do the trick. I am missing the unrar dependency on my Cygwin. Do you know which Cygwin package satisfies it?

hasse69 commented 7 years ago

You need to download it from here http://www.rarlab.com/rar/unrarsrc-5.5.3.tar.gz Then un-tar it in the source folder of rar2fs. Then do make lib. Note that the makefile for libunrar uses -fPIC, does not work well with cygwin so just remove it... After the build completes rar2fs configure will find it automatically. By default it links to it statically so do not worry about where to install it.

hasse69 commented 7 years ago

Success

$ make
make  all-am
make[1]: Entering directory '/home/Hugo/project/rar2fs.git'
gcc -DHAVE_CONFIG_H -I.  -I./unrar -DCYGFUSE -I/usr/include/fuse -D_FILE_OFFSET_BITS=64 -DFUSE_USE_VERSION=26 -DNDEBUG -Wall -fno-stack-protector -g -O2 -DGITREV_=0xd622dc1 -fno-omit-frame-pointer -Wextra -Wno-unused-result  -MT rar2fs.o -MD -MP -MF .deps/rar2fs.Tpo -c -o rar2fs.o rar2fs.c
In file included from /usr/include/fuse/fuse_common.h:26:0,
                 from /usr/include/fuse/fuse.h:26,
                 from rar2fs.c:39:
/usr/include/fuse/winfsp_fuse.h: In function ‘fsp_fuse_env’:
/usr/include/fuse/winfsp_fuse.h:367:19: warning: missing initializer for field  reserved’ of ‘struct fsp_fuse_env’ [-Wmissing-field-initializers]
     static struct fsp_fuse_env env = FSP_FUSE_ENV_INIT;
                   ^
/usr/include/fuse/winfsp_fuse.h:251:12: note: ‘reserved’ declared here
     void (*reserved[3])();
            ^
mv -f .deps/rar2fs.Tpo .deps/rar2fs.Po
gcc -o rar2fs.exe dllext.o optdb.o filecache.o iobuffer.o sighandler.o hashtable.o dircache.o dirlist.o rarconfig.o rar2fs.o  -rdynamic    -lfuse-2.8 ./unrar/libunrar.a  -lstdc++ -ldl -lrt
gcc -DHAVE_CONFIG_H -I.  -I./unrar -DCYGFUSE -I/usr/include/fuse -D_FILE_OFFSET_BITS=64 -DFUSE_USE_VERSION=26 -DNDEBUG -Wall -fno-stack-protector -g -O2 -DGITREV_=0xd622dc1 -fno-omit-frame-pointer -Wextra -Wno-unused-result  -MT mkr2i.o -MD -MP -MF .deps/mkr2i.Tpo -c -o mkr2i.o mkr2i.c
mv -f .deps/mkr2i.Tpo .deps/mkr2i.Po
gcc -o mkr2i.exe mkr2i.o -rdynamic -ldl -lrt
make[1]: Leaving directory '/home/Hugo/project/rar2fs.git'

It does not matter if you link to -lfuse or -lfuse-2.8, the linker is clever enough to figure out that -lfuse is in fact /usr/lib/libfuse-2.8, similar to how the run-time linker resolves libraries I guess.

hasse69 commented 7 years ago

But now things happen I have no clue about ;)

$ ./rar2fs.exe -f s d
Cannot create WinFsp-FUSE file system: mount point in use.

Hardly in use :)

Time for some sleep. If you have any information on how we proceed from here please share it. (but I am impressed, this is the first time I managed to build rar2fs in cygwin)

Btw, you mentioned something about stat(2) system calls? Do I need to replace them with something else or change the type from struct stat to struct fuse_stat?

hasse69 commented 7 years ago

Btw, here is the /usr/include/fuse.h file missing in cygfuse...

/*
   This header is for compatibility with older software using FUSE.

   Please use 'pkg-config --cflags fuse' to set include path.  The
   correct usage is still '#include <fuse.h>', not '#include
   <fuse/fuse.h>'.
*/

#include "fuse/fuse.h"
billziss-gh commented 7 years ago

Congratulations!

[I am back from dinner and had a couple so...]

Btw, here is the /usr/include/fuse.h file missing in cygfuse...

I seem to recall omitting this purposefully since WinFsp-FUSE provides FUSE 2.8 functionality. But it is probably worthwhile to add for compatibility reasons.

There is also the fuse_exited problem that I need to fix.

But now things happen I have no clue about ;)

$ ./rar2fs.exe -f s d
Cannot create WinFsp-FUSE file system: mount point in use.

Mounting behavior is different between UNIX and Windows. On Windows the mount point should not exist prior to mounting the file system.

To mount on a drive X: this should work:

$ ./rarfs -f source X:

To mount on a directory dir this should work (the directory should not exist prior to launching rarfs):

$ ./rarfs -f source dir

To unmount the file system press ^C if you are running it with -f or simply kill it otherwise. For example:

$ pkill rarfs

Btw, you mentioned something about stat(2) system calls? Do I need to replace them with something else or change the type from struct stat to struct fuse_stat?

You do not have to do anything special about your file system's getattr. However some file systems try to stat the mount point prior to mounting. Because in WinFsp-FUSE the mount point should not exist prior to mounting the stat fails. So in general one has to #ifdef the stat call out on Windows.

billziss-gh commented 7 years ago

WORKS! 🎉 🎉 🎉

capture

I rar'ed the rar2fs source code and placed it into a file rar2fs.rar for this test. Then launched it with the command line:

$ ./rar2fs.exe -f ~/Desktop/rar2fs.rar y:

You can see the results above.

To get it to compile I built libunrar as per your instructions and then configured with ./configure CFLAGS=-DCYGFUSE --with-fuse=/usr/include/fuse. Then finally the following hackpatch.

diff --git a/rar2fs.c b/rar2fs.c                                                                      
index 1ce53e7..24f06ed 100644                                                                         
--- a/rar2fs.c                                                                                        
+++ b/rar2fs.c                                                                                        
@@ -75,6 +75,8 @@                                                                                     
 #include "dirlist.h"                                                                                 
 #include "rarconfig.h"                                                                               

+#define fuse_exited(f) (0)                                                                           
+                                                                                                     
 #define MOUNT_FOLDER  0                                                                              
 #define MOUNT_ARCHIVE 1                                                                              

@@ -4651,10 +4653,10 @@ static int check_paths(const char *prog, char *src_path_in, char *dst_path_in,
         char p1[PATH_MAX];                                                                           
         char p2[PATH_MAX];                                                                           
         char *a1 = realpath(src_path_in, p1);                                                        
-        char *a2 = realpath(dst_path_in, p2);                                                        
+        char *a2 = dst_path_in;//realpath(dst_path_in, p2);                                          
         if (!a1 || !a2 || !strcmp(a1, a2)) {                                                         
                 if (verbose)                                                                         
-                        printf("%s: invalid source and/or mount point\n", prog);                     
+                        printf("%s: <1>invalid source and/or mount point\n", prog);                  
                 return -1;                                                                           
         }                                                                                            
         dir_list_open(arch_list);                                                                    
@@ -4667,12 +4669,14 @@ static int check_paths(const char *prog, char *src_path_in, char *dst_path_in,
                 blkdev_size = get_blkdev_size(&st);                                                  

         /* Check path type(s), destination path *must* be a folder */                                
+#if !defined(__CYGWIN__)                                                                             
         (void)stat(a2, &st);                                                                         
         if (!S_ISDIR(st.st_mode)) {                                                                  
                 if (verbose)                                                                         
-                        printf("%s: invalid source and/or mount point\n", prog);                     
+                        printf("%s: <2>invalid source and/or mount point\n", prog);                  
                 return -1;                                                                           
         }                                                                                            
+#endif                                                                                               

         /* Check file collection at archive mount */                                                 
         if (mount_type == MOUNT_ARCHIVE) {                                                           
@@ -4683,7 +4687,7 @@ static int check_paths(const char *prog, char *src_path_in, char *dst_path_in,  
                         return -1;                                                                   
                 } else if (ret <= 0) {                                                               
                         if (verbose)                                                                 
-                                printf("%s: invalid source and/or mount point\n",                    
+                                printf("%s: <3>invalid source and/or mount point\n",                 
                                                 prog);                                               
                         return -1;                                                                   
                 }
hasse69 commented 7 years ago

This is great news :) I need to go through the changes needed in configure.ac to automatically detect the need for CYGFUSE and the code changes. But this was almost too easy? Maybe you can look at the other warning(s) related to WinFSP, like for it to compile clean :)

hasse69 commented 7 years ago

Btw, why should a user need to define CYGFUSE? To me it looks like it should be done in fuse.h (indirectly through winfsp_fuse.h or directly) in the cygfuse package that should do that? _EDIT: I added define CYGFUSE to winfspfuse.h and it works just fine, not need for a user to set it manually. Can you look into it?

It seems to work fine if I mount a local folder inside the cygwin environment to e.g. W:, but if I try something like:

$ rar2fs -f /cygdrive/z w:

I get access/permission problems from Windows when trying to access W:. Any clue why that migth be the case? I guess a common use-case would be to have files mounted to an external drive and then mount it on another through rar2fs.

Also, since WinFSP mounts the source as a regular disk and not a network volume it means it cannot be unmounted from Windows itself. Is this something that can be looked into?

billziss-gh commented 7 years ago

I need to go through the changes needed in configure.ac to automatically detect the need for CYGFUSE and the code changes. But this was almost too easy?

Maybe you have just done a good job in making rar2fs portable? :)

BTW, the changes we had to do in rar2fs are in general the changes I have to do on most FUSE file systems to get them to build / run.

Maybe you can look at the other warning(s) related to WinFSP, like for it to compile clean :)

Yes. In fact I thought I had fixed those, not sure why they showed up again.

Btw, why should a user need to define CYGFUSE? To me it looks like it should be done in fuse.h (indirectly through winfsp_fuse.h or directly) in the cygfuse package that should do that?

The reason is that the same headers can be used on Cygwin to link directly with WinFsp (i.e. winfsp-x64.dll). Before I created cygfuse to make things easier on Cygwin one could reference the include/lib files directly from the WinFsp installation directory. This was of course awkward for Cygwin programs, so I eventually created cygfuse. But I want to keep both methods available.

It seems to work fine if I mount a local folder inside the cygwin environment to e.g. W:, but if I try something like:

$ rar2fs -f /cygdrive/z w:

I get access/permission problems from Windows when trying to access W:.

Not sure. It could be that the "access denied" is coming from rar2fs itself (e.g. if you are trying to open /cygdrive/z: I am not sure what Cygwin tries to open). It may also be coming from the permission checks that the FUSE layer does. You can try running with the option -o uid=-1,gid=-1, which is a special instruction to WinFsp-FUSE to pretend that the current user has all permissions (kind of like doing geteuid, getegid on UNIX -- of course if you are already doing something similar in your code, this should not be necessary). Finally you can also try -o umask=000 to allow all access to everyone.

Also, since WinFSP mounts the source as a regular disk and not a network volume it means it cannot be unmounted from Windows itself. Is this something that can be looked into?

To unmount use kill / pkill as discussed earlier. This unmounts cleanly. However if you want to make a file system into a network volume, use the --VolumePrefix=\server\share switch (notice the single backslash after the =).

Finally there is a way to make the file system work fully with net use. Here is a document that explains how: WinFsp Service Architecture.

hasse69 commented 7 years ago

Right. I will add automatic detection of CYGFUSE in configure.ac. Regarding the permission issues, the problem is most likely not in rar2fs, it works if I mount on a local folder in the cygwin rootfs, it is only if I mount as a drive in Windows I get the problem, and only if I mount from a cygdrive to a Windows drive. Mounting a local folder onto a drive works fine.

Correction, is does not work at all to mount to a local directory, the target folder becomes d -> /cygdrive/c/Device/Volume{2ed161b7-37f4-11e7-bf10-94de800663a1}/ which does not exist :(

Using -o umask=000 solves the mounting to a disk drive, uid=1 and gid=1 does not. Also, none of these options seems to make it possible to mount to a local cygwin folder, it cannot be found due to the strange soft linke created above :(

And I did not really get the --VolumePrefix=\server\share? Where should I add that?

billziss-gh commented 7 years ago

Correction, is does not work at all to mount to a local directory, the target folder becomes d -> /cygdrive/c/Device/Volume{2ed161b7-37f4-11e7-bf10-94de800663a1}/ which does not exist :(

Yes, this is a problem with Cygwin. Unfortunately it does not recognize Windows "junctions" (which is what Windows uses for mounting). Windows "junctions" were one of the original "reparse points" (predating the newer Windows "symlinks") so this is rather unfortunate.

These junctions will work from Windows apps though. Try accessing a file from Notepad or cmd.exe.

One extra caveat with junctions. Due to a Windows limitation file systems that are mounted through junctions should in general be case-insensitive or things may not work correctly (Windows has the unfortunate tendency of mixing the file name case when going through a junction). This is only true with junctions, when a file system is mounted as a drive, the file system can be case-sensitive or case-insensitive without additional issues.

Using -o umask=000 solves the mounting to a disk drive, uid=1 and gid=1 does not.

To clarify the correct switch is -o uid=-1,gid=-1, not -o uid=1,gid=1 (notice the minus -).

And I did not really get the --VolumePrefix=\server\share? Where should I add that?

Rar2fs does its own command line processing so this does not work currently. However the WinFsp-FUSE layer accepts some extra command line parameters. Unfortunately when I was adding this functionality I did not think everything through and made this --options instead of -o options.

Perhaps rar2fs could be changed to pass the following parameter to the WinFsp-FUSE layer and the file system would then be mounted as a network drive automagically:

$ ./rar2fs -f ~/Desktop/rar2fs.rar y: '--VolumePrefix=\rar2fs\test'

This would mount the file system on the y: drive, but the y: drive would also be available under the UNC syntax \\rar2fs\test.

hasse69 commented 7 years ago

Perhaps rar2fs could be changed to pass the following parameter to the WinFsp-FUSE layer and the file system would then be mounted as a network drive automagically:

I am not too sure that is possible. You are right in that rar2fs process its own arguments but the ones not consumed by rar2fs is by API design passed on to fuse which will probably not allow them to be passed on to the next layer. But I saw you already have plans to add -o equivalent options for this and other switches which is most likely the best option.

hasse69 commented 7 years ago

@grimmthetallest The commit 609bff876f2a74ef3b2d4d91bc8ae0ccae9907a8 should now provide everything necessary to build rar2fs on top of WinFSP + cygwin/cygfuse. Follow the instructions linked to in this thread and report back with status and/or questions.

billziss-gh commented 7 years ago

@hasse69 thanks for the quick turnaround on this!

hasse69 commented 7 years ago

Well, I guess my efforts making rar2fs portable on a variety of platforms in combination with your amazing fuse API emulation layer made this more or less a walk in the park ;) I am just a bit disappointed I did not become aware of this sooner.

billziss-gh commented 7 years ago

I can confirm that I can build rar2fs with these commands:

$ wget http://www.rarlab.com/rar/unrarsrc-5.5.3.tar.gz
$ tar xzf unrarsrc-5.5.3.tar.gz
$ sed s/-fPIC//g -i unrar/makefile
$ make -C unrar lib
$ autoreconf -i
$ ./configure --with-fuse=/usr/include/fuse
$ make

However trying to run it:

$ ./rar2fs ~/Desktop/rar2fs.rar y:
./rar2fs: invalid source and/or mount point
$ ./rar2fs ~/Desktop/rar2fs.rar foo
./rar2fs: invalid source and/or mount point

I think the issue is that rar2fs still does a stat on the mountpoint: [link]

I added rar2fs to the known file systems for WinFsp. THANKS!

hasse69 commented 7 years ago

Oh? That was interesting. I intentionally left out on parts of your patch because I never got the problem when trying it myself. I only removed the call to realpath and then it worked for me. I wonder why. Need to add another patch for this then,,,

hasse69 commented 7 years ago

@grimmthetallest Use this commit instead cf2298f7c212207c0408eba987fe2dae9938fbc0 It fixes the problem with the target folder under cygwin/cygfuse.

billziss-gh commented 7 years ago

With the latest commit I can now run it successfully:

$ ./rar2fs ~/Desktop/rar2fs.rar y:
rar2fs: PID 2736: mounted y:
$ ls -l /cygdrive/y
total 544
...
$ pkill -9 rar2fs

A couple of things:

billziss-gh commented 7 years ago
  • I notice that to kill the file system, I have to do kill -9 (when daemonized). I am assuming you set a signal handler that disables SIGINT, SIGTERM.

Or maybe this is an artifact that fuse_exited is currently a stub.

hasse69 commented 7 years ago

Since your file system is read-only and usually runs of a disk file/directory, you have no file system consistency issues (I think) and you may be able to enable full caching for the file system.

Since rar2fs is an overlay fs and forwards requests to its back-end it is not completely read-only and implements its own cache logic. So when you mount a folder you can definitely mount it read/write. When you mount a file it becomes read-only by design.

I notice that to kill the file system, I have to do kill -9 (when daemonized). I am assuming you set a signal handler that disables SIGINT, SIGTERM.

No, rar2fs does not install handlers for SIGINT and SIGTERM. It leaves the responsibility to fuse to handle those signals. Signal handling in fuse is a bit tricky, that was the reason I split the processing loop and check for fuse_exited() specifically. From the rar2fs loop:

        /*
         * This is a workaround for an issue with fuse_loop() that does
         * not always release properly at reception of SIGINT.
         * But this is really what we want since this thread should not
         * block blindly without some user control.
         */
        while (!fuse_exited(f) && !wdt.work_task_exited) {
                sleep(1);
                ++rar2_ticks;
        }

Or maybe this is an artifact that fuse_exited is currently a stub.

Could be. But I cannot say for sure.

EDIT: Removing the call to fuse_exited() and only looking at the flag set after fuse_loop_mt() exits does not make any difference. This means that fuse_loop()/_mt() does not exit properly when demonized. I think this was a similar problem with native fuse and for that reason the fuse_exited flag was checked since file system was internally unmounted but the user was never informed resulting in a hanging mount point needing a fusermount -zu. However, in the case of cygfuse I am not sure it matters because the SIGKILL seems to cleanup properly.

grimmthetallest commented 7 years ago

Use this commit instead cf2298f It fixes the problem with the target folder under cygwin/cygfuse.

I will test this today. Thanks to you both for the quick work!!

Is there an easy way for cygwin to "see" files in an NTFS mountpoint, or do they have to be copied to a directory in the Linux subsystem first?

hasse69 commented 7 years ago

If the NTFS volume is available as a Windows drive it should be accessible through /cygdrive.