unmounting FUSE through FileSystemHost cleanly

advdv commented 7 years ago

Great work on getting such a clean interface for cross platform FUSE! When I was trying this out today I couldn't get the Filesystem to unmount cleanly though. For example, adapting the memfs example's main() like this causes the filesystem to hang around when the program exits:

func main() {
    exitCh := make(chan os.Signal)
    signal.Notify(exitCh, os.Interrupt)

    memfs := NewMemfs()
    host := fuse.NewFileSystemHost(memfs)
    fmt.Println("mounting...")
    go host.Mount(os.Args)

    <-exitCh
    fmt.Println("unmounting...")
    host.Unmount()
}

on OSX/Linux the Unmount() doesn't seem to do anything, on Windows the library itself seems to unmount but not under control of my own program but rather somewhere inside cgofuse itself.

billziss-gh commented 7 years ago

The problem

Unmount is misnamed. I contemplated for the longest time whether to add it to the API at all and reluctantly added it. I think this may have been a mistake.

The problem is that FUSE does not have a clean way to unmount a file system. Instead it has a function for signaling the FUSE loop that it can exit. This function is called fuse_exit and can only be safely used from inside a file operation handler (e.g. fuse_operations::open or FileSystemInterface.Open). In fact under OSXFUSE fuse_exit may not work at all (at least under Go).

Unmount simply calls fuse_exit and it has the same limitations and problems as fuse_exit:

On Linux it works, but only if called from within a file operation.
On OSX it is supposed to work from within a file operation. However it does not work reliably.
On Windows it works no matter where it is called (inside a file operation, from a different thread, etc.)

The other problem is that the FUSE layer handles its own signals. It is rather perilous to attempt to change signals in a FUSE program without understanding all the details of the FUSE loop. Here is a very interesting thread that discusses problems with signal handling and fuse_exit:

http://fuse.996288.n3.nabble.com/libfuse-exiting-fuse-session-loop-td10686.html

on OSX/Linux the Unmount() doesn't seem to do anything, on Windows the library itself seems to unmount but not under control of my own program but rather somewhere inside cgofuse itself.

What is likely happening here (keep in mind I am still a Go novice):

OSX/Linux: the signal.Notify call sets a signal handler for SIGINT. On OSX/Linux the FUSE layer checks if SIGINT already has a handler and if that is the case it does not set that signal handler. When you press ^C you get notified on your channel and call Unmount which does not work (because it is not called from within a file operation).
Windows: because windows does not have signals I am not sure what signal.Notify does (perhaps calls SetConsoleCtrlHandler?). The WinFsp-FUSE layer always handles ^C "events" by stopping the FUSE loop.

How to fix this

There are unfortunately no easy fixes. Here are a few ideas:

Remove Unmount. It does not do what its name suggests and is confusing.
Rename Unmount to Stop as per @ncw's suggestion. Slightly better but still confusing. It does not convey the message that it can only be called from within a file operation.
Properly fix Unmount for all platforms. I discuss this option next.

Unmount on Linux

We cannot issue the umount(2) system call, because it requires super-user privileges. So we must launch fusermount. [Yuck!]

Unmount on OSX

OSX allows issuing an unmount(2) from non-root. So this may work. If we do this we should also pass MNT_FORCE to ensure that the file system gets unmounted even if it is in use.

Unmount on Windows

On WinFsp-FUSE fuse_exit actually works regardless of where it is called.

advdv commented 7 years ago

Thank you for the expansive answer! I see the dilemma. Let me take some time to think about a possible solution for my case and I'll share the results. This way we may find a suitable middleway.

billziss-gh commented 7 years ago

@advanderveer thanks. I have actually been working for a solution in the last couple of hours. But I will be happy to see what you come up with.

billziss-gh commented 7 years ago

BTW, my experiment is in the hostMain branch.

billziss-gh commented 7 years ago

Commit 927776886103a09b2f1d37e3727b328de4e9342c fixes this. Please test under your scenario and let me know.

advdv commented 7 years ago

This does what I expect it to do on Linux and OSX (Cool!). On windows, having the winfsp capture of the signal is unexpected and I'm not sure how to work around it, but i'll do some research myself for that. If I find anything i'll post it here for future reference.

The main issue is fixed so i'll close this

billziss-gh commented 7 years ago

@advanderveer I am glad that Unmount() works for you.

[NOTE: Unmount is (currently) safe to use between Init() and Destroy() on the user mode file system. I should clarify this further in the docs.]

On windows, having the winfsp capture of the signal is unexpected and I'm not sure how to work around it, but i'll do some research myself for that.

When I was writing the WinFsp-FUSE layer, I did try to implement the libfuse signal semantics. Unfortunately this is not possible on Windows, because of the lack of true signal support.

Are you trying to cleanup just before your file system exits (through a signal or otherwise)? It might be worth trying to fix this in cgofuse (or even in WinFsp), so that cgofuse clients do not have to worry much about the specifics of the underlying implementation.

So please do share your research findings if/when you have them.

UPDATE: For example, we could add a guarantee that Destroy() gets called even on a SIGINT. This does not happen today with libfuse, and would mean that cgofuse would have to set its own signals on UNIX.

advdv commented 7 years ago

Unfortunately i'm not aware of what a "normal" fuse impelementation is supposed to do with signals but i'm simply trying to make sure that a user who start my CLI application (that mounts a file system while running) exits with the users system in the shape it was before running my application.

I found that the following does what I want:

func main() {
    memfs := NewMemfs()
    host := fuse.NewFileSystemHost(memfs)

    //if not windows we need to manage unmount on our own, concurrently get notified for SIGINT/SIGTERM and unmount, this will cause the main mount to return and exit the program
    if runtime.GOOS != "windows" {
        sigCh := make(chan os.Signal, 1)
        signal.Notify(sigCh, os.Interrupt, syscall.SIGTERM)
        go func() {
            <-sigCh
            if !host.Unmount() {
                os.Exit(2) //unmount failed
            }
        }()
    }

    if !host.Mount(os.Args) {
        os.Exit(1) //mount failed
    }
}

As a user of this library I expected to be able to use signal.Notify on windows as I would in other Go programs. It might have something to do with how golang implements signal handling on windows: here

Unfortunately i'm not knowledgable about either FUSE or windows to know if the way I expect it to act is sensible and the solution I describes above is statisfactory.

You wouldn't need to go through hoops for a better solution but maybe it is possible to configure the signal capturing on windows (opt in) such I can decide to call unmount myself on windows as well?

billziss-gh commented 7 years ago

As a user of this library I expected to be able to use signal.Notify on windows as I would in other Go programs. It might have something to do with how golang implements signal handling on windows

I had a read at golang's "signal" support for windows:

They install a "ConsoleCtrlHandler" in runtime/os_windows.go
This eventually arrives at ctrlhandler1 (via sys_windows_amd64.s).

Ctrlhandler1 is a fairly simple function which calls sigsend on Ctrl-C.

Unfortunately this assumes that the golang runtime is the only entity handling ^C in a golang program. This is not true in our case. [Windows allows multiple handlers for "console ctrl events".]

You wouldn't need to go through hoops for a better solution but maybe it is possible to configure the signal capturing on windows (opt in) such I can decide to call unmount myself on windows as well?

This would have to be changed on the WinFsp-FUSE layer. Unfortunately a change like this would be problematic for a number of reasons.

Since you have a solution for this I propose that we:

Either do nothing. If someone else has your need we point them to your solution.
Or that we resolve this in a cross-platform way by making a couple of guarantees for cgofuse.

These guarantees would be:

That cgofuse will completely unmount the file system (unless it gets forcibly terminated by kill -9). No zombie mounts.
That the Destroy() method always gets called by cgofuse (again unless the file system get forcibly terminated). This guarantee would ensure that a file system would always have a chance to clean up after itself (beyond simple Unmount).

Of course this proposal does not do what you want (allow signal.Notify to work on all platforms when using cgofuse). But it at least eliminates many of the reasons to use signal.Notify.

ncw commented 7 years ago

I think the if runtime.GOOS != "windows" wrapper is fine for the signal handling. TBH I don't understand why fuse doesn't unmount the fs when the process providing it goes away, but I expect there is a technical reason for it!

I have some almost identical code in rclone for unmounting on a signal which I'll wrap in if runtime.GOOS != "windows" .

billziss-gh commented 7 years ago

TBH I don't understand why fuse doesn't unmount the fs when the process providing it goes away, but I expect there is a technical reason for it!

I believe the reason is both historical and technical, but I am not the right person to answer this question. Here is an interesting thread on this subject (I do not necessarily agree with their reasoning):

https://sourceforge.net/p/fuse/mailman/message/30221453/

I have some almost identical code in rclone for unmounting on a signal which I'll wrap in if runtime.GOOS != "windows" .

So I gather that there is no perceived need to have this fixed in cgofuse then.

ncw commented 7 years ago

I believe the reason is both historical and technical, but I am not the right person to answer this question. Here is an interesting thread on this subject (I do not necessarily agree with their reasoning):

Hmm, interesting thread...

So I gather that there is no perceived need to have this fixed in cgofuse then.

I think documenting the difference would be fine.

billziss-gh commented 7 years ago

I wrote:

... that we resolve this in a cross-platform way by making a couple of guarantees for cgofuse.

These guarantees would be:

That cgofuse will completely unmount the file system (unless it gets forcibly terminated by kill -9). No zombie mounts.

That the Destroy() method always gets called by cgofuse (again unless the file system get forcibly terminated). This guarantee would ensure that a file system would always have a chance to clean up after itself (beyond simple Unmount).

Heads up! The fact that cgofuse wants to be cross-platform, but did not deal with the differences between unmounting behavior on different platforms, kept bothering me. So I decided to fix it tonight.

Commit 047c3f83db04bfbd2299bdbcdc34d781f8de4466 adds the aforementioned guarantees. This is currently on the auto-unmount branch, but I will be merging into master soon.

BTW, this commit does not currently handle SIGPIPE although it probably should. Golang has somewhat interesting behavior on SIGPIPE [link].

ncw commented 7 years ago

I think that making extra cross platform guarantees is a great idea :-) Love the idea of no more zombie mounts. (I can't suspend my laptop at the moment because I have about 20 zombie mounts ;-)

Why are you worried about SIGPIPE? Does kernel use SIGPIPE to talk to libfuse or something? I think go's handling of sigpipe will mean that it doesn't exit normally.

billziss-gh commented 7 years ago

I think that making extra cross platform guarantees is a great idea :-) Love the idea of no more zombie mounts. (I can't suspend my laptop at the moment because I have about 20 zombie mounts ;-)

Great. This is now merged into master.

Why are you worried about SIGPIPE? Does kernel use SIGPIPE to talk to libfuse or something? I think go's handling of sigpipe will mean that it doesn't exit normally.

It is another signal that may kill a FUSE program thus leaving zombie mounts. Libfuse normally handles it by ignoring it. Golang's handling of SIGPIPE seems rather nuanced, so I think it is best to not catch it in cgofuse.

So cgofuse currently gets notified (and cleans up) on SIGINT, SIGTERM and SIGHUP only.

winfsp / cgofuse