Present volumes on remote system as Dokan filesystem on Windows

[masaeedu]

Many of the pain points of using volumes with docker-machine arise from the model of trying to present or synchronize the client computer's filesystem to the docker host machine. It would make for an easier workflow if this model was inverted, and docker-machine presented volumes on the targeted host as filesystems (or "drives") on Windows.

Presenting virtual filesystems on Windows now has good support via the Dokan project, and it should be possible to present any files on the host machine (including the volume roots) by wrapping them in a FUSE filesystem. This would make it convenient to populate a named volume with arbitrary data from the client machine, reducing the hassle of host volumes and mapping paths from your Windows client machine to the Docker VM.

[afbjorklund]

@masaeedu : could you do a Windows version of #4018, but using Dokan instead of FUSE ? There's some pointers in https://igikorn.com/sshfs-windows-10/ on what software is required:

• https://github.com/dokan-dev/dokany
• https://github.com/Foreveryone-cz/win-sshfs

[masaeedu]

@afbjorklund Is Win-SSHFS a UI application or a library that presents filesystem operations?

[afbjorklund]

@masaeedu : actually I have no idea, though it looks like a GUI application from the description...

For the UNIX version I just used the sshfs command line application (and fusermount), but if it is easier to use the API on Windows then by all means do that (not sure how you call .NET from Go ?).

[masaeedu]

@afbjorklund Probably easiest do whatever needs to happen outside Go as a child process and communicate over env vars/args/stdio. I'll take a look today, maybe I just need to add a command line interface on top of Win-SSHFS.

[afbjorklund]

@masaeedu : that would be great, might as well use the standard sshfs syntax then:

usage: sshfs [user@]host:[dir] mountpoint [options]

Where options are our standard -o flags, with things like Port and IdentityFile. Most of these come from ssh directly, so are similar/same to what ssh/scp/rsync use.

[afbjorklund]

Maybe you can use this directly ? https://github.com/dokan-dev/dokan-sshfs

Wonder if there is a way to test this using MinGW and Wine, i.e. without Win

[masaeedu]

@afbjorklund I'll look at that. Unfortunately I realized I can't exactly present the same CLI over win-sshfs because Dokan works with drives. It can't be mounted to arbitrary locations on an existing filesystem. I guess you could use symlinks or junctions to work around that, but people using software that doesn't support that would be in for problems.

[afbjorklund]

I don't think that's a real problem, as long as the user gives a drive letter (like "n") for the mount point and then looks in N:\ for the files ? Needs to be validated and documented, of course.

But one will need real Windows to do this, both because of Visual Studio .sln and because of .NET 4.0: [ERROR] FATAL UNHANDLED EXCEPTION: System.IO.FileNotFoundException: Could not load file or assembly 'System.Windows.Forms, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' or one of its dependencies.

Command itself (DokanSSHFS.exe) looks useful, though:

        public void help()
        {
            Console.Error.WriteLine("SSHFS");
            Console.Error.WriteLine("  -d drive letter [default n]");
            Console.Error.WriteLine("  -h host name");
            Console.Error.WriteLine("  -u user name");
            Console.Error.WriteLine("  -p port [default 22]");
            Console.Error.WriteLine("  -i ssh private key");
            Console.Error.WriteLine("  -r path to remote dir [default /]");
            Console.Error.WriteLine("  user@host drive");
            Console.Error.WriteLine();

https://ci.appveyor.com/api/projects/liryna/dokan-sshfs/artifacts/dokan-sshfs.zip (+ https://github.com/dokan-dev/dokany/releases/download/v1.0.2/dokan.zip)

[masaeedu]

@afbjorklund You could unit test your Go code if you wrote it properly, but ultimately you'll have to integration test on Windows if you want to know whether it works. The part that needs verification is whether it interfaces with the Windows APIs correctly so that you can add/remove/fetch files.

[afbjorklund]

Actually, the current test only verifies that it can find the exe (in the $PATH) and that it gets the arguments right... That sshfs (and/or DokanSSHFS.exe) actually works, is "someone elses problem". :-) Wonder how you unmount something on Windows ? Presumably there is a command-line similar to fusermount -u lurking somewhere, or perhaps just not support the friendly -u option.

[afbjorklund]

https://technet.microsoft.com/en-us/library/cc772586.aspx says: mountvol [<Drive>:]<Path> /p

[masaeedu]

@afbjorklund From my understanding you just kill the process that is using the Dokan API. You don't need to manually unmount. See for example the docs on the sample tool they bundle with dokan: https://github.com/dokan-dev/dokany/wiki/Use-Mirror-example

[afbjorklund]

Presumably just clicking something (or shutting off the computer) would work too...

Can you write the patch for the Windows CLI, or did you want me to take a stab at it ?

I suppose Windows users would always need to provide those charming DOS drives. (otherwise it would default to mounting the files on N:, at least if you only have one) The Unix version defaults to using the same path on the client as in the machine itself... But the extra mountpoint/driveletter was already there, in the (trivial) new mount syntax:

docker-machine ssh default mkdir -p /tmp/test
docker-machine mount default:/tmp/test n
docker run --rm -v /tmp/test:/tmp/test busybox ls /tmp/test
docker-machine mount -u /tmp/test n

[masaeedu]

@afbjorklund Ok, so the command line opts snippet you showed me is kind of out of date. The only code that used that is in Program.cs and is commented out. Uncommenting it, you find it is no longer compatible with the currently referenced Dokan API. I'm going to try and factor the code into a library and move the stuff the UI is doing into a console application.

[afbjorklund]

Typical, it looked too good to be true... Hope that we can find a sshfs for Windows that works ?

[masaeedu]

@afbjorklund It'd be great if we could find something pre-made, don't think it's likely though. Best bet is to hack out a console interface to the logic performed by the UI in the existing dokan-sshfs application. That's what I'm working on now.

[afbjorklund]

@masaeedu : Any luck with this ?

The available win-sshfs 1.5.12.8 (with dokany 0.7.4) seems to be working OK for casual use, but does have some issues with key exchange and newer mechanisms on newer OpenSSL.

• https://github.com/Foreveryone-cz/win-sshfs/releases/tag/1.5.12.8
• https://github.com/dokan-dev/dokany/releases/tag/v0.7.4

And of course it's a GUI application, so doesn't lean itself very well to any CLI interaction... So would be great with something updated, based on dokan-sshfs and the newer dokany ?

• https://github.com/dokan-dev/dokan-sshfs
• https://github.com/dokan-dev/dokany

[masaeedu]

@afbjorklund It took some wrangling, but I managed to factor out the logic into a library here: https://github.com/masaeedu/win-sshfs/tree/asad/Sshfs/SSHFS.Lib. Now we need a simple console application around that library. I've kind of forgotten the specifics of this issue, could you refresh my memory on what command line interface we need?

[masaeedu]

I've also rebased the changes in the embedded SSH.NET submodule onto https://github.com/sshnet/SSH.NET/tree/master, so hopefully that should alleviate any SFTP-related issues you're describing.

[afbjorklund]

The current interface looks something like this: https://github.com/afbjorklund/machine/blob/30de5bb722ec23a71fc9f7d9a3f1f9236129970e/commands/mount_test.go

Not sure if we want to keep the exact same syntax, and probably need to replace mount path with drive letter ? But something like this: https://linux.die.net/man/1/sshfs

[masaeedu]

@afbjorklund Ok sweet, it works! As a nice bonus, it just cleans up after itself once you kill the process, so the CLI code is very simple. I've pushed up what I have in https://github.com/masaeedu/win-sshfs/tree/asad/Sshfs/SSHFS.CLI, right now it's using keyboard-interactive auth.

I need to make some simple changes to the CLI and the logic so it tries public key, then password, then keyboard-interactive auth in order, but this shouldn't be too difficult.

[masaeedu]

@afbjorklund Ok, everything is pushed, and I've tested it out with docker-machine. There's good news and bad news. The good news is that I can mount and access the /tmp folder on my docker-machine as a drive using the following Powershell:

$inspect = docker-machine inspect linux; 
.\SSHFS.CLI.exe -d N -r /tmp -h "$($inspect | json Driver.IPAddress)" -u "$($inspect | json Driver.SSHUser)" -k "$($inspect | json Driver.SSHKeyPath)"

The bad news is that trying to do the same thing for the volume mountpoints doesn't seem to work:

$inspect = docker-machine inspect linux; 
docker volume ls --format "{{.Mountpoint}}" | % { 
    .\SSHFS.CLI.exe -d N -r $_ -h "$($inspect | json Driver.IPAddress)" -u "$($inspect | json Driver.SSHUser)" -k "$($inspect | json Driver.SSHKeyPath)"
}

I probably need to turn on debugging info to see what's going wrong, right now the mounted drive just craps out when you try to access it.

[afbjorklund]

@masaeedu : Excellent, sounds like you are making good progress! I haven't yet heard what the new maintainer of this project (i.e. docker/machine) thinks of it, but it would make a great compliment to #4018 !

Will try it next week... From Windows 7, if that matters.

[masaeedu]

@afbjorklund About a year later, this is working and reasonably usable on Windows; see https://github.com/masaeedu/win-sshfs/releases/download/2.0.41/SSHFS.CLI.zip. Could some of your work merging the docker-machine mount over SSHFS on Linux be reused for Windows?

[masaeedu]

It seems the issues I was originally having still persist, and they're caused by a permissions issue: the docker user doesn't have read permissions in mnt/sda1/var/lib/docker/volumes/. Not sure if there's some other folder that the user does have access rights to.

EDIT: After creating a password for the root user, I'm able to mount the volume folder to a drive.

[afbjorklund]

What about /home/docker?

[masaeedu]

@afbjorklund All folders I can cd to as the docker user I can mount under the docker user just fine. Or were you saying volumes are exposed somewhere in /home/docker?

[afbjorklund]

What we usually do, is that we take a folder that is already on the VM. And then we mount that folder also on the host. This way, any changes done to the folder are propagated to the VM, and then mounted into containers.

Maybe this helps: https://docs.docker.com/machine/reference/mount/

Note that most systems do it the other way around, by projecting e.g. your home into the VM! The benefit of keeping the files on the VM is that we only have to transfer files actually being edited, not everything being accessed.

[masaeedu]

@afbjorklund Yes, I understand the general process. I'm trying to do the same thing; mount a folder that exists on the docker machine VM into my Windows desktop. The folders I'm trying to mount are the volume folders, i.e. /mnt/sda1/.../volumes/<giant volume id>.

These folders are only accessible to root, not to the the docker user. As a result, when I use docker volume ls -formant {{MountPoint}} to list these mountpoints and use my win-sshfs thing to mount them, I can't use the docker user, which is the only one I have a private key for. I must instead ssh in as the root user, which isn't ssh-able by default. Hopefully this clarifies the problem?

[afbjorklund]

But I don't see why you are trying to access the docker internals ? We would use a normal folder on the VM (one that is accessible to "docker"), and then let the docker daemon handle the mapping (bind mount) of those into containers.

docker run -v $HOME/foo:/foo ...

So I would only try to mount the /home/docker/foo directory, but I would let Docker handle any /var/lib/docker/volumes/foo created as volumes. Could use docker cp to access them, perhaps ? And probably use somewhere persisted to disk rather than tmpfs, for things you want to keep around.

[masaeedu]

@afbjorklund I'm using named volumes instead of bind mounts. In other words, I'm doing the equivalent of docker run -v jenkinsvol:/... rather than mapping to a hardcoded path in the host. This is a common need when you're working on a shared swarm instance, using stack definitions, etc.