Migrate from d4m-nfs to native Docker NFS Volumes discussion

[mauro-ni]

It doesn't work with Docker for Mac Version 17.12.0-ce-mac46

$ ~/d4m-nfs/d4m-nfs.sh
[d4m-nfs] You will need to provide your Mac password in order to setup NFS.
Password:
[d4m-nfs] Copy the Moby VM APK Cache back.
[d4m-nfs] Create the script for Moby VM.
[d4m-nfs] Start and restop nfsd, for some reason restart is not as kind.
The nfsd service is already running.
[d4m-nfs] Wait until NFS is setup.
[d4m-nfs] Wait until D4M is running.
[d4m-nfs] Setup 'screen' to work properly with the D4M tty, while at it name it 'd4m'.
[d4m-nfs] Run Moby VM d4m-nfs setup script.
[d4m-nfs] Waiting until d4m-nfs setup is done
.........................................................................................
.........................................................................................
.........................................................................................
.........................................................................................
.........................................................................................
.........................................................................................
.........................................................................................
.........................................................................................
.........................................................................................
.........................................................................................
.........................................................................................
.........................................................................................
.........................................................................................
.........................................................................................
.........................................................................................
......................

[pensiero]

+1

[Plopix]

EDIT to explain better what I understood after more debug.

The new version moved to Linuxkit that is a read-only system. The screen is still working but it is not possible to install nfs-util and then we cannot mount the host. I have no clue how to do that on Linuxkit or if it is even possible, that kind of sucks, I am currently looking to docker-sync as I start to think that will be the only way to have acceptable performances.

But you might have a magic solution! I am sure you have one, right? :/

cc @if-kenn

[if-kenn]

The brief time I have had to look into this comes to the same conclusion as @Plopix. With the latest release of Docker for Mac, Docker has again changed the way in which MobyVM sees the Mac's file system.

This would force us to reverse to a previous, more fragile and much harder to maintain way that we injected the script to setup NFS.

With this latest change, I need to revisit if the osxfs volume performance tuning has finally caught up enough to stop the need for d4m-nfs. This whole time I have been honestly hoping to deprecate this software once there was a better solution from Docker. I will report back my findings and the future of d4m-nfs. For now I am going to put a big warning in the README.

[if-kenn]

README has been updated, performance testing and outcome will have to wait until tomorrow.

[Plopix]

thanks @if-kenn, to me the osxfs volume performance tuning does not help as our main issue here is the write performances on a mount and not the read. The tunning is pretty much just a cache system but I might be wrong I did not test extensively as the NFS method was still better ;)

[Plopix]

Also I dig a bit more on docker-sync even if I don't like the duplication and the fact we need a kindof daemon, it seems much more simpler to config than before. This would do the trick (of course using the external volume in the docker-compose-dev).

But the NFS solution was nice, simple, and much more transparent! we have to find a way!

# docker-sync.yml
version: "2"
options:
  compose-file-path: 'docker-compose.yml'
  compose-dev-file-path: 'docker-compose-dev.yml'
syncs:
  project-sync:
    sync_strategy: 'native_osx'
    src: './'
    sync_userid: '501' # to be adapted
    # optional, a list of excludes
    sync_excludes: ['.git', '.gitignore', '.DS_Store', '.idea', 'docker-sync.yml', 'docker-compose.yml', 'docker-compose-dev.yml', 'etc.']

[yeka]

I'll just put it here in case someone already upgrade their docker and want the previous version: https://download.docker.com/mac/stable/21090/Docker.dmg

[avinson]

@if-kenn my recent testing of osxfs volume tuning (about a month ago) found that rails rake tasks ran about 10-15 times faster under d4m-nfs

[if-kenn]

So some bad news. While you can still screen into the Linux VM, it is a read only root file system. The way that d4m-nfs works is to be able to add the nfs-utils apk package. This means that when running apk add nfs-utils sntpc you get Read-only file system. I have unsuccessfully tried to remount the root files system as read/write. At this point, it looks like Docker for Mac has effectively killed d4m-nfs.

[if-kenn]

I am looking at the possibility to roll a new OS image for Docker for Mac from Linuxkit to understand if it is possible to mount / read/write or roll an image with the nfs package already on it. The problem with this it is not light weight (read: could take a while to make the image and something we cannot support) or there would have to be a trust factor of any OS image that would come from doing it and centrally disseminating it.

[if-kenn]

So an update, I have been playing with Linuxkit and have been able to have a writable root file system, install nfs-utils and run rpcbind. What I have not been able to do is mount the Mac's NFSd, it appears that it is being blocked by how VPNKit is setup. The documentation on all of this is not the easiest to understand what needs to be done. If anyone has knowledge of VPNKit and how it relates to Linuxkit, that would be quite helpful.

[if-kenn]

So, a very small break through. On a customized Linuxkit build, I have been able to successfully mount NFS from Mac host using the IP (192.168.65.2) I found on https://github.com/moby/vpnkit/blob/master/docs/ethernet.md.

That said, I have yet to find a way to programmatically get that IP, and after that there would still be the need to get rpcbind service used in the correct Linuxkit way, and then feed the d4m-nfs info over to it.

[if-kenn]

It looks like according to https://github.com/linuxkit/linuxkit/blob/master/docs/platform-hyperkit.md we can rely on the virtio-net IP to be 192.168.65.2 by default unless it was to be assigned a fixed IP via linuxkit run -ip.

[pensiero]

We could use it as default value, and in case specify it it as a configuration value, what do you think?

[if-kenn]

@pensiero Yes, that is my intention, but let me be very clear, this is not an easy fix, it will not just be a drop in replacement and the solution will not come quickly if at all. In that light please keep the following in mind:

• there is very sparse Linuxkit/VPNkit/etc documentation to help
• it will not be in the immediate future
• it will very well have to a be new github repository

So if you are looking for a fix soon, please downgrade to an older version of Docker for Mac.

I am intending to keep updating on progress here in order to provide transparency or if others have insights that might help.

[justincormack]

See the details here https://github.com/docker/for-mac/issues/2419#issuecomment-357921637

We moved to a read only rootfs as we now boot off an ISO, as Kubernetes is too large to boot out of a writeable RAM filesystem. It would have been easier if you had always used a privileged container for NFS, rather than assuming that the host would have a working Alpine system, as this was never a guarantee. You should be able to run NFS fine with either a privileged Docker container or a privileged containerd container. The host static IP shouldn't change; there is also a DNS name for it documented. You should not need to respin a whole LinuxKit image, and if you use a privileged Docker container set to always restart it should work reasonably well.

[if-kenn]

@justincormack from what I am understanding from David Scott's comment and what I can make of yours above, I have tried to prototype what is suggested, to no luck. Unless there is some magic that I am missing, it seems like building a new Linuxkit image is still needed.

[Yivan]

@if-kenn It make several hours i try to mount NFS too (on Docker for Windows, but i got the same problems of you).

What i don't anderstand is there is native nfs volume in docker, why not use them ? (end of this page: https://docs.docker.com/engine/reference/commandline/volume_create/#driver-specific-options) I tried on windows with no luck... people having the same timeout problem never get answer from the docker team : (( (please see: https://github.com/docker/for-win/issues/1241). Maybe on mac it can work, you made a try of nfs volume ? I opened an issue here to have a clear answer on this subject: https://github.com/docker/for-win/issues/1700

From what i found, it is because docker for win (and from what i see maybe docker for mac too) don't have the NFS utils to mount nfs. Did you manage to find a way without relying on a custom VM/ISO linuxkit image ?

[Yivan]

Hum really strange! It seems with docker-compose we can mount nfs folder... (i don't have the connection timeout problem that i have with docker volume create & docker run)

volumes:
    nfsshare:
        driver_opts:
            type: "nfs"
            o: "addr=10.0.75.1,hard,nolock,rw"
            device: ":/server-dev"

It works fine! I don't undestand how docker manage to do it has there is no nfs tools on the VM.... Maybe it can give you a new route to make d4m working.

[Yivan]

It seems the no-lock option did the trick: The following work great:

docker volume create --driver local --opt type=nfs --opt o=addr=10.0.75.1,hard,nolock,rw --opt device=:/server-dev mynfs
docker run -it -v mynfs:/server-dev alpine sh

[yvess]

@Yivan so with your one volume creating command, you can work like before? Is the volume persistent, or do you need to recreate it after docker restart.

[Yivan]

@yvess yes you can work as usaually and the volume is like other volume, so it persiste as long you don't delete the VM. Even if you delete the volume, the data are kept because the volume just mount your host datas (it doesn't conains the data, it just countain the nfs mount point).

[mauro-ni]

@Yivan currently I'm using shared folder for application code in my docker-compose.yml.

version: '2'
services:
  web:
  ...
  volumes:
    - ../www:/home/app/www
  ...

How can I convert this so that NFS is used?

Many thanks.

[mauro-ni]

@Yivan with docker-compose I get a timeout error even if I provide the nolock option

[Yivan]

If you mount it directly from a container does it work ? (you can create an Ubuntu container, install nfs client, and mount the NFS share). It should work.

• if not, there is pbl with your NFS Server or IP is not good.
• if it work, it should work in docker-compose too, so problem is maybe firewall, try desactivate it

[mauro-ni]

@Yivan could you please create a demo and share it (new git repo)? I'd like to use NFS for the folder containing my rails apps.

[Yivan]

There is no need i set a new repo for just two lines of code... For instance, if the NFS server has IP 10.0.75.1 and the export name is /server-dev, the code is:

docker volume create --driver local --opt type=nfs --opt o=addr=10.0.75.1,hard,nolock,rw --opt device=:/server-dev mynfs
docker run -it -v mynfs:/server-dev alpine sh

Did you try what i suggest you (mount it manually, disable firewall, etc.) ? What are the results ? I should add if you try mounting it manually you must add some CAPS to you container, at least SYS_ADMIN and DAC_READ_SEARCH (see here for more information to set them: https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities)

[ryansch]

Give this fix a try: https://github.com/docker/for-mac/issues/2419#issuecomment-361781272

[mauro-ni]

Docker for mac EDGE seems to support NFS: https://docs.docker.com/docker-for-mac/release-notes/#docker-community-edition-18030-ce-rc1-mac54-2018-02-27-edge

[pensiero]

@maurosbu 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉

[mauro-ni]

@pensiero I would like to try to mount app code in a named volum with NFS. Currently I'm using bind mount (docker compose).

[pensiero]

me too, but that's something

[mauro-ni]

@pensiero I'm working on a dozen of rails applications and my development environment is based on https://github.com/phusion/passenger-docker . Without NFS the development is painful due to the speed problems of Docker on Mac. Hope to see a tutorial soon.

[patakijv]

Is there any update here? I am seeing the same issue with d4m Version 17.12.0-ce-mac55 (23011). Did the NFS changes mentioned above in this thread found for the edge release not make it yet into this d4m version or is there some sort of change needed in d4m-nfs still for this d4m release? Or some configuration needed for d4m that wasn't previously needed?

[if-kyle]

@patakijv You can check https://docs.docker.com/docker-for-mac/release-notes/ -- i do not see any mention of NFS being moved to stable.

At this time, we still have our teams holding on the last good release mentioned in the README. It is unlikely more time will be spent trying to resolve this issue until docker-for-mac releases a stable version with NFS support to see the performance.

[mauro-ni]

A stable version supporting NFS volumes has been released: https://docs.docker.com/docker-for-mac/release-notes/#stable-releases-of-2018

[MetalArend]

@maurosbu Do you have some example gist as to how to set this up? Docker documentation seems to be lacking on this particular feature.

[mauro-ni]

@MetalArend I don't know how to set this up. As you said, there isn't any documentation about this feature.

[seanhandley]

Yeah, I feel the missing pieces here are:

• what do I need to configure on my Mac?
• what should a docker-compose file look like?

[myers]

I was able to get nfs working in a container sharing a directory from my mac.

$ id
uid=501(myers) gid=20(staff) groups=20(staff),12(everyone),61(localaccounts),79(_appserverusr),80(admin),81(_appserveradm),98(_lpadmin),501(access_bpf),701(com.apple.sharepoint.group.1),401(com.apple.access_screensharing-disabled),33(_appstore),100(_lpoperator),204(_developer),250(_analyticsusers),395(com.apple.access_ftp),399(com.apple.access_ssh)

So, my uid = 501 and guid = 20, that's what I want all the files to look like in the nfs mount

/etc/exports

"/Users" -alldirs -mapall=501:20 localhost

/etc/nfs.conf

#
# nfs.conf: the NFS configuration file
#
nfs.server.mount.require_resv_port = 0

$ sudo nfsd restart

docker-compose.yml

version: "2"

services:
  guinea-pig:
    image: ubuntu
    volumes:
      - "nfsmount:/mnt/homedir"

volumes:
  nfsmount:
    driver: local
    driver_opts:
      type: nfs
      o: addr=host.docker.internal,lock
      device: ":/Users/myers/"  # location on nfs server of file/dir you want to mount

You will want to change your user in the docker container to have the same uid/gid as your user on the mac side of things.

[guillaumerose]

Hi, I'm from the Docker for Desktop team.

We added rpcbind and rpc.statd at VM startup in order to fix NFS support for persistent volume in Kubernetes. See https://github.com/docker/for-mac/issues/2515#issuecomment-365257862

Let me know if you still have problem. You can reach directly me/us directly on Docker community Slack. Sorry for the disturbance.

[sponomarev]

Hey @guillaumerose, can you elaborate a little bit, doest it mean that d4m-nfs is fixed now you you provide a native way to use NFS for sharing volumes?

[guillaumerose]

There is nothing in Docker for Mac related to NFS, sorry. We are working on better file sharing (with and without osxfs), but it takes lot of time.

[mauro-ni]

@guillaumerose what does Support NFS Volume sharing. mean? https://docs.docker.com/docker-for-mac/release-notes/#stable-releases-of-2018

[sponomarev]

Yeah, I was confused with it too.

[myers]

@maurosbu I think they added some support for nfs in the moby vm, like what this project attempts to do, but it was perhaps not so much to address the mac filesystem part, but rather as part of better support for kubernetes.

[Plopix]

Did someone test if now the Linux Kit let us do mount now?

We added rpcbind and rpc.statd at VM startup in order to fix NFS support for persistent volume in Kubernetes.

That would mean we can mount like before and use d4m-nfs like before? Or does that just mean now we can safely use an external volume over nfs? or both?

[Plopix]

I can answer my own question I have just tested.

=> no we cannot use d4m-nfs as before even with this change

But we can definitively do as mentioned by @myers, just did and it is working great. It sounds better in term of logic, in term of performance, I checked with a time dd if=/dev/zero of=speedtest bs=1024 count=100000 and I was around 69MB/s (same with d4m-nfs)

One thing though, you need to export explicitly what you want to mount. You cannot export: /Users/plopix and then mount /Users/plopix/Documents/DEV/ProjectOne Hope it helps some of you. ++

[myers]

@Plopix

One thing though, you need to export explicitly what you want to mount. You cannot export: /Users/plopix and then mount /Users/plopix/Documents/DEV/ProjectOne

have you tried the -alldirs option on the mac side's /etc/exports

from mac os's man 5 exports:

The -alldirs flag allows the host(s) to mount at any point within the file system, including regular files if the -R option is used on nfsd.

[Plopix]

hey hey @myers of course! That is working! Awesome! Thank you!