search

mineshaftgap / d4m-nfs

Docker for Mac with NFS for performance improvements over osxfs
295 stars 26 forks source link

Include benchmarks in readme #7

Closed motin closed 5 years ago

motin commented 7 years ago

Check this out: https://github.com/docker/for-mac/issues/77#issuecomment-256559826

Thank you @goneflyin so much for creating d4m-nfs!

This is incredible, and should be bragged about in the readme :)

Benchmark - dd if=/dev/zero of=speedtest bs=1024 count=100000

* On host
100000+0 records in
100000+0 records out
102400000 bytes transferred in 0.307839 secs (332641246 bytes/sec)

real    0m0.327s
user    0m0.020s
sys 0m0.296s

* Within Docker for Mac without a host volume
100000+0 records in
100000+0 records out
real    0m 2.06s
user    0m 0.01s
sys 0m 2.04s

* Within Docker for Mac within a host volume
100000+0 records in
100000+0 records out
real    0m 31.69s
user    0m 0.11s
sys 0m 3.66s

* Within Docker for Mac within a host volume using NFS
100000+0 records in
100000+0 records out
real    0m 1.73s
user    0m 0.00s
sys 0m 0.60s

NFS is 18x faster than OSXFS - and even faster (or at least on par with) than not using a host volume at all (which blows the file-synchronization workarounds away from consideration)!

This gist currently includes the benchmark from @Journerist, feel free to add more if necessary, albeit the above already tells the story imo.

To reproduce this benchmark on your own machine, step into a temporary folder and run:

git clone https://github.com/IFSight/d4m-nfs.git
git clone https://gist.github.com/488998820e31f65c4ae01b4bf4d42146.git bench-77
d4m-nfs/d4m-nfs.sh
bench-77/bench-d4m-nfs.sh
luetge commented 7 years ago

If I run these commnds in an empty directory ~/tmp I get

docker: Error response from daemon: Mounts denied:
The path /mnt/tmp
is not shared from OS X and is not known to Docker.
You can configure shared paths from Docker -> Preferences... -> File Sharing.
See https://docs.docker.com/docker-for-mac/osxfs/#namespaces for more info.

Any advice on how to fix that? Thanks, this looks promising!

motin commented 7 years ago

@luetge /mnt is mounted by running d4m-nfs/d4m-nfs.sh, ie the third command. Paste your complete output from the four commands, and we'll check what went wrong.

davidjeddy commented 7 years ago

Ran this on a 2015 MacBook Pro, results are amazing.

mark - dd if=/dev/zero of=speedtest bs=1024 count=100000

luetge commented 7 years ago

This is the output I am getting (I removed all Docker images and cleared the ~/tmp dir).

~/tmp $ git clone https://github.com/IFSight/d4m-nfs.git
git clone https://gist.github.com/488998820e31f65c4ae01b4bf4d42146.git bench-77
d4m-nfs/d4m-nfs.sh
bench-77/bench-d4m-nfs.sh
Cloning into 'd4m-nfs'...
remote: Counting objects: 61, done.
remote: Compressing objects: 100% (55/55), done.
remote: Total 61 (delta 20), reused 0 (delta 0), pack-reused 6
Unpacking objects: 100% (61/61), done.
Cloning into 'bench-77'...
remote: Counting objects: 5, done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 5 (delta 0), reused 2 (delta 0), pack-reused 0
Unpacking objects: 100% (5/5), done.
You will need to provide your Mac password in order to setup NFS.
Password:
Start and restop nfsd, for some reason restart is not as kind.
Stopping the nfsd service (use 'disable' to make permanent)
Starting the nfsd service
Wait until NFS is setup..
Wait until D4M is running.
Setup 'screen' to work properly with the D4M tty, while at it name it 'd4m'.

Log into D4M as root so next commands can run.

Make symlink to apk cache dir on Mac.

Install nfs-utils, make the /mnt dir, start rpcbind, wait, then mount Mac NFS.

Pausing for NFS mount to be ready so this can be used in another script.

Please note:
------------
• Only /Users/daniel directory is mounted, this might change if there is a request to be all user directories, or other locations.
• The /Users mount under D4M still exists and will continute to be slow, the d4m-nfs mount is under /mnt.
• When mounting Docker volumes, you need to change paths like /Users/daniel mounts with /mnt.
• To connect to the D4M moby linux VM use: screen -r d4m
• To disconnect from the D4M moby linux VM tty screen session use Ctrl-a d.
• To run d4m-nfs faster and/or offline, leave ~/d4m-apk-cache and hello-world image.
• If you switch between D4M stable and beta, you might need to remove ~/d4m-apk-cache and hello-world image.

Benchmark - dd if=/dev/zero of=speedtest bs=1024 count=100000
* On host
100000+0 records in
100000+0 records out
102400000 bytes transferred in 0.364833 secs (280676275 bytes/sec)

real    0m0.384s
user    0m0.024s
sys 0m0.309s

* Within Docker for Mac without a host volume
Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
3690ec4760f9: Pull complete
Digest: sha256:1354db23ff5478120c980eca1611a51c9f2b88b61f24283ee8200bf9a54f2e5c
Status: Downloaded newer image for alpine:latest
100000+0 records in
100000+0 records out
real    0m 0.37s
user    0m 0.02s
sys 0m 0.35s

* Within Docker for Mac within a host volume
100000+0 records in
100000+0 records out
real    0m 35.61s
user    0m 0.09s
sys 0m 1.59s

* Within Docker for Mac within a host volume using NFS
docker: Error response from daemon: Mounts denied:
The path /mnt/tmp
is not shared from OS X and is not known to Docker.
You can configure shared paths from Docker -> Preferences... -> File Sharing.
See https://docs.docker.com/docker-for-mac/osxfs/#namespaces for more info.
..

The same happens if I then execute only the last two commands, with slightly different output.

d4m-nfs/d4m-nfs.sh
bench-77/bench-d4m-nfs.sh
You will need to provide your Mac password in order to setup NFS.
Password:
Start and restop nfsd, for some reason restart is not as kind.
Stopping the nfsd service (use 'disable' to make permanent)
Starting the nfsd service
Wait until NFS is setup..
Wait until D4M is running.

Please note:
------------
• Only /Users/daniel directory is mounted, this might change if there is a request to be all user directories, or other locations.
• The /Users mount under D4M still exists and will continute to be slow, the d4m-nfs mount is under /mnt.
• When mounting Docker volumes, you need to change paths like /Users/daniel mounts with /mnt.
• To connect to the D4M moby linux VM use: screen -r d4m
• To disconnect from the D4M moby linux VM tty screen session use Ctrl-a d.
• To run d4m-nfs faster and/or offline, leave ~/d4m-apk-cache and hello-world image.
• If you switch between D4M stable and beta, you might need to remove ~/d4m-apk-cache and hello-world image.

Benchmark - dd if=/dev/zero of=speedtest bs=1024 count=100000
* On host
100000+0 records in
100000+0 records out
102400000 bytes transferred in 0.355742 secs (287849077 bytes/sec)

real    0m0.378s
user    0m0.025s
sys 0m0.330s

* Within Docker for Mac without a host volume
100000+0 records in
100000+0 records out
real    0m 0.45s
user    0m 0.04s
sys 0m 0.41s

* Within Docker for Mac within a host volume
100000+0 records in
100000+0 records out
real    0m 37.26s
user    0m 0.18s
sys 0m 2.16s

* Within Docker for Mac within a host volume using NFS
docker: Error response from daemon: Mounts denied:
The path /mnt/tmp
is not shared from OS X and is not known to Docker.
You can configure shared paths from Docker -> Preferences... -> File Sharing.
See https://docs.docker.com/docker-for-mac/osxfs/#namespaces for more info.
..

Sorry for the long post and thanks for helping!

if-kenn commented 7 years ago

@luetge df4-nfs relies on having a the /tmp mounted in order to pass files between the Mac and Moby VM. The message that you have above suggests that it might not be in the Docker for Mac preferences. You can see an example of the minimal setup here:

https://raw.githubusercontent.com/IFSight/d4m-nfs/master/examples/img/d4m-min-file-sharing.png

If you continue to have problems, please open up a new issue as this one is talking about including benchmarks in the README.

gagarine commented 7 years ago

I did a test with Drupal 8, and it's work pretty well. Definitely pages load lot faster than with OSXFS and it's pretty easy to setup. It use very little CPU. But I think it's slower than volume and http://docker-sync.io/ .

We should perhaps try more "realistic" benchmark that actually open and read thousands of files inside a complicated directory structure. Also program can request sometimes a lot of time the same files, the file system should be able to cache those request (if the file didn't change).

if-kenn commented 7 years ago

@gagarine we are actively using if for our local dev stacks (mariadb/redis/phpfpm/nginx/varnish/haproxy), probably 10-20 different Drupal 7 sites and maybe 4+ Drupal 8 sites.

We looked at syncing solutions in the past and I can't right now remember what the issue was, but we decided that 1/2 second different in page load was not a deal breaker. I have no doubt there could be faster solutions out there, this is just our solution.

gagarine commented 7 years ago

@if-kenn nice to hear you use it on Drupal projects. Before using docker I had response time under >0.20ms time, 1/2 is a lot and I feel it.

Say that, I was not criticising your tools, I just like when limitations are documented. I'm not even sure it is slower, I need to test it. My post was more a reply to "which blows the file-synchronization workarounds away from consideration" from the first post.

Another limitation I taught is permission, I need to do some test for that.

I was exploring those plus and minus because I'm writing the documentation for https://github.com/laradock/laradock and want to give users a clear overview of the advantages of each solution so they can choose the best tools for their needs.

if-kenn commented 7 years ago

No worries, not taken as criticism! I hear you about the slower feel, we probably don't feel it too much since we have opcache/redis/varnish doing a lot of optimization for us, also for me the trade off is worth it for me not to hear from any of my engineers "well it is not happening on my system" :). I am hoping Docker just fixes OSXFS and this project is no longer needed!

The permission limitation is generally worked around by having it export as root instead of the working user.

I have been working on the ability for usage to be more automatic via watching the Moby VM tty socket, inspired from the IFTTT team who uses this.

longseespace commented 7 years ago

UPDATE: I did a Factory Reset in Docker for Mac and it solved the issue.

I did a test on my Mac but it doesn't seem to improve performance when using NFS (it's even worse). What am I doing wrong?

Test results:

Benchmark - dd if=/dev/zero of=speedtest bs=1024 count=300000
* On host
300000+0 records in
300000+0 records out
307200000 bytes transferred in 1.486866 secs (206609069 bytes/sec)

real    0m1.506s
user    0m0.066s
sys 0m1.036s

* Within Docker for Mac without a host volume
300000+0 records in
300000+0 records out
real    0m 1.21s
user    0m 0.08s
sys 0m 1.13s

* Within Docker for Mac within a host volume
300000+0 records in
300000+0 records out
real    0m 4.62s
user    0m 0.05s
sys 0m 0.47s

* Within Docker for Mac within a host volume using NFS
300000+0 records in
300000+0 records out
real    0m 5.21s
user    0m 0.03s
sys 0m 0.51s

Docker for Mac: Version 1.13.1 (15353) I configured the File Sharing correctly (only /tmp is mounted)

etc/d4m-nfs-mounts.txt

/Users:/Users
/Volumes:/Volumes
/private:/private

/etc/exports


# d4m-nfs exports

"/Users" -alldirs -mapall=501:20 localhost
"/Volumes" -alldirs -mapall=501:20 localhost
if-kenn commented 5 years ago

We are officially deprecating this project. We will leave it in place in case it can help some people migrate to NFS volumes from the command line or compose.