Shared Volumes Slow

[dimasnake]

Expected behavior

File access in volumes should be comparable to access times in non-volumes, similarly to Linux installations of docker

Actual behavior

File access in volumes is many times slower than on non-volumes.

Information

Version: 1.12.3-beta29.2 (8280) Channel: Beta Sha1: 902414df0cea7fdc85b87f0077b0106c3af9f64c Started on: 2016/11/01 21:19:46.408 Resources: C:\Program Files\Docker\Docker\Resources OS: Windows 10 Pro Edition: Professional Id: 1607 Build: 14393 BuildLabName: 14393.351.amd64fre.rs1_release_inmarket.161014-1755

Steps to reproduce the behavior

Get on the commandline of a lightweight docker container

root@a6b2e82c167b:/# dd if=/dev/zero of=test.dat bs=1024 count=100000
100000+0 records in
100000+0 records out
102400000 bytes (102 MB) copied, 0.569183 s, 180 MB/s

and mount a volume:

root@a6b2e82c167b:/var/www# dd if=/dev/zero of=test.dat bs=1024 count=100000
100000+0 records in
100000+0 records out
102400000 bytes (102 MB) copied, 5.11662 s, 20.0 MB/s

In 9 times slower.

[nickjj]

@ml-msollars Yeah, that's why I roll with my above set up.

I have an SSD and an i5 3.2ghz CPU with 16GB of RAM, etc.. Waiting 10 seconds is too much to be productive when your web development code -> feedback loop is usually less than a minute.

If you follow the above guide those 10 seconds will turn into 2 seconds or however fast it would have updated on Linux.

[janhartigan]

Assuming you're running ddr4 with decent clock rate, the only variable left is the cpu. Could be that the i7 6700k is making a big difference in tandem with the ssd. Just guesswork at this point, but I have built three separate machines with the specs listed above and things operate quite smoothly.

[masaeedu]

FWIW you don't actually need vmware or any special hypervisor to unify the UI of an arbitrary Linux machine with your Windows desktop. Any Linux machine will do, and this machine can just as well be a Docker container with all your favorite tools installed (plus the docker socket volume mounted in).

Just start an X server session in Cygwin, and start the container with the DISPLAY env var pointing to your Windows machine's Docker-routable IP (assuming the Docker host is remote to your machine).

E.g. here's me running Firefox in a container:

[nickjj]

@janhartigan Things get slow at the I/O level. It's because a large web application might need to load hundreds of assets for each request (in development mode). Trying to transfer a ton of small assets is going to be very slow over any type of network based file sharing.

@masaeedu I never used cygwin but that sounds like it might be a problem to use a set up like that. For example, can you run Docker within Docker? Because that container would need to run Docker itself so you can run your own Docker based apps.

Also how would file mounting work? Does Cygwin create its own 100% safe file system which you could then mount into the container that has all of these tools -- while running at native Linux speeds?

[masaeedu]

@nickjj I don't use dind, but you can if you want. I just bind mount in the Docker socket so I have a single Docker host and caching works.

Does Cygwin create its own 100% safe file system which you could then mount into the container that has all of these tools -- while running at native Linux speeds?

I'm not really sure what this means. My Docker host isn't running on my machine; it's a remote VM I started somewhere on the network using docker-machine. Any host volume mounts you perform should use paths relative to the Docker host's file system. This isn't any different if you're manually setting up a VM to run Docker instead of just automatically provisioning it with docker-machine create foo.

For convenience I have a CIFS share with this Docker machine that I have set up as a drive on my windows box, and I just dump all my code in there while I'm working on it.

To put this another way, Cygwin isn't managing my file system or anything else; I don't even use the Cygwin shells. The only part of Cygwin I'm using is startx to forward the remote GUI to my desktop.

[nickjj]

@masaeedu In the self created VM you have the Docker daemon, the Docker CLI and all of your source code living directly inside of that VM. You never deal with any type of mounts and from Docker's POV it's just running on Linux and you never need to think about dind. I get really close to my SSD's full read/write speeds since no network based (or horribly slow vbox default) mounting takes place.

With your set up (and this is where I'm confused because I don't use cygwin) aren't you still mounting in your source code from Windows into the docker-machine driven host? Does CIFS run on Windows 10?

Do you do full time web development with this set up where you have your graphical Linux based Docker container running your code editor + your own Dockerized apps?

[masaeedu]

@nickjj It's entirely up to you where you put your source. With the scenario I described, you're still free to start up your container, which brings up xterm, from which you can git clone mystuff and run code . and start working in there. Your Windows machine simply presents the GUI from the container, the Windows filesystem stays apart from the Docker host's filesystem, and never the mane doth tweet.

I prefer to keep my code on Windows, because I occasionally need to use Visual Studio in the same project as linuxy stuff, so I use CIFS, which is a non-Cygwin, fairly mature technology, to mount a folder from the Docker host into my desktop machine as a UNC share.

[janhartigan]

@nickjj I understand, but in my environments I don't have those problems. I run a stupid large number of containers, some of them are really complicated sites with tons of assets. I also have a mysql volume mounted and I run an integration test suite that makes thousands of requests to that database and finishes in less than 4 minutes. Requests are just as fast or faster than I've ever experienced on windows alone.

[masaeedu]

Does CIFS run on Windows 10?

The CIFS server runs on the Docker host, it gets mounted as a network drive in Windows.

Do you do full time web development with this set up where you have your graphical Linux based Docker container running your code editor + your own Dockerized apps?

I have to do a lot of fiddly work involving setting up build systems and infrastructure in Docker containers, which I do using this setup.

[nickjj]

@masaeedu What benefits are there to your type of set up vs just running a unified vmware driven VM?

Because I browse and do all of my coding in that graphical vmware VM and it runs really well. 60fps video, no lag when typing, you can have multiple floating independent windows and it works across dual monitors.

Would you mind recording a short video showcasing the result of using your method? Something like running an editor on a complex dind-based web app where the editor is all set up to get context sensitive linting and code complete based on the runtime/libraries/source code of your app?

[masaeedu]

What benefits are there to your type of set up vs just running a unified vmware driven VM?

I don't have to pay for vmware, and it is faster because I'm not running on a layer 2 hypervisor.

I'll make a screen recording after work at some point.

[nickjj]

The "vmware player" edition is free btw, but it does have 1 draw back.

It's no longer maintained and vmware workstation edition completely pulled out support for running a Linux guest in unity mode. That's why I'm asking all of these questions because the one downside to my set up is you're stuck using xubuntu 14.04 because no other distros seems to work well since they also stopped supporting the guest tool add-ons.

I mean this version is rock solid and it's all fine and dandy but long term I'd rather move to something that's blazing fast and lets me use any distro/version I want within reason.

I'm really looking forward to your recording. Hit me up at nick.janetakis@gmail.com if you don't want to post it here, but I think this discussion is still relevant to Docker for Windows users because at the end of the day we just want to use Docker on Windows with top end performance (and for me at least, having a real Linux environment to work with is an added perk).

[nickjj]

I just threw together a 2 minute video that demos the vmware based development cycle if anyone is interested: https://www.youtube.com/watch?v=Xu05XpRCxQU

The TL;DR is it takes around 75ms for a Rails app to render tens of thousands of lines of JS and SCSS in development mode. You also get near instant feedback when making code changes.

[rm-bergmann]

I stumbled accross this interesting thread as I am looking for a good solution for my dev environment. I use (and will always use) windows with an ubunutu VM. I am still very new to docker. I have always used VM's and vagrant and always had performance problems using a shared directory so I have never used it, and I keep my project files on the linux file system.

The only disadvantage of this for me is my IDE (installed on windows) connects remotely to the VM, and I miss out on various features, for example search, find and replace, unit testing on file save etc.

I found a solution to mount a docker machine NFS filesystem which could potentially solve the problem? I haven't tried it yet.

https://thepracticalsysadmin.com/how-to-fix-docker-shared-folder-performance-issues/

I have been trying to use the linux subsystem on windows but it's still in beta and many things still don't work very well - but it could be a good solution in the future.

@nickjj I really liked your 'Unity Mode' set up but I was dissapointed to find that it's not supported any more wont work with ubunut 16.04. If you find a good long term solution I'd love to hear about it.

[nickjj]

@rm-bergmann On the bright side, your services will likely end up running within Docker, so getting the latest package versions for things like Postgres or nginx is easy. Using 14.x isn't as bad as it seems.

But I will say this. I spent all of yesterday testing a bunch of different things, including installing hyper-v on my box.

Docker for Windows + WSL + X server combo will render that same Rails app in the above video in about 200ms instead of 75ms which to me is very usable, but the real problem is when assets are being changed it quickly jumps to 6s (because all of the assets need to be recompiled) where as with my old Unity set up the same thing took 2.5s.

6s vs 2.5s is a huge difference because asset/UI changes are something you may change 3-4 times a minute. Being interrupted for 5s+ every time is a lot and I think this speed limitation is mainly due to how sharing drives work with hyper-v so I'm not sure if Docker for Windows can really be sped up that much? Hopefully I'm wrong!

Until then, I think I'm going to pass on it and explore running Docker straight up in a VM like I've always been doing but use hyper-v manually. This way I can copy in files directly to the VM and install other packages to run X server driven apps, etc.. I am pretty confident this will work for the future, and hyper-v could always be swapped out with vmware player if you're not running Windows 10 pro and since Unity mode is no longer in play, you could use any Linux distro / version you want.

[frutality]

Guys, what do you think about implementing NFS or that's impossible? I am using vagrant with virtualbox provider for several years, and yes, shared folders there are slow too. Also vagrant docs says NFS folders do not work on Windows hosts. Vagrant will ignore your request for NFS synced folders on Windows. but this is not true, thanks to https://github.com/winnfsd/vagrant-winnfsd

I mean, it just works without complex configs, dependencies etc. Install and go. Maybe it can be implemented in docker too?

Just for example, with default shared folder I had ~500ms response time at Laravel app. With NFS - ~100 ms. With Docker I have ~300 ms, which is better than virtualbox of course, but could be much better.

I am trying to forget about vagrant and using only docker. It is indeed usable even with ~300ms response time, but it would be great if we can get not only similar containers at dev and production environments, but also similar perfomance.

[nddipiazza]

i have a gradle build that only works in Linux. So windows PC's could still build the software... trying to use docker container to run the gradle build using volume share. previously about 8 months ago it didn't even work at all. it finally started working at least with one of the later version of docker... but it still is too slow for daily use.

Is anti-virus possibly partially to blame? I excluded the shared folder location from the virus scanner. My disk usage sits at 100% while the gradle build is going and it's just terrible slow. Same build using virtualbox is pretty good speed. Comparable to ubuntu native machine

any tips on how to get Windows + Shared volume to go faster? Instead of complaining about it i was hoping to hear more useful tips how to get the most speed possible out of it.

[barat]

@nddipiazza you can try docker-sync (it should work with windows) ... but it's still workaround/hack. Docker for Windows/Mac is quite long with us and devs still didn't fixed this huge issue. I'm just wondering if maybe something new will pop out and detronize docker by working ootb on any OS. If such thing will happen - I'll just switch. I understand that something may be hard, but this issue is like a Year old !

[nddipiazza]

i just think it's great for some simple usages and non-performance bound stuff. but for daily development workstation it will just make everything very painful.

[janhartigan]

Another update from me: In the past week, I set up a fairly complex compose environment (like ~15 shared volumes and ~12 containers) on two separate Windows machines.

The first one was the same crappy old machine I initially had this problem on, with the only difference being a brand new SSD. It works perfectly fine (pages load with no delay). Our builds are dog slow, but that's no different than running it directly from Windows...i.e. the problem is not Docker, but the CPU and memory.

I even set it up on a relatively new laptop with a pretty good (but not especially great) i7-7700HQ, an SSD, and ddr4 ~2600. None of the problems here are surfacing.

I've now built 4 independent machines with roughly these specs and I have had no problems with shared volumes:

• i7 6700k
• Any modern ssd, but this one in particular
• DDR4 3600 ram (this one)

Again...not saying you guys aren't having problems, but just trying to dispel the notion some here are proposing: that shared volumes on Docker for Windows are completely unusable in all environments. It works perfectly for me now in 3 separate Windows builds, but all of them have SSDs. Only 2 of them have respectable memory and CPU.

[er1z]

@janhartigan — although using SSDs I still can notice benefits from using a native filesystem comparing to CIFS-based one (which is being used by Dokcer for Windows for data binding). Symfony project, dev mode: 0.5s vs 20s.

[Yivan]

Same here, we done extending benchmarks and Docker for Win is just a VM with SMB making it slow like SMB ; ) We get far better performance using NFS for sharing but seems there is a bug preventing using NFS : (( (please see https://github.com/docker/for-win/issues/1241) So for now the only good alternative is Docker Toolbox + NFS.

I hope we could manage to mount NFS on Docker For Win so we get the best (simple install ore easier than Docker Toolbox) and NFS share fore the speed!

To give values, a symfony 3 apps (with /cache on the container at least):

• Homepage: 1 sec (SMB), 230 ms (NFS)
• Ajax request: 900ms (SMB), 130ms (NFS) NFS provide very very good experience on ajax GUI development for sure : )) SMB seems to have a very long transaction delay at the start...

[sav-valerio]

Any updates? At least some 'tested' workarounds. This issue really makes Docker for Windows unsuitable for web development.

EDIT: I just found docker-sync (https://github.com/EugenMayer/docker-sync). If you are on Windows I just updated the documentation, so you should not have any problems with the installation.

As a workaround until Docker finally fixes this, using docker-sync makes everything a lot more usable.

[georaldc]

@sav-valerio I tried following your instructions and everything went smoothly until the Unison step. The make UISTYLE=text step fails with a bunch of errors for me

make -C src UISTYLE=text
make[1]: Entering directory  '/path/to/unison-2.51.2/src'
UISTYLE = text
Building for Unix
NATIVE = true
THREADS = false
STATIC = false
OSTYPE =
OSARCH = Linux
ocamlopt: ubase/rx.mli ---> ubase/rx.cmi
ocamlopt -g -unsafe-string -I lwt -I ubase -I system -I fsmonitor -I fsmonitor/linux -I fsmonitor/windows -I system/generic -I lwt/generic -c /path/to/unison-2.51.2/src/ubase/rx.mli
make[1]: ocamlopt: Command not found
Makefile.OCaml:412: recipe for target 'ubase/rx.cmi' failed
make[1]: *** [ubase/rx.cmi] Error 127
make[1]: Leaving directory '/path/to/unison-2.51.2/src'
Makefile:6: recipe for target 'text' failed
make: *** [text] Error 2

[Perni1984]

Looking at https://docs.microsoft.com/en-us/virtualization/windowscontainers/deploy-containers/linux-containers#bind-mounts it seems that the new LCOW (Linux Container on Windows) are using NTFS directly for bind mounts, which should result in faster performance (although there are still some issues with popular web development packages because the POSIX implementation for the bind mount is not finished yet).

Did anyone test the performance of LCOW for web development?

[sav-valerio]

@georaldc yikes, I forgot. Run make opt after make world in the OCaml source folder, then continue following the OCaml related steps in the section.

[tonyaxo]

I solved this problem. Before it was:

time dd if=/dev/zero of=test.dat bs=1024 count=100000

100000+0 records in
100000+0 records out
102400000 bytes (102 MB) copied, 10.9452 s, 9.4 MB/s

real    0m10.952s
user    0m0.150s
sys     0m1.870s

On SSD!

My solution use Windows Subsystem for Linux

1. Install Docker for windows

2. Install WSL and Ubuntu

   ! Dont unpack ubuntu dist in system C: use D etc

3. Install docker to WSL: take installation instruction there and configuration like there

   ! Dont use sudo mkdir /c && sudo mount --bind /mnt/c /c, you shoud do it like sudo ln -s /mnt/c /c

4. Run docker under WSL

Result:

100000+0 records in
100000+0 records out
102400000 bytes (102 MB) copied, 0.10863 s, 943 MB/s

real    0m0.117s
user    0m0.010s
sys     0m0.090s

[Perni1984]

@tonyaxo Why would you need Docker for windows for that, if you install/run docker under wsl?

[georaldc]

@Perni1984 because the docker daemon doesn’t run under wsl (unless this has changed recently)?

[tonyaxo]

@Perni1984

@georaldc because the docker daemon doesn’t run under wsl (unless this has changed recently)?

I didnt find any other way to run it.

[tonyaxo]

@georaldc @Perni1984

unless this has changed recently

Good news https://blogs.msdn.microsoft.com/commandline/2017/12/04/background-task-support-in-wsl/

[vasekboch]

You cannot install Docker Engine under WSL. There is not enough API implemented to support this. So the background task support won't help. The solution is LCOW, that is currently in progress of development.

[Perni1984]

@vasekboch Ok, I already research a bit on LCOW but didn't find any performane tests etc. - did anybody test this?

Relevant github issue: https://github.com/moby/moby/issues/33850 MS documentation: https://docs.microsoft.com/en-us/virtualization/windowscontainers/deploy-containers/linux-containers

[Yivan]

Hello,

I tested LCOW and it wasn't faster (actually if ifnd it was even slower...).

@tonyaxo How can you improve performance of sharing files (CIFS or NFS) between containers and windows host using WSL, as the docker daemon is always the Docker for Windows one ? WSL can only be used to run cli command transmitted to the docker daemon, that's all.

I confirm that WSL is not on the way and not ready to support a docker full install (with daemon). Problem is docker use linux subspace/namespace and some advanced linux kernel functionnality which are for now not available in the windows linux subsytem kernel.

[Perni1984]

@Yivan do you really need to use CIFS or NFS to share files with the host when using LCOW? As far as I understood this correctly NTFS should be used directly when bind mounts are used (although the implementation is not far enough to be used in production)

[Yivan]

@Perni1984 No you don't need to set manually a sharing server (cifs/nfs). From what i remember (it was around 2 month ago), LCOW linux container doesn't access directly to host NTFS file system (windows container have direct access yes). So behind the scene there seems to be some file sharing even in LCOW. Maybe this has changed in more recent version and now LCOW with linux container provide fast direct access to the host files.

Here is the issue i opened, it has some speed test inside: https://github.com/linuxkit/lcow/issues/7

[tonyaxo]

@Yivan

@tonyaxo How can you improve performance of sharing files (CIFS or NFS) between containers and windows host using WSL, as the docker daemon is always the Docker for Windows one ? WSL can only be used to run cli command transmitted to the docker daemon, that's all.

I dont know how it works. But wsl docker cli faster widows docker cli 660 times. It is a fact. I think wsl docker transport files to windows docker demon. Why dont you try it and report about you result?

[Yivan]

@tonyaxo I tried it using the two common ways:

• installing docker (it is better to install same version than docker for Windos) on wsl too use docker cli directly inside wsl (and disabling TLS security breach) => this is the way you do ti
• creating a wrapper script to call docker cli windows binary of docker for windows (this second is the best way i think so it doesn"t involve disabling TLS as you call directly the same docker cli binary than those on the CMD/Powershell and you got the same docker version on each side)

Either way, what do you mean by faster, which test did you done, could you provide details please ? You have to test the speed from inside the container on a binded directory, not inside the WSL (for sure it is fast, it use DrvFs which map windows directory inside WSL. But docker daemon don't know of WSL and so don't use the WSL filesystem). Because WSL is just used to push command to the docker daemon of docker for windows. So ti changes nothing for container using wsl or not.

I tried it, result are here: https://github.com/linuxkit/lcow/issues/7 (see my post just above please)

[Perni1984]

@Perni1984 No you don't need to set manually a sharing server (cifs/nfs). From what i remember (it was around 2 month ago), LCOW linux container doesn't access directly to host NTFS file system (windows container have direct access yes). So behind the scene there seems to be some file sharing even in LCOW. Maybe this has changed in more recent version and now LCOW with linux container provide fast direct access to the host files.

@Yivan I found a corresponding issue that is confirming LCOW is using the 9p2000 protocol. I read some forum entries about better performance compared to CIFS/SMB. Although the windows implementation of the 9p2000 protocol is still not supporting chmod / uid, gid settings which is the underlying issue for making bind mounts not working correctly for most of the containers.

see https://github.com/docker/for-win/issues/2042#issuecomment-389800567

My research shows that there is a protocol extension called 9p2000.u which would enable to implement a chmod /uid,gid settings, and I guess that is what will be done - otherwise LCOW will not be really usable for a lot of applications in the containers.

Unfortunately I don't have found any performance results worth sharing for 9p2000 or LCOW.

BTW does anybody know how to ping/reach the Microsoft Developers that are doing the 9p2000 development on the windows side?

[trickreich]

So the usage of Docker for web development is only really usable with Linux?

[er1z]

@trickreich Indeed.

[raupie]

I've used WSL + Docker for web development on Windows for building WordPress based sites. That works fine or basic html/css framework stuff. However when you get into trying to use a heavy php framework or node that's when things fall apart. The I/O performance for reading the node_modules or vendor folder takes way too long and everything grinds to a hault. I've reformated my primary PC to Linux to take advantage of docker for my web dev needs. It's had pros and cons.

[trickreich]

@raupie When I'm developing just small things like html/css, then I don't need Docker for that ;) So you are sure, I'm talking about developing Symfony applications. For that I've switched from OSX to Linux at the beginning of the year. Docker really works great for that! But I have also some downsides with the OS at all (Mixed DPI, etc.).

[er1z]

Reloading caches under Symfony & shared folders (which occurs every request in dev env) is too long. Nope, it's pointless.

[raupie]

@trickreich. Oops now that I read that, that's pretty comical. I guess what I meant was if I need php/Apache it worked pretty well. Yeah, my work is a symfony shop (ezplatform CMS) and as @er1z said it's pointless. There are tricks you can do such as changing the cache dir to stay inside the container and not in the shared volume. There's also a git repo called docker-sync which uses unison to sync the files/folders between local and docker container. So you don't sync the vendor folder in a shared volume and you sync the vendor dir whenever you update or install composer. That's a headache but does work.

[trickreich]

But the fastest options is still Linux, that's the point :)

[raupie]

100%, that's why I switch to Linux. It just worked as expected. It sucks because WSL is soooo close to being useful from a web development workflow. My work IT won't reformat our windows workstations so we have to use a remote vsphere vm to develop in which has drawbacks. Hate the idea of developing in a local virtual machine as well. Managing two machines is annoying.

[trickreich]

Yes.. same here!

I thought WSL could be maybe another option until i read this article: https://www.phoronix.com/scan.php?page=article&item=windows10-wsl-docker&num=2

[luckydonald]

It's usable from macOS. There are less differences between mac and linux, so it runs a lot better then windows at least, as it doesn't have to fake so much (file permissions, cAsEiNsEnSiTivE file systems, ...) But yeah, on window's it's a deal breaker, I'm the only mac in the company therefore we can't use it.

[trickreich]

@luckydonald I've switched from OSX to Linux because it's not usable! It's also terrible slow. https://github.com/docker/for-mac/issues/77