Finding this project pretty hopeless

[assembledadam]

Like it seems many others, I came here because Virtualbox directory sharing is rubbish.

This has fixed the speed issue of volume sharing, but it appears at the expense of introducing some other pains:

• When you have a lot of files (thousands), rsync isn't great here. It takes several minutes to copy over all the files.
• When you reboot the VM, the folder appears to vanish, requiring a fresh re-copy of all the files again. Already at this point this is impractical.
• I can't get the ignore functionality working. I specified the -i switch with a file, containing paths of directories to ignore. It detected this (echo'ed them in the script) but didn't ignore them. Either way, am I right in thinking it just copies a TAR file and then deletes the ignored files? If so that wouldn't be very useful at all (as deleting takes ages too).
• There is a significant latency in detecting change to files, in the order of 5 seconds or so. I've attempted to add the '-l' switch to reduce the fswatch latency but this has no effect (possibly because it requires the -o switch, but this script breaks using that mode). It could be due to fswatch having to monitor a large number of files, but again this makes it pretty impractical.

/Moan. Been at this for several hours now. Boot2Docker doesn't appear to work with NFS (permission denied despite the OSX share working for Ubuntu). Going to try to get https://github.com/codekitchen/dinghy working.

[brikis98]

Thanks for giving the project a try. Some responses/questions/comments on the points you brought up:

• We recently introduced a change to use tar for the initial sync, which speeds things up a bit, but perhaps not enough for really huge projects. How long is it taking? How many files?
• Are you rebooting the VM? Or destroying it and recreating it? The files are directly copied onto the boot2docker VM, so I don't see any reason they would disappear if all you did was reboot, but if you deleted and recreated it, then they certainly would.
• Is it not ignoring files on the initial sync or during the watch mode too? If it's only on the initial sync, then perhaps the recent tar change introduced a bug where the excludes are not being respected, and I'd appreciate you filing a separate bug for that specific issue.
• We do not support any flags for fswatch. If the latency flag (-l) fixes your issue, I'd definitely accept a PR to pass that flag through. To test if this is the case, could you run fswatch by itself (without docker-osx-dev) and see if you can make it work fast enough with the -l flag? FWIW, it uses FSEvents on OSX, so if that doesn't work, I don't know of a file change mechanism that would work any faster...

[assembledadam]

Sorry I was a little moany, attempting to solve a comparatively trivial issue in the big picture grates on you after hours!

Ended up getting Dinghy working relatively quickly last night, and it seems for my use case it works well. I think since you tried it they have got rid of the need for a vagrant layer so it's much simpler - may be worth giving it a go.

Completely uncached/unoptimised my app booted in between 700-900ms with Virtualbox shared folders, with local files (via docker-osx-dev) this brought it down to 250-350ms, and with dinghy it's sitting at ~290ms without any of the other troubles, and seemingly almost instant file updates. Satisfied for now!

• The app and all it's dependancies is 1.34GB, it was taking several minutes (at least 3 minutes, probably more), with 115,024 files. I just tried to time it but I'm getting permission issues with chown now from the script (probably something I've changed).
• I was just rebooting the VM. The /Users directory vanished after a reboot and required a full resync. FYI there are no shared folders on the VM.
• I actually didn't test if it was ignoring files in the sync, but it was definitely not respecting it during sync (several thousand of the 115k files are node_module files which I didn't need synced, plus ignoring .git etc)
• Yeah I edited the docker-osx-dev script to modify 'fswatch_cmd' to include the latency flag, but no joy there (as I say because I think the batch flag (-o) may be needed, but looks like the script relies on the output of fswatch without it. It could also be because of the sheer number of files, which as you say is obviously going to be a limitation whatever file change lib is used.

[thalesfsp]

@assembledadam I commented the same thing. Here, we are using NodeJS, as you can imagine, a big project have a lot of dependencies, so the node_modules/ folder have a lot of files. docker-osx-dev could be an excellent tool, but the initial rsync, and other points already pointed, make it hard to adopt. Thanks!

[fyddaben]

@thalesfsp I am a front-end engineer . and use gulp /yeoman. so this project helped me a lot about share problem.but more question occurs.such as sync problem ...

[ain]

Sync problem also has a different aspect: size. I've got a project with 13 GB of assets that are used at runtime, meaning, despite the nice and fast sync, Docker runs out of space in some 15s to minute, see #169 for a report.

[otbe]

I was just rebooting the VM. The /Users directory vanished after a reboot and required a full resync. FYI there are no shared folders on the VM.

Same here. Some clarification: docker was installed before I tried the first time docker-osx-dev. After a restart of the docker vm a docker-osx-dev call will ask you again to remove the shared "/Users" vboxsf folder. Or the other way around: after a reboot of your vm, your /Users folder is shared again and you could use it like before. This is very useful for me, because sometimes I have to use other images, where i want to mount a folder within my user folder. This folder should be synced both ways, without docker-osx-dev. Is this a bug? Any opinions?

@assembledadam Maybe you could try to containerize your build environment? We have developed a small facade project for some of our core build tools (gulp+addons) for web apps to share configs and custom plugins. This facade, lets call it "build-tool", was the only dependency of our web apps. So we could put it into a docker image and use it. Some advantages of this approach:

• ultra fast setup for new projects/developers
• no (frontend) dev has to npm i dependencies (yeah :D)
• ONE environment for build tools independent of host OS (e.g. ubuntu base image)

[brikis98]

@otbe: Not sure I understand you correctly, but in general, if you start the VM yourself with default options (e.g., boot2docker start or docker-machine create --driver virtualbox default), it will enable vboxsf and mount your /Users folder. If you start the VM using docker-osx-dev (just running the script is enough), it'll start it for you and set the necessary flags to disable vboxsf so /Users is not mounted.

[assembledadam]

@otbe That's close to what I've done, got a really nice workflow sorted out. I use Dinghy to sort out my dev base, alongside docker-compose with my pre-baked images (+ using AWS for prod and some dev resources). Been up and running for a few weeks now.

[otbe]

@brikis98 Thats fine for me. So I could choose between the behaviors. Run vm via docker-machineor docker-osx-dev.

@assembledadam Nice to hear. We just start to use docker this way and you confirm me :+1: Will give Dinghy a try!