search

cweagans / docker-bg-sync

A container that syncs files between two configurable directories.
224 stars 39 forks source link

Document new edge caching flag in docker #9

Open derimagia opened 7 years ago

derimagia commented 7 years ago

In case you didn't know, docker-for-mac has got a very much needed flag which makes this container not as needed, thankfully. :cached is implemented now and :delegated I think will be soon. They added a blog post now as well. Maybe get a link to it in the readme?

https://blog.docker.com/2017/05/user-guided-caching-in-docker-for-mac/

cweagans commented 7 years ago

I saw this, but until I have time to do some benchmarks, I'm not entirely convinced that the new functionality solves the problem. It's a bandaid at best, and it would really only improve read speeds inside the container. Don't get me wrong - this is a good thing, but it doesn't fix (for example) npm install which creates and deletes hundreds of thousands of temporary files (or at least it did whenever I last looked at it) as it goes through the process of installing dependencies. Write speed is still very much a concern.

At the very least, the improvements in read speed should make this container run a bit faster :)

derimagia commented 7 years ago

I agree, I don't think it's a solution to everything - Although I would look into yarn or to store the npm cache stuff in something that isn't mounted, but that's not the point - it should be seamless. You shouldn't need to worry about that stuff.

I think the addition to this helps more than any improvement so far. Mostly it improves where running code. (In for example Symfony for example, where it's a lot of files loaded on every request).

It does solve write speeds, there's going to be :delegated. Although that's technically not as "reliable" but I can see myself using it for things like the npm writing. Overall it just narrows the case where this is needed which is a good thing overall.

cweagans commented 6 years ago

Related:

leymannx commented 6 years ago

:cached and :delegated is quite nice and can improve performance quite a bit. But often if not to say normally in a local development environment for which cweagans/by-sync is designed for there'll be the :consistent flag in use. And using :consistent to reflect bidirectional file syncing is still a really huge performance downer.

geerlingguy commented 4 years ago

Just adding a note here that :delegated is about 2x slower than using an NFS-mounted volume (at least in my Drupal 8 testing), about the same (2x slower) than using docker-bg-sync.

With NFS volumes, the host computer setup can be a little annoying (especially on macOS Catalina, with potential permissions issues), but it's smooth sailing once it's set up. With this sync setup (docker-bg-sync), you have to wait for the (somewhat lengthy and CPU-stressing) initial sync before doing anything, and allow a 3rd party docker container to run with privileged access, which I'm a little leery of doing as a general rule (though I do trust @cweagans very much ;).

Still wish Docker team could make osxfs a lot faster :(

cweagans commented 4 years ago

Still wish Docker team could make osxfs a lot faster :(

I wish they could make it open source so that other people could make it faster. They've demonstrated that they can't or won't. :(

I'd love for this project to be totally unnecessary. NFS kind of gets there, which is nice, but it's certainly not an easy solution.

allow a 3rd party docker container to run with privileged access, which I'm a little leery of doing as a general rule

You don't necessarily need to do this. You just need to if you want more inotify watches than what you get by default.

geerlingguy commented 4 years ago

You don't necessarily need to do this. You just need to if you want more inotify watches than what you get by default.

I guess I'm unclear on what inotify watches do, then; the default according to my logs was 8192 (or thereabouts); does that mean unison will only watch 8192 of the 32,000+ files that were in my Drupal codebase, and only synchronize changes in those files? If so that makes the sync kind of useless in my case (without privileged: true and that inotify setting), since I would need it to synchronize all the files in the codebase.

cweagans commented 4 years ago

does that mean unison will only watch 8192 of the 32,000+ files that were in my Drupal codebase, and only synchronize changes in those files?

Yep, that's correct. But if you have fewer than 8192 files, then running this container with privileged: true is unnecessary. Doesn't help for Drupally stuff of course. There might also be some kind of polling mode that you can put Unison in -- I'm not sure on that one.

geerlingguy commented 4 years ago

True true, not everyone's PHP or other non-Drupal projects have zillions of dependencies with hundreds of files each :)

FYI I made my private codebase public so I can share the test results for NFS vs this vs everything else I've tried; see these two issues for the process and the benchmarks:

tl;dr this utility is neck-in-neck with docker-sync (though easier to set up IMO), and similar in overall speed to NFS (initial sync taking a long time, and some quirks encountered thereafter), though faster in the long run.

And of course it's about 2-4x faster than :cached or :delegated, and like 50x faster than the default mechanism with osxfs!