Not syncing both directions suddenly

nurtext commented 7 years ago

Error/Feature Requestion/Docs

All docker container are up, but not syncing between /host_sync and /app_sync. /app_sync to /host_sync is working.

Docker Driver

d4m

Sync strategy

native_osx

docker-compose.yml

version: '2'

services:
    api:
        container_name: api
        image: api
        build: .
        ports:
            - 8888:80
        volumes:
            - ./app:/srv/api/app
            - ./bin:/srv/api/bin
            - ./src:/srv/api/src
            - ./tests:/srv/api/tests
            - ./web:/srv/api/web
            - ./vendor:/srv/api/vendor
        environment:
            DATABASE_HOST: "127.0.0.1"

docker-compose-dev.yml

version: '2'

services:
    api:
        volumes:
            - api-app-sync:/srv/api/app:nocopy
            - api-bin-sync:/srv/api/bin:nocopy
            - api-src-sync:/srv/api/src:nocopy
            - api-tests-sync:/srv/api/tests:nocopy
            - api-web-sync:/srv/api/web:nocopy
            - api-vendor-sync:/srv/api/vendor:nocopy

volumes:
    api-app-sync:
        external: true

    api-bin-sync:
        external: true

    api-src-sync:
        external: true

    api-tests-sync:
        external: true

    api-web-sync:
        external: true

    api-vendor-sync:
        external: true

docker-sync.yml

version: '2'

options:
    verbose: true

syncs:
    api-app-sync:
        src: './app/'

    api-bin-sync:
        src: './bin/'

    api-src-sync:
        src: './src/'

    api-tests-sync:
        src: './tests/'

    api-web-sync:
        src: './web/'

    api-vendor-sync:
        src: './vendor/'

`docker ps` output

CONTAINER ID        IMAGE                            COMMAND                  CREATED             STATUS              PORTS                            NAMES
72ee4eb5a02b        api                              "docker-php-entryp..."   2 minutes ago       Up 2 minutes        9000/tcp, 0.0.0.0:8888->80/tcp   api
7ec87ccaa84f        eugenmayer/unison:hostsync_0.2   "/entrypoint.sh su..."   15 minutes ago      Up 2 minutes        500/tcp                          api-vendor-sync
de22a140b897        eugenmayer/unison:hostsync_0.2   "/entrypoint.sh su..."   16 minutes ago      Up 2 minutes        500/tcp                          api-web-sync
b1d749f9c992        eugenmayer/unison:hostsync_0.2   "/entrypoint.sh su..."   16 minutes ago      Up 2 minutes        500/tcp                          api-tests-sync
62d9a55979a3        eugenmayer/unison:hostsync_0.2   "/entrypoint.sh su..."   16 minutes ago      Up 2 minutes        500/tcp                          api-src-sync
0af351c80ed8        eugenmayer/unison:hostsync_0.2   "/entrypoint.sh su..."   16 minutes ago      Up 2 minutes        500/tcp                          api-bin-sync
45d5f50d0659        eugenmayer/unison:hostsync_0.2   "/entrypoint.sh su..."   16 minutes ago      Up 2 minutes        500/tcp                          api-app-sync

`docker-sync list` output

      ok  Found configuration at /Users/cedric/api/docker-sync.yml

api-app-sync On address : api-bin-sync On address : api-src-sync On address : api-tests-sync On address : api-web-sync On address : api-vendor-sync On address :

OS

macOS 10.11.6

zedtux commented 7 years ago

Short update on this ticket, I'm exploring the -fastcheck flag in order to solve this issue.

Fast Update Detection

If your replicas are large and at least one of them is on a Windows system, you may find that Unison's default method for detecting changes (which involves scanning the full contents of every file on every sync—the only completely safe way to do it under Windows) is too slow. Unison provides a preference fastcheck that, when set to true, causes it to use file creation times as 'pseudo inode numbers' when scanning replicas for updates, instead of reading the full contents of every file.

When fastcheck is set to no, Unison will perform slow checking—re-scanning the contents of each file on each synchronization—on all replicas. When fastcheck is set to default (which, naturally, is the default), Unison will use fast checks on Unix replicas and slow checks on Windows replicas.

This strategy may cause Unison to miss propagating an update if the modification time and length of the file are both unchanged by the update. However, Unison will never overwrite such an update with a change from the other replica, since it always does a safe check for updates just before propagating a change. Thus, it is reasonable to use this switch most of the time and occasionally run Unison once with fastcheck set to no, if you are worried that Unison may have overlooked an update.

Fastcheck is (always) automatically disabled for files with extension .xls or .mpp, to prevent Unison from being confused by the habits of certain programs (Excel, in particular) of updating files without changing their modification times.

I updated my docker-sync.yml file by adding sync_args: '-fastcheck no', then I did a docker-sync-stack clean and then docker-sync-stack start. Since yesterday afternoon is working so I'm continuing with this config and will update this ticket if any issue occurs again.

zedtux commented 7 years ago

I worked the whole morning with this setup without any sync issue anymore. Looks promising! 😎

EugenMayer commented 7 years ago

@zedtux i do not expect this to work, since our issue is rather not unison and its detection, much more it is the OSXFS event layer stalling - and fastcheck will not help here at all, since it tries to make unison faster ( and probably it will ) - but without fs events, unison does nothing - and that is what i strongly expect to happen in those cases.

If it would be unison, restarting the sync container ( unison ) would help - it does not. Restarting the stack would help too ( obviously ) - it does not. Restarting d4m does -> restarting hyperkit -> resetting OSXFS

zedtux commented 7 years ago

fastcheck will not help here at all, since it tries to make unison faster ( and probably it will )

Well actually I'm slowing down it by setting it to no.

Okay, maybe I'm wrong, but anyway, in my case I had sync issue after a while, and thanks to this, it is now working fine.

Sorry for the noise.

EugenMayer commented 7 years ago

@zedtux no sorry needed, i am glad you try to fiddle this out - and mine are assumptions on stuff we did find out - but my opinion is far from being a fact. So thinking in all direction is great help - by no means its noice - its rather trying and at very least, removing one possibility from the equation or even find the solution

michaelbaudino commented 7 years ago

That's actually a good idea to try @zedtux 👍 But like @EugenMayer I don't think it'll work, because (in my understanding!) what -fastcheck no does is to use file content (instead of file modification time) to determine if the file has changed. But (as far as I understand, again!), those checks (whatever method is used) are triggered by filesystem events… and those filesystem events not being passed through by OSXFS is the root of our problem. Thus, whatever check detection we configure, I'm afraid it won't be used anyway 😢

But that's absolutely no noise, and indeed worth a try 🤗

In the same kind of thinking, maybe unison-fsmonitor has an option to check the whole tree every N seconds rather than relying on FS events? If such a setup prevents this issue, it would confirm our hypothesis again and would definitely narrow down the scope of investigation (I didn't find any documentation about such an option, unfortunately, and lost myself in the Unison Makefiles and OCaml code when trying to figure out how it was working… src/fsmonitor/linux/watcher.ml? src/fswatch.ml? src/unison-fsmonitor.py? 😱)

mike-es commented 7 years ago

I have a few notes that support the suspicion of OSXFS events stalling. I am working on a PHP app using Laravel

files created in php container sync to host
files created on host sync to /host_sync but unison fails to sync to /app_sync with no logs
restarting unison causes files to sync (supervisorctl restart unison)
all sync containers break at the same time (3 total)
time/tz are the same between host and sync containers
files created directly in /host_sync sync to /app_sync
it seems like sync breaks when laravel throws an error (logs are excluded)

Docker version: Docker version 17.09.0-ce, build afdb6d4

Docker sync version: 0.4.6

macOS version: 10.12.6 (16G29)

zedtux commented 7 years ago

Thank you guys for your comments. Honestly, since I set this flag, all my files are well sync, full stop.

Maybe give it a try, and see if it solves the issue. If it does, then we can try to tacle this issue from a new angle.

DavidDeSimone commented 7 years ago

My team and I are also experiencing this issue. I am able to reproduce it pretty easily by switching around our repo to older changesets. I'm seeing it with a changeset that effects around 15,000 files. The behavior I was seeing was consistent with what other's have commented, inotifywatch/inotifywait were not seeing events, restarting d4m fixes the issue.

Since I am able to repro this, I will try to verify @zedtux suggestion.

I am attempting to create a sample project that can isolate this issue. We will be able to get help from the docker team easier if we can give them a project that can easily show off what is happening. It seems to be difficult to trigger for small data changes (> 200M). I might just clone a large repo like emacs and jump around that to trigger the behavior in a way that will be accessible to everyone.

murnieza commented 7 years ago

Our team is also heavy suffering from this issue. We even dropped docker for now and switched to Vagrant env.

mike-es commented 7 years ago

I tried @zedtux suggestion by adding sync_args: '-fastcheck no' to 2/3 sync configs. I still run into this issue on all 3 sync containers.

DavidDeSimone commented 7 years ago

Based on my testing, it seems as if there is a building queue of inotify events that either isn't being flushed, or is very slow to flush. If I rapidly switch branches, I will no longer see events on a monitored file if I edit it from my mac, however I always see changes instantly if I update from within the container. The timeline is:

Rapidly change a large number of files (15000+)
Edit a file that is not in the changeset from OSX. (no inotify)
Edit that same file from within the container (inotify)

Sometimes if I wait long enough, all of the "pending" inotify events seem to happen at once, however this doesn't always happen.

zedtux commented 7 years ago

@mike-es did you run the docker-sync-stack clean? What do you do which makes the sync no more working? Just working normally? Switching branches? Some thing else?

mike-es commented 7 years ago

I'm using docker-sync start|clean but yes I did clean. I caused the issue by switching branches on the repo without -fastcheck no which I would expect to break that sync but not the others. However, all of them broke.

Is that configuration still working for you?

mike-es commented 7 years ago

@zedtux I just ran another test. I ran docker-sync clean and added sync_args: '-fastcheck no' to all sync configs. When I brought up my stack syncing was broken right off the bat. I have not restarted d4m today.

mike-es commented 7 years ago

I ran another test inspired by docker/for-mac#681. I wanted to see if inotify events broke for other containers that have nothing to do with docker-sync.

Dockerfile:

FROM alpine

RUN apk --no-cache add inotify-tools
COPY wait.sh /bin/

Wait script (wait.sh):

#!/bin/sh
FORMAT=$(echo -e "\033[1;33m%w%f\033[0m written")
"$@"
while inotifywait -q --format "$FORMAT" /mnt
do
  "$@"
done

Run command (672463996aa6 is image build from Dockerfile above):

docker run --rm -it -v /path/to/UnrelatedProject:/mnt 672463996aa6 /bin/wait.sh

Test 1: I switched branches multiple times in UnrelatedProject which is a fairly large code base. Each time I switched I got inotify events from the wait script. No issues.

Test 2: I started up my stack using docker-sync start and docker-compose up. I created a file in UnrelatedProject and saw inotify events. No issues.

Test 3: I switched branches in 1 (out of three) synced repos while watching docker-sync logs -f. The files synced correctly. Then I created a file in UnrelatedProject and saw the inotify event. I repeated this a few more times and slowly started losing inotify events from UnrelatedProject until eventually all events stopped and docker-sync stopped working.

Test 4: I brought down my stack (docker-compose down and docker-sync clean) and I inotify was still broken for UnrelatedProject.

Conclusion 1: This issue has nothing to do with docker-sync and is purely a d4m issue, possibly related to file descriptors since I'm syncing 3 different repos and switching branches.

Conclusion 2: Based on Test 1 causing no issues, the docker-sync unison image is doing something funny that is interfering with the FSEvents/inotify of d4m as a whole. I don't have a lot of knowledge in this area but this seems unlikely.

**UPDATE**

I was able to reproduce the issue by starting 4 of the above containers and not touching docker-sync at all. I bind mounted the 3 repos in my stack and 1 for UnrelatedProject. After switching branches in a few of the repos the inotify events slowed then stopped. This points to a d4m issue and not docker-sync.

Unfortunately this is a major blocker for my team using Docker at all. I switched to the rsync strategy which is working pretty well, but still running into a few issues.

zedtux commented 7 years ago

Is that configuration still working for you?

@mike-es yes it is.

When I brought up my stack syncing was broken right off the bat.

I never faced something like this.

@mike-es what version of d4m are you using?

I tried to change branches and so on but all is working fine for me 😕

EugenMayer commented 7 years ago

@mike-es impressive deduction and system work here. So know with https://github.com/EugenMayer/docker-sync/issues/410#issuecomment-337402855 we can take the suggestion we had https://github.com/EugenMayer/docker-sync/issues/410#issuecomment-316604056 and https://github.com/EugenMayer/docker-sync/issues/410#issuecomment-316614966 is a fact finally.

its fsevents related and since you can force it using a higher amount of containers what it tells us is, the amount of events is the key factor, correlating to the amount of files "changeable" - this explains why some of us have no issues or very rare and some of use have more of those. It depends on how many files you change at one go. So gulp/grunt/webpack stacks will have more problems then ruby/PHP/java projects.

@mike-es and all thank you a lot on participating on this issue and helping to iron out the core of the problem.

I would say, we probably know, using the isolated description of @mike-es create a d4m issue to see if they can give us more insight or even a fix or even something how to flush the event queue without restarting the stack. What do you guys think?

EugenMayer commented 7 years ago

Oh @mike-es i have see you already joined a existing issue on that - we had that on our cause list already.

@all please join that issue, bump up using your thumb up to get traction to that issue please: https://github.com/docker/for-mac/issues/681

zedtux commented 7 years ago

Just to inform you that after more than a day without any issues, I added the flag to all the other projects (in the same docker-sync.yml file), I did clean everything and started again the stack but I'm now facing again the sync issue 😕

EugenMayer commented 6 years ago

@MichaelHindley instead of ranting right away, you might rather starting reading the thread - this is disrespectful you you are pleased to use something else.

docker-sync is not broken, we even cannot do anything about that here at all.

If you use d4m straight, no docker-sync involved, mount in a volume and run a watch task in your container, then start changing code - after some time, as outline here, OSXFS will struggle with FS events and your watch task wont work anymore. No docker-sync involved and zero you can do about this.

But why i am even repeating this at all, its just the same story again, you did not even dare to read this thread or understand, you are just here to dump your rant.

Beside that, thousands of us using docker-sync on a daily base - and yes, once a day this can happen. Inconvinient, yes, steals you 5 minutes restarting d4m and the stack, yes.

but you lose those 5 minutes every hour without using docker-sync waiting for your random framework to evne load up

EugenMayer commented 6 years ago

@MichaelHindley we do not "advertise" - thats not a commercial product. Thats not a bold claim to fool you in and take your money. People in here have 0 benefits from fooling you, lieng to you or gaining anything from false claims.

You are talking about being defensive, while you go straight in swinging the axt.

Please, feel free to make any suggestion were you have spend time and effort to think through it anytime - other then that -if you do not like docker-sync because "it just does not work and all the other stupid people got fooled and just do not get it" ..

please pick a better tool working for you and leave. Not going to comment on this further, just dragging it offtopic

mmrko commented 6 years ago

Meanwhile, the rsync strategy works beautifully. Kudos to @EugenMayer & co. for their hard (and gratuitous) work! :)

EugenMayer commented 6 years ago

@mmrko yes, rsync is not affected by this, thats why having multiple strategies is one of our goals - pick the axe for your case. The problem with rsync is on the other side, CPU load, since it uses fswatcher to watch for file changes - and due to macOS being horrible in FS-Events, even that is just bad as it is. So its pretty bullet proof, but not entirely.

But more or less, all strategies have their little edges, thats assured. But running drush for streight up 2 minutes for a drush cc all on the cli alone, instead of 3 seconds - is a much harder "edge" for most of us, i guess.

I would love to remove docker-sync from this world, if it would not be needed .e.g d4m fixed it. That will be a great day for us all - no overhead, no glitches, no extra configuration - just docker.

Well d4m is not just docker, its docker on a non linux kernel...and thats where we all start.

EugenMayer commented 6 years ago

Closed the thread for collaborators only so this thing does not get offtopic even more - way to important issue with way to valuable information, no need for all the other.

Will reopen the issue in 1 week when things calmed down. If you feel in the need to express yourself about this, open a new issue in the queue so this one keeps the focus

EugenMayer commented 6 years ago

Anybody having this issue, please see https://github.com/EugenMayer/docker-sync/issues/497 and run the reproduction script, post your details.

Also ensure you test it 2 times, one without my d4m fs fix and one with: https://gist.github.com/EugenMayer/07d24d4b1b88da3e48d90052192649a2

Does the result change for you? This could really help use nailing down the issue.

Why i am assuming that my fix helps is, that neither i have this issue often ( about 1 time in 3 months ) neither anybody of my coworkers - we are doing heavy development though, it means, one project most probably having 0.5 million files, 5 docker-sync endpoints starting at the same time and a stack which consume 8GB Ram + quiet some CPU. Still it hardly ever happens for us ( i think for most of us, it never happened at all). We have other projects, bigger Java projects, bigger ng4 stacks and so on, still it hardly ever happens for us.

So there must be something we do that this does not happen. One thing i can tell is:

we are either using Sierra or Mountain Lion
we are all using that fix above
we are all having about 8-16GB RAM assigned to d4m
at least 2-4 CPUs ( depending if it is a desktop or laptop )

EugenMayer commented 6 years ago

i am currently thinking about closing this issue and open a new one, extracting all the facts / informations we had here, also how to test owns system to provide informations so we can see pattern. I would link this issues as the source of deeper discussions.

Any concerns about that? Main reason is, to consolidate the results and make it easier to understand how this issues shows up, how you can for now work around and what the reasons for this are, without reading so many posts.

zedtux commented 6 years ago

@EugenMayer I agree with you, it should also allow new comers to quicker understand the issue and where we stand.

OO00O0O commented 6 years ago

And I think we should have in mind, what projects we are doing. Because I'm working on symfony project, and I only sometimes have conflicts and had this no sync problem only maybe twice in 5 months. But frontend team working with same config, having this problem few times a week. They using Angular 1/2.

version: "2"
syncs:
  backend-sync:
    src: '../backend'
    sync_strategy: 'unison'
    sync_excludes: ['.git', '.idea']
    # sync_excludes: ['.git', '.idea', 'var/cache']
    sync_excludes_type: 'Path'
    watch_strategy: 'fswatch'

  frontend-sync:
    src: '../frontend'
    sync_strategy: 'unison'
    sync_excludes: ['.git', '.idea']
    sync_excludes_type: 'Path'
    watch_strategy: 'fswatch'

b1alpha commented 6 years ago

Is this fixed? A few devs are starting to see docker-sync just stop. logs dont show any issue its just not picking up any changes. clean and start didnt seem to resolve then about 5min later it started working again

ddanier commented 6 years ago

@EugenMayer Your gist (https://gist.github.com/EugenMayer/07d24d4b1b88da3e48d90052192649a2) seems not to work any more. I get "fatal: not a git repository (or any of the parent directories): .git". Is there an updates version? Thank you!

OO00O0O commented 6 years ago

Our team gave up. We went to bare metal dev env. Some have VMware.

markshust commented 6 years ago

If you use the native delegated volumes, docker-sync isn't needed anymore as far as I'm concerned. I've had solid performance on Magento 2 (TONS of folders/files) with no delay or halts in syncing.

EugenMayer commented 6 years ago

closing this, use d4m edge to get it very robust, we have a about one sync stop per week at max

EugenMayer commented 6 years ago

@ddanier this is no longer needed with d4m edge

ohcibi commented 5 years ago

I just started using docker-sync with latest versions of everything. Initial sync works but then nothing gets sync. Even with explicitly running docker-sync sync. I just saw an error message:

unix:///tmp/supervisor.sock refused connection

When I tried to restart everything as suggested here.

EugenMayer commented 5 years ago

I would suggest opening a ticket, paste your configuration there and continue from there

n00b21337 commented 5 years ago

Maybe just to sum this up. Solution for this would be to either put docker4mac edge or use rsync as sync strategy?

EugenMayer commented 5 years ago

d4m will do it (in the meantime)

Or to just understand how to configure your excludes and have a proper project layout mitigating the issues of d4m.

Reduce the amount of files you

watch
you sync

to only the parts you edit.

infostreams commented 4 years ago

Maybe I was to stupid and didn't install everything correctly, but I think I followed the installation instructions correctly. However, docker-sync never worked for me.

I mount folder X from my Mac to docker-sync, and I'm running docker-sync with docker-machine. If I changed a file in folder X on my Mac, I would see it in the host_sync/ folder, but unison never picked up the change, and the change never ended up in the app_sync/ folder, and thus never in my own docker container.

However, if I opened a shell in the docker container running docker-sync, and manually made a change in the host_sync/ folder from within the docker-sync container (e.g. by doing touch /host_sync/examplefile.txt), then unison would see that something changed, and copy it to the app_sync/ folder. So, the problem was with Unison, somehow.

After a long search it turns out that this particular behavior is related to a WONT-FIX bug in VirtualBox (see here and here). Basically, a change to a file that was initiated on my Mac does not result in a filesystem event within the container, and is therefore never picked up by Unison. Unison's documentation even explicitly states this (see documentation: "In particular, the Unix implementation does not compare the actual contents of files to their previous contents, but simply looks at each file's inode number and modtime; if neither of these have changed, then it concludes that the file has not been changed.")

So, I'm now running a separate program inside the docker-sync container that regularly polls my host_sync/ folder, and whenever it detects a change, it touches the file, which causes a filesystem event inside the container, which then causes Unison to pick up the change, which means my changes end up in my own container 🎉

I initially installed inotify-proxy to do this, but I didn't manage to get it to work properly, so I coded up my own version in PHP. It's plenty fast and seems to work as intended. Hope this helps someone here.

If this whole story above is old news and there's existing code in docker-sync to deal with this, can you point me to it, and either update the documentation to point it out clearly, or make it the default behavior? As I interpret the Unison documentation ("considered unchanged if inode is the same") and the VirtualBox documentation ("we don't pass on inode changes") I don't think the existing setup could have ever worked for Docker running on VirtualBox.

EugenMayer commented 4 years ago

@infostreams this and more reasons are, why you should use OSXFS, thus d4m and not vbox/fusion, since the FS driver of OSXFS is better in those terms and fires all the events.

There is no doubt that docker4mac and OSXFS has issues too, but i consider it the better alternative.

Thanks for sharing your story in that detailed fashion though, very appreciated!

infostreams commented 4 years ago

@EugenMayer I don't think the documentation mentions anywhere that this, out of the box, doesn't work with docker-machine. If I had known that I could have resolved my issues by switching to docker4mac, I would have done that. As it is I spent a day or so on this problem. Can you update the documentation to say something along these lines? It would have saved me (and no doubt others) a lot of frustration.

EugenMayer commented 4 years ago

@infostreams checking the docs, i really have no topic about d4m vs docker-machine dedicated to this topic beside the tests under https://docker-sync.readthedocs.io/en/latest/miscellaneous/performance.html

But we had this https://github.com/EugenMayer/docker-sync/issues/346 ..so exactly your finding so indeed we could have saved some time for you.

But that is how docs are, they are never complete.

We have a topic about how to pick the right strategy at https://docker-sync.readthedocs.io/en/latest/advanced/sync-strategies.html and that currently suggest OSXFS eventhough it does not explain all the details, including those details - but following that guide would have saved you time too :)

Anyway, we should probably add a "docker-machine vs d4m/d4w" topic - are you willing to do a PR for that? That would be awesome

Update: it was there: https://docker-sync.readthedocs.io/en/latest/advanced/sync-strategies.html

It works under Docker for Mac only - missing file system events under vbox/fusion. See native_osx does not work with docker-machine vbox / fusion

under cons

infostreams commented 4 years ago

Hi, thanks for pointing out that reference. However, I completely missed that single sentence, and I wouldn't be surprised if many others missed it as well. I will open a pull request to add a single sentence to the installation instructions for Mac, because only saying "With native_osx we no longer have any host dependencies" is slightly misleading since it only works on one of the two ways to run Docker on Mac. Thanks for your great work!

ostrolucky commented 4 years ago

Hey guys, I have run into this issue recently. I've read the thread and it mentions it shouldn't happen with d4m edge anymore. But here we are, with d4m 2.2.3.0 edge and it still happens. Restart of my Mac fixed it. I think I didn't try restarting Docker itself though, could maybe help as well.

EugenMayer commented 4 years ago

@ostrolucky this is known and edge does not fix it ( or anything ). Either FSevents get stuck inside macOS ( macOS restart needs restart ) or inotify events inside the kernel of d4m ( docker restart might help ) and sometimes it's unison inside the sync container getting overheated by the amount of inotify events ( huge file change chunks ) which can be fixed by either restarting unison and sometimes in addition removing the lockfile of unison

To be frank, this is unfixable by docker-sync - this is broken conceptually by macOS ( horrible FSevents implementation ), leading to a half baked OSXFS implementation on top of it, and in the end unison, which simply cannot handle fast changes of 20k files like an npm install would do

ostrolucky commented 4 years ago

this is known and edge does not fix it ( or anything )

Good, this is just to make that transparent, because reading this thread it implies this was fixed in d4m edge. So good to make that clear that it indeed doesn't fix it.

which simply cannot handle fast changes of 20k files

Do you think mutagen can? If it does, perhaps we should write a strategy for it in docker-sync. Because TBH it's annoying having to exclude such folders with dependencies. Although we are a bit OT now, but there is no community forum or anything like that for docker-sync AFAIK.

EugenMayer commented 4 years ago

@ostrolucky there is no community forum and yes, the latter might be fixable by mutagen, but nobody really tried - see #603 for a issue about implementing mutagen. Feel free to join there and maybe even contribute with a PR or try to bring mutagen to those edge cases.

I will close this thread with repeating the answer above so people now the "essence" - got your point, good idea

EugenMayer commented 4 years ago

d4m does not fix it - there are several layers of issues:

Either FSevents get stuck inside macOS ( macOS restart needs restart ) \
or inotify events inside the linux kernel of d4m ( docker restart might help )
and sometimes it's unison inside the sync container getting overheated by the amount of inotify events ( huge file change chunks ) which can be fixed by either restarting unison and sometimes in addition removing the lockfile of unison

To be frank, this is unfixable by docker-sync - this is broken conceptually by macOS ( horrible FSevents implementation ), leading to a half baked OSXFS implementation on top of it, and in the end unison, which simply cannot handle fast changes of 20k files like an npm install would do.

So there are cases where sync stops, many of them fall in the one mentioned above - and we can only try to mitigate those

EugenMayer / docker-sync