Open EugenMayer opened 5 years ago
@unnawut i would also add a documentation part to that, documenting the most common sync-stop issues and how to work arround them like npm install
or compose install
or simply choosing proper project layouts and sync excludes
For the record: For me it seems that only issue b) happens. But for me doing a docker-sync stop && docker-sync start
gets the sync back working. I never have to do a docker-sync clean
nor a docker restart to get it back working.
Not sure if that has been reported so maybe I would repeat someone else but we're working around this by manually going to the unison docker container and executing kill -1 [PID]
on the unison
process. Then after 2-3 minutes it completes going over the huge Symfony codebase that we have and continues to work normally.
Perhaps not the right place to report this, but I ran into the 'stop syncing' issue and googled upon several issue reports. What I forgot is that I moved my main shared directory (set in Preferences of Docker) and added a symlink to the previous location. (I moved it out of my ~/Documents directory). All worked well, but not smooth and often no syncs. After I added the new location directly, so not the symlinked path, in Docker preferences, all started working just fine... Please dismiss this if not at all any helpful here...
Thanks for your work Eugen!
Hi, when docker-sync worked it was a heaven sent. Way way faster. But now I am facing the issue with it working anymore. When my app starts there are no sweeping changes like they can be triggered with 'npm i' or build tasks.
I am seeing this issue here: https://docker-sync.readthedocs.io/en/latest/getting-started/troubleshooting.html#reproduction-verify-your-changes-have-been-sync-by-unison-app-sync
The only way I can sync by running
docker-sync stop && docker-sync start
in a separate shell. Which is still faster and less resource intensive in my case than regular volumes.
My fix for now has been this:
fswatch app/ | while read num ; do docker-sync clean && docker-sync start done
I put that in a shell script and run it when I start working on the app. Only calling 'docker-sync sync' is not doing anything.
Love to get it all working without those band aids:) Happy to provide more details or a sample project.
Thank you, Stan
@whoisstan thanks for that information. Well there are all sorts of those all arround to basically mittigate the issue of fswatch events being stuck on OSX or if native_osx, OSXFS events stuck on linux - pick the lesser death to die here.
We cannot really do anything here - i think currently, the best way to go here is using monit on native_osx to try to detect when OSXFS goes banana - restart it cleanly without user intervention.
But one of the best ways to remove those issues is:
npm i
composer install
and alikes AFTER docker-sync start - run them before ( so no FS events are tracked before -> it does not stuck )@EugenMayer maybe this has to be strongly called in the documentation. kind of like realistic/practical guide to using docker-sync. it's still way faster and less heavy then docker volumes.
you might want to check the cached/delegated bind options while using docker volumes!? for a small project they do just fine docker-sync was heaven-sent for doing things like npm install in a container to me
I use delegated. In my project there are 10 containers representing cache, db, queue and so on. Plus custom software in docker images, all images forming a platfrom , using docker-sync with nodemon to rebuild is great for the workflow and just way faster then regular volumes.
On Mon, Aug 5, 2019 at 12:09 PM Enrico Graziani notifications@github.com wrote:
you might want to check the cached/delegated bind options while mounting volumes!? for a small project they do just fine docker-sync was heaven-sent for doing things like npm install in a container to me
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/EugenMayer/docker-sync/issues/646?email_source=notifications&email_token=AABE3CCDK7GKJISCRBHGPLDQDBGCTA5CNFSM4GZ5GQCKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3SJQLY#issuecomment-518297647, or mute the thread https://github.com/notifications/unsubscribe-auth/AABE3CGNJVB435JAQAOPEA3QDBGCTANCNFSM4GZ5GQCA .
-- "Local color. Soak it up" Virginia Vidaura
maybe this has to be strongly called in the documentation. kind of like realistic/practical guide to using docker-sync
Eugen suggested me to add a troubleshooting guide some time ago but I havn't got around to that yet. Will do within the next few days. Feel free to nudge if that doesn't happen.
Hi all. I'm pretty sure I'm missing some troubleshooting ideas.
Feel free to suggest your ideas in this PR: https://github.com/EugenMayer/docker-sync/pull/672. I'll be happy to acknowledge you if there's a public link to the suggestion that I could link to.
Hi everyone, I'm sorry to spam but for those "docker-sync is running but not syncing" issues;
I wanted to add one more troubleshooting step that worked for me! In my case, it was actually a Unison issue, not a docker-sync problem: https://github.com/hnsl/unox/issues/24
brew unlink unox && brew link unox
- solved my issue immediately.
I hope this helps. Again, sorry for potential spam, I just wanted to record this down somewhere for other people that may be in the same spot.
Thanks for Docker Sync, it has been a life saver for us.
One more thing we can do is restart sync client when it crashes/run it via supervisord. Currently this is implemented for servers only. I'm working on improving reliability of docker-sync (pretty much this issue) and sync client can also crash for various reasons, while other things are fine. See e.g. https://github.com/hnsl/unox/issues/26 and https://github.com/bcpierce00/unison/issues/336
@ostrolucky be aware that we have different options for different strategies
So this is a little more complicated due to the different strategies
Unison client used inside native_osx strategy needs to be restarted in case of failure as wel.
unison: server restart ( monit )
Supervisord is taking this role, not monit. My post is about program crashing, not when program is stuck on high cpu usage, which I personally didn't experience happening yet.
harder without more client dependencies
Well simple, naive solution is something like while 1;do unison xxx;done
. That's no additional deps. But yeah ideally we would use something like supervisord for clients as well, but that shouldn't be too big of problem. Dependency could also be made optional if you are not comfortable with making users install more stuff.
So this is a little more complicated due to the different strategies
From what i can tell this is just about wrapping binary execution of each sync client in a (same) function, so it should be pretty consistent over all strategies and not very complex.
Unison client used inside native_osx strategy needs to be restarted in case of failure as wel.
no, when using the unison-strategy, there is no OSXFS mount, it is a server-to-client sync using macOS fs events on the client and linux inode events on the container side ( server )
Right now, there is no way around, that sync can stop. Period :)
There are different reasons for that and we can only try to "work arround" or mitigate the impact of one, the other is d4m related and we cannot do anything about it.
Scenarios where this happens ( more often ):
npm ci / fresh npm install
: same as above -> Try to always do that before you start the stack ( docker sync )composer install
is enough -> also try to run this before you start the stackIn most cases, this has nothing to do with docker-sync at all, but with OSXFS getting stucked in the FS event queue, which then also stops events for unison in our docker image ( linux, so inode events ) and thus breaks syncing.
There is no fix to that - sometimes restarting docker sync helps, this means, that is an actual unison "too many events" issue, which can also happen for unison.
So what we have here: a) OSXFS gets stuck -> docker for mac restart ist the only option, sometimes even only OS restart b) unsion gets stuck -> restarting docker-sync with
docker-sync stop && docker-sync clean && docker-sync start
helpsThere might be a way to auto-detect case b) since we already have monit ( AFAIK some already do that ) and then auto-heal itself.
We neither can avoid nor fix b) in general - it's not even sure it would need reimplemenation in Unison or its actually a OSXFS vs Inode event propagation related issue.