docker-sync doesn't start in background

seanhandley commented 6 years ago

I've been using docker-sync without issue for a few weeks. Today I ran out of disk space and the main docker process stopped responding. I had to force a reboot for the machine and now disk space is plentiful and docker is working.

Unfortunately, docker-sync is no longer syncing my file changes nor is it allowing itself to start daemonized. I have made no config changes

$ docker-sync start
          ok  Starting native_osx for sync app-sync
doing initial sync with cp
Contacting server...
Looking for changes
(hangs)

If I make file changes they are not synced into the container (verified by running docker-compose exec <container> bash and using cat to check the file contents.

When directly calling it in the foreground, it gives the following output and eventually terminates itself:

 docker-sync start --foreground
       note:  Starting in foreground mode
          ok  Starting native_osx for sync app-sync
          ok  app-sync container still running, restarting unison in container
unix:///tmp/supervisor.sock no such file
     success  Sync container started
     success  Showing unison logs from your sync container: app-sync
Reconciling changes
Synchronization complete at 12:31:35  (0 item transferred, 41 skipped, 0 failed)
  skipped: .bash_history (contents changed on both sides)
  skipped: .dbdata/ib_buffer_pool (contents changed on both sides)
  skipped: .dbdata/ib_logfile0 (contents changed on both sides)
  skipped: .dbdata/ib_logfile1 (contents changed on both sides)
  skipped: .dbdata/ibdata1 (contents changed on both sides)
  skipped: .dbdata/ibtmp1 (contents changed on both sides)
  skipped: .dbdata/mysql/innodb_index_stats.ibd (contents changed on both sides)
  skipped: .dbdata/mysql/innodb_table_stats.ibd (contents changed on both sides)
  skipped: .dbdata/stuart_development/active_admin_comments.ibd (contents changed on both sides)
  skipped: .dbdata/stuart_development/driver_rating.ibd (contents changed on both sides)
  skipped: .dbdata/stuart_development/job_quote.ibd (contents changed on both sides)
  skipped: .dbdata/stuart_development/order_line.ibd (contents changed on both sides)
  skipped: .dbdata/stuart_development/package_properties.frm (contents changed on both sides)
  skipped: .dbdata/stuart_development/package_properties.ibd (contents changed on both sides)
  skipped: .dbdata/stuart_development/package_transport_types.frm (contents changed on both sides)
  skipped: .dbdata/stuart_development/package_transport_types.ibd (contents changed on both sides)
  skipped: .dbdata/stuart_development/payment_method.frm (contents changed on both sides)
  skipped: .dbdata/stuart_development/payment_method.ibd (contents changed on both sides)
  skipped: .dbdata/stuart_development/payment_method_rule.frm (contents changed on both sides)
  skipped: .dbdata/stuart_development/payment_method_rule.ibd (contents changed on both sides)
  skipped: .dbdata/stuart_development/payment_platform_rule.frm (contents changed on both sides)
  skipped: .dbdata/stuart_development/payment_platform_rule.ibd (contents changed on both sides)
  skipped: .dbdata/stuart_development/phone_call_sessions.frm (contents changed on both sides)
  skipped: .dbdata/stuart_development/phone_call_sessions.ibd (contents changed on both sides)
  skipped: .dbdata/stuart_development/place.frm (contents changed on both sides)
  skipped: .dbdata/stuart_test/base_user.ibd (contents changed on both sides)
  skipped: .dbdata/stuart_test/billing_account.ibd (contents changed on both sides)
  skipped: .dbdata/stuart_test/client.ibd (contents changed on both sides)
  skipped: .dbdata/stuart_test/client_account.ibd (contents changed on both sides)
  skipped: .dbdata/stuart_test/client_device.ibd (contents changed on both sides)
  skipped: .dbdata/stuart_test/oauth_access_tokens.ibd (contents changed on both sides)
  skipped: .dbdata/stuart_test/oauth_applications.ibd (contents changed on both sides)
  skipped: .dbdata/stuart_test/oauth_resource_owners.ibd (contents changed on both sides)
  skipped: .dbdata/stuart_test/referral.ibd (contents changed on both sides)
  skipped: .dbdata/stuart_test/referral_code.ibd (contents changed on both sides)
  skipped: .dbdata/stuart_test/wallet.ibd (contents changed on both sides)
  skipped: .docker-sync/daemon.log (contents changed on both sides)
  skipped: .git/index (contents changed on both sides)
  skipped: .stuart_test_db_manager (contents changed on both sides)
  skipped: log/newrelic_agent.log (contents changed on both sides)
  skipped: spec/requests/api/v2/clients_spec.rb (contents changed on both sides)
real    3m 9.48s
user    0m 40.39s
sys 0m 14.88s

Docker Info

$ docker info
Containers: 6
 Running: 5
 Paused: 0
 Stopped: 1
Images: 18
Server Version: 17.09.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 3f2f8b84a77f73d38244dd690525642a72156c64
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.49-moby
Operating System: Alpine Linux v3.5
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.855GiB
Name: moby
ID: RYTB:L66P:CPD3:L4MP:TKID:ES54:RKJW:6MZF:IRCJ:KP5C:EZ26:X4HS
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 68
 Goroutines: 78
 System Time: 2017-12-11T12:41:42.185953192Z
 EventsListeners: 1
No Proxy: *.local, 169.254/16
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Sync strategy

native_osx/default

your docker-sync.yml

version: "2"
syncs:
  app-sync:
    sync_excludes: ['.git/', 'tmp']
    watch_excludes: ['.*/.git', 'tmp']
    src: '.'

OS

macOS High Sierra 10.13.1
docker 17.09.0-ce-mac35

EugenMayer commented 6 years ago

Please reset your docker for mac entirely and try again (Factory reset). Be sure to clean .docker-sync folder before trying again. Does this help?

seanhandley commented 6 years ago

Just tried this. It still hangs at Looking for changes:

$ docker-sync start
Status: Downloaded newer image for eugenmayer/unison:hostsync_0.2
          ok  Starting native_osx for sync app-sync
doing initial sync with cp
Contacting server...
Looking for changes

EugenMayer commented 6 years ago

please try the default boilerplate here https://github.com/EugenMayer/docker-sync-boilerplate does this one work for you? If not, please restart your mac once and try again

seanhandley commented 6 years ago

Interestingly, it eventually wakes up and does this:

Reconciling changes
Propagating updates
UNISON 2.48.4 started propagating changes at 13:44:15.14 on 11 Dec 2017
[BGN] Copying .DS_Store from /host_sync to /app_sync
[END] Copying .DS_Store
[BGN] Copying .bash_history from /host_sync to /app_sync
[END] Copying .bash_history
[BGN] Copying .bundle from /host_sync to /app_sync
[END] Copying .bundle
[BGN] Copying .cane from /host_sync to /app_sync
[END] Copying .cane
[BGN] Copying .codeclimate.yml from /host_sync to /app_sync
[END] Copying .codeclimate.yml
[BGN] Copying .dbdata from /host_sync to /app_sync
^C/Users/sean/.rbenv/versions/2.2.5/lib/ruby/gems/2.2.0/gems/docker-sync-0.5.2/lib/docker-sync/sync_strategy/native_osx.rb:84:in `system': Interrupt
    from /Users/sean/.rbenv/versions/2.2.5/lib/ruby/gems/2.2.0/gems/docker-sync-0.5.2/lib/docker-sync/sync_strategy/native_osx.rb:84:in `start_container'
    from /Users/sean/.rbenv/versions/2.2.5/lib/ruby/gems/2.2.0/gems/docker-sync-0.5.2/lib/docker-sync/sync_process.rb:105:in `start_container'
    from /Users/sean/.rbenv/versions/2.2.5/lib/ruby/gems/2.2.0/gems/docker-sync-0.5.2/lib/docker-sync/sync_manager.rb:99:in `block in start_container'
    from /Users/sean/.rbenv/versions/2.2.5/lib/ruby/gems/2.2.0/gems/docker-sync-0.5.2/lib/docker-sync/sync_manager.rb:98:in `each'
    from /Users/sean/.rbenv/versions/2.2.5/lib/ruby/gems/2.2.0/gems/docker-sync-0.5.2/lib/docker-sync/sync_manager.rb:98:in `start_container'
    from /Users/sean/.rbenv/versions/2.2.5/lib/ruby/gems/2.2.0/gems/docker-sync-0.5.2/tasks/sync/sync.thor:163:in `daemonize'
    from /Users/sean/.rbenv/versions/2.2.5/lib/ruby/gems/2.2.0/gems/docker-sync-0.5.2/tasks/sync/sync.thor:47:in `start'
    from /Users/sean/.rbenv/versions/2.2.5/lib/ruby/gems/2.2.0/gems/thor-0.19.4/lib/thor/command.rb:27:in `run'
    from /Users/sean/.rbenv/versions/2.2.5/lib/ruby/gems/2.2.0/gems/thor-0.19.4/lib/thor/invocation.rb:126:in `invoke_command'
    from /Users/sean/.rbenv/versions/2.2.5/lib/ruby/gems/2.2.0/gems/thor-0.19.4/lib/thor.rb:369:in `dispatch'
    from /Users/sean/.rbenv/versions/2.2.5/lib/ruby/gems/2.2.0/gems/thor-0.19.4/lib/thor/base.rb:444:in `start'
    from /Users/sean/.rbenv/versions/2.2.5/lib/ruby/gems/2.2.0/gems/docker-sync-0.5.2/bin/docker-sync:14:in `<top (required)>'
    from /Users/sean/.rbenv/versions/2.2.5/bin/docker-sync:23:in `load'
    from /Users/sean/.rbenv/versions/2.2.5/bin/docker-sync:23:in `<main>'

So it looks like it's working very slowly. The default boilerplate seemed to work - I'll try a reboot/rinse/repeat.

seanhandley commented 6 years ago

Same issue after a reboot 😞

EugenMayer commented 6 years ago

So basically, after a disk-full issue, docker-sync is malfunction even though you:

reset your d4m to factory settings
you restarted your mac
you cleaned up .docker-sync

Right? Please consider removing d4m completely and reinstall it - if that still does not help, maybe try the edge version of it, since there have been some HS fixes.

Other then that, i am totally blind picking - i have no clue how to systematically help you in this case ;/

EugenMayer commented 6 years ago

Could you try this performance check for you https://github.com/EugenMayer/docker-sync/issues/497 ?

Could you try to run https://gist.github.com/EugenMayer/07d24d4b1b88da3e48d90052192649a2 and see, if that improved your performance?

What did you change since the "disk ran full"?

seanhandley commented 6 years ago

Actually (sorry!) I think it's alright.

I think it's taking a long time to do the 1st sync and then subsequent syncs appear to be working fine.

The problem is probably that my expectation was that it would be a lot faster than 4 minutes for that 1st sync. Does that seem like an unusually long first sync to you?

seanhandley commented 6 years ago

Thanks for your help - closing this 👍

EugenMayer commented 6 years ago

Thats very long, but it depends on what you are syncing i guess

ericpoirier commented 6 years ago

Not sure if I ran out of space in the first place but my mac kept restarting on me and docker-sync stopped syncing files.

This is what I did to fix it:

Reset d4m to factory defaults
Restart my mac

Thanks for your help!

EugenMayer / docker-sync