search

sous-chefs / docker

Development repository for the docker cookbook
https://supermarket.chef.io/cookbooks/docker
Apache License 2.0
1.35k stars 794 forks source link

Docker Service Flapping #717

Closed michaelpjohnson closed 7 years ago

michaelpjohnson commented 8 years ago

Cookbook version

2.7.1

Chef-client version

[2016-06-09T16:10:09+00:00] INFO: *** Chef 12.0.3 ***

Platform Details

Ubuntu 14.04 running in AWS

Scenario:

I am passing a simple role to AWS Autoscale to launch an instance. The role only has the docker_service LWRP.

Steps to Reproduce:

Install an autoscaling group(with chef) on ubuntu 14.04 and make sure it has this block:

docker_service 'default' do
  host ["tcp://#{node['ipaddress']}:2376", 'unix:///var/run/docker.sock']
  cluster_store "consul://xxxxxxx:8500"
  cluster_advertise 'eth0:2376'
  daemon true
  dns ['8.8.8.8']
  dns_search ['xxxxxxxxx']
  ip node['ipaddress']
  retries 5
  retry_delay 10
  ip_forward true
  iptables true
  log_driver 'syslog'
  log_opts 'syslog-address=udp://xxxxxxxxxxx'
  action [:create, :start]
end

Expected Result:

Docker to come up and be stable. This works in test kitchen. It also works if the node comes up with no role, and then I add one later. A new chef-client run will install it, but I need it to work with our autoscaling.

Actual Result:

Docker flaps continuously:

[2016-06-09T16:12:11+00:00] INFO: Processing template[/usr/lib/docker/docker-wait-ready] action create (/var/chef/cache/cookbooks/docker/libraries/docker_service_base.rb line 97)
[2016-06-09T16:12:11+00:00] INFO: template[/usr/lib/docker/docker-wait-ready] created file /usr/lib/docker/docker-wait-ready
[2016-06-09T16:12:11+00:00] INFO: template[/usr/lib/docker/docker-wait-ready] updated file contents /usr/lib/docker/docker-wait-ready
[2016-06-09T16:12:11+00:00] INFO: template[/usr/lib/docker/docker-wait-ready] owner changed to 0
[2016-06-09T16:12:11+00:00] INFO: template[/usr/lib/docker/docker-wait-ready] group changed to 0
[2016-06-09T16:12:11+00:00] INFO: template[/usr/lib/docker/docker-wait-ready] mode changed to 755
[2016-06-09T16:12:11+00:00] INFO: Processing template[/etc/init/docker.conf] action create (/var/chef/cache/cookbooks/docker/libraries/docker_service_manager_upstart.rb line 11)
[2016-06-09T16:12:11+00:00] INFO: template[/etc/init/docker.conf] backed up to /var/chef/backup/etc/init/docker.conf.chef-20160609161211.208210
[2016-06-09T16:12:11+00:00] INFO: template[/etc/init/docker.conf] updated file contents /etc/init/docker.conf
[2016-06-09T16:12:11+00:00] INFO: Processing template[/etc/default/docker] action create (/var/chef/cache/cookbooks/docker/libraries/docker_service_manager_upstart.rb line 25)
[2016-06-09T16:12:11+00:00] INFO: template[/etc/default/docker] backed up to /var/chef/backup/etc/default/docker.chef-20160609161211.212508
[2016-06-09T16:12:11+00:00] INFO: template[/etc/default/docker] updated file contents /etc/default/docker
[2016-06-09T16:12:11+00:00] INFO: template[/etc/default/docker] sending restart action to docker_service_manager_upstart[default] (immediate)
[2016-06-09T16:12:11+00:00] INFO: Processing docker_service_manager_upstart[default] action restart (/var/chef/cache/cookbooks/docker/libraries/docker_service.rb line 74)
[2016-06-09T16:12:11+00:00] INFO: Processing service[docker] action stop (/var/chef/cache/cookbooks/docker/libraries/docker_service_manager_upstart.rb line 44)
[2016-06-09T16:12:11+00:00] INFO: Processing directory[/usr/lib/docker] action create (/var/chef/cache/cookbooks/docker/libraries/docker_service_base.rb line 90)
[2016-06-09T16:12:11+00:00] INFO: Processing template[/usr/lib/docker/docker-wait-ready] action create (/var/chef/cache/cookbooks/docker/libraries/docker_service_base.rb line 97)
[2016-06-09T16:12:11+00:00] INFO: Processing template[/etc/init/docker.conf] action create (/var/chef/cache/cookbooks/docker/libraries/docker_service_manager_upstart.rb line 11)
[2016-06-09T16:12:11+00:00] INFO: Processing template[/etc/default/docker] action create (/var/chef/cache/cookbooks/docker/libraries/docker_service_manager_upstart.rb line 25)
[2016-06-09T16:12:11+00:00] INFO: Processing service[docker] action start (/var/chef/cache/cookbooks/docker/libraries/docker_service_manager_upstart.rb line 36)
[2016-06-09T16:12:12+00:00] INFO: service[docker] started
[2016-06-09T16:12:12+00:00] INFO: template[/etc/default/docker] sending restart action to docker_service_manager_upstart[default] (immediate)
[2016-06-09T16:12:12+00:00] INFO: Processing docker_service_manager_upstart[default] action restart (/var/chef/cache/cookbooks/docker/libraries/docker_service.rb line 74)
[2016-06-09T16:12:12+00:00] INFO: Processing service[docker] action stop (/var/chef/cache/cookbooks/docker/libraries/docker_service_manager_upstart.rb line 44)
[2016-06-09T16:12:13+00:00] INFO: service[docker] stopped
[2016-06-09T16:12:13+00:00] INFO: Processing directory[/usr/lib/docker] action create (/var/chef/cache/cookbooks/docker/libraries/docker_service_base.rb line 90)
[2016-06-09T16:12:13+00:00] INFO: Processing template[/usr/lib/docker/docker-wait-ready] action create (/var/chef/cache/cookbooks/docker/libraries/docker_service_base.rb line 97)
[2016-06-09T16:12:13+00:00] INFO: Processing template[/etc/init/docker.conf] action create (/var/chef/cache/cookbooks/docker/libraries/docker_service_manager_upstart.rb line 11)
[2016-06-09T16:12:13+00:00] INFO: Processing template[/etc/default/docker] action create (/var/chef/cache/cookbooks/docker/libraries/docker_service_manager_upstart.rb line 25)
[2016-06-09T16:12:13+00:00] INFO: Processing service[docker] action start (/var/chef/cache/cookbooks/docker/libraries/docker_service_manager_upstart.rb line 36)
[2016-06-09T16:12:15+00:00] INFO: service[docker] started
[2016-06-09T16:12:15+00:00] INFO: template[/etc/default/docker] sending restart action to docker_service_manager_upstart[default] (immediate)
[2016-06-09T16:12:15+00:00] INFO: Processing docker_service_manager_upstart[default] action restart (/var/chef/cache/cookbooks/docker/libraries/docker_service.rb line 74)
[2016-06-09T16:12:15+00:00] INFO: Processing service[docker] action stop (/var/chef/cache/cookbooks/docker/libraries/docker_service_manager_upstart.rb line 44)
[2016-06-09T16:12:16+00:00] INFO: service[docker] stopped
[2016-06-09T16:12:16+00:00] INFO: Processing directory[/usr/lib/docker] action create (/var/chef/cache/cookbooks/docker/libraries/docker_service_base.rb line 90)
[2016-06-09T16:12:16+00:00] INFO: Processing template[/usr/lib/docker/docker-wait-ready] action create (/var/chef/cache/cookbooks/docker/libraries/docker_service_base.rb line 97)
[2016-06-09T16:12:16+00:00] INFO: Processing template[/etc/init/docker.conf] action create (/var/chef/cache/cookbooks/docker/libraries/docker_service_manager_upstart.rb line 11)
[2016-06-09T16:12:16+00:00] INFO: Processing template[/etc/default/docker] action create (/var/chef/cache/cookbooks/docker/libraries/docker_service_manager_upstart.rb line 25)
[2016-06-09T16:12:16+00:00] INFO: Processing service[docker] action start (/var/chef/cache/cookbooks/docker/libraries/docker_service_manager_upstart.rb line 36)
[2016-06-09T16:12:17+00:00] INFO: service[docker] started
[2016-06-09T16:12:17+00:00] INFO: template[/etc/default/docker] sending restart action to docker_service_manager_upstart[default] (immediate)
[2016-06-09T16:12:17+00:00] INFO: Processing docker_service_manager_upstart[default] action restart (/var/chef/cache/cookbooks/docker/libraries/docker_service.rb line 74)
[2016-06-09T16:12:17+00:00] INFO: Processing service[docker] action stop (/var/chef/cache/cookbooks/docker/libraries/docker_service_manager_upstart.rb line 44)
[2016-06-09T16:12:18+00:00] INFO: service[docker] stopped
michaelpjohnson commented 8 years ago

While it flaps, the ports change from 2376 to 7946. I suspect this has something to do with the swarm setup and consul?

root@ip-X-X-X-X:/var/chef/cache/cookbooks# netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 X-X-X-X:2376            0.0.0.0:*               LISTEN      11555/docker
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1240/sshd
tcp6       0      0 :::22                   :::*                    LISTEN      1240/sshd
root@ip-X-X-X-X:/var/chef/cache/cookbooks# netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 X-X-X-X:2376            0.0.0.0:*               LISTEN      11555/docker
tcp        0      0 X-X-X-X:7946            0.0.0.0:*               LISTEN      11555/docker
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1240/sshd
tcp6       0      0 :::22                   :::*                    LISTEN      1240/sshd
root@ip-10-137-224-119:/var/chef/cache/cookbooks# netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 X-X-X-X:7946            0.0.0.0:*               LISTEN      11555/docker
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1240/sshd
tcp6       0      0 :::22                   :::*                    LISTEN      1240/sshd
someara commented 7 years ago

Does this still happen?

someara commented 7 years ago

I'm going to close this for now, since it seems to be specific to your config, and I'm rather helpless to recreate it. Let me know if you figure this out. Im curious about what's going on