search

rcbops / chef-cookbooks

RCB OPS - Chef Cookbooks
Other
118 stars 102 forks source link

keepalived is restarted n times where n is the number of services that are behind haproxy #736

Open breu opened 10 years ago

breu commented 10 years ago

could be due to the fact we don't use use_inline_resources in the keepalived vrrp provider

claco commented 10 years ago

For future readers: we use this pattern in just two places:

https://github.com/rcbops-cookbooks/cinder/commit/358ce9cc61790e7c769b4d21bc8578346b706fc3 https://github.com/rcbops-cookbooks/nova/commit/c0c6a0ffe6a90ee16bab6ce931d52c2f2aeb219e

We think the origin is here:

https://github.com/rcbops-cookbooks/openstack-ha/blob/master/recipes/default.rb#L81

claco commented 10 years ago

Here's my guess:

r = template "vrrp_#{new_resource.name.upcase}" do

openstack-ha calls that provider in a loop, and if the v*ps are the same for each service, we're creating a clone of the template resource for each call. If the clones return true for the change on a run, all of them are returning true, which then causes the vrrp provider to send multiple restarts to the service. (only one restart happens though I believe).

Update: Looks like the restart happens for each notification since they're immediate.

breu commented 10 years ago

moving to 4.3.3.

breu commented 10 years ago

echo'ing @claco last comment...

[2014-01-28T21:16:46+00:00] INFO: keepalived_vrrp[vi_192_168_0_251] sending restart action to service[keepalived] (immediate)
Recipe: keepalived::default
  * service[keepalived] action restart[2014-01-28T21:16:46+00:00] INFO: Processing service[keepalived] action restart (keepalived::default line 72)
[2014-01-28T21:16:48+00:00] INFO: service[keepalived] restarted

    - restart service service[keepalived]

[2014-01-28T21:16:48+00:00] INFO: keepalived_vrrp[vi_192_168_0_251] sending restart action to service[keepalived] (immediate)
  * service[keepalived] action restart[2014-01-28T21:16:48+00:00] INFO: Processing service[keepalived] action restart (keepalived::default line 72)
[2014-01-28T21:16:49+00:00] INFO: service[keepalived] restarted

    - restart service service[keepalived]

[2014-01-28T21:16:49+00:00] INFO: keepalived_vrrp[vi_192_168_0_251] sending restart action to service[keepalived] (immediate)
  * service[keepalived] action restart[2014-01-28T21:16:49+00:00] INFO: Processing service[keepalived] action restart (keepalived::default line 72)
[2014-01-28T21:16:50+00:00] INFO: service[keepalived] restarted

    - restart service service[keepalived]

[2014-01-28T21:16:50+00:00] INFO: keepalived_vrrp[vi_192_168_0_251] sending restart action to service[keepalived] (immediate)
  * service[keepalived] action restart[2014-01-28T21:16:50+00:00] INFO: Processing service[keepalived] action restart (keepalived::default line 72)
[2014-01-28T21:16:51+00:00] INFO: service[keepalived] restarted

    - restart service service[keepalived]

So it does indeed restart it very often