recipe[chef-client::service] is broken when service is restarted from child process

[jperville]

In the same spirit as #284, I noticed that any attribute change that results in restarting the chef-client service will fail when the service restart is initiated from a child process of that chef-client service. Bug spotted on Ubuntu 14.04 with upstart as init_type, but it probably happens on other platforms.

How to reproduce:

• setup periodic chef-runs using recipe[chef-client::recipe']
• when chef-client finishes, verify that the chef-client service is running
• change the value of node['chef_client']['interval'] (or any attribute that alters the content of /etc/default/chef-client
• wait for periodic chef-run to start
• when chef-client finishes, notice that the chef-client service is no longer running

The problem is that when the /etc/default/chef-client file is updated, chef requests a restart of service[chef-client], which happens to be the parent process of the periodic chef-client process.

What happens next is that executing "service chef-client stop" will abort the child process (the periodic chef-run), which won't have the chance to execute "service chef-client start".

Then the chef-client service will be broken (not running) until someone starts it again manually.

[djdefi]

+1

[runningman84]

does this look like the same issue?

015-09-09T10:58:36+00:00] INFO: link[/etc/service/carbon-cache] created
[2015-09-09T10:58:36+00:00] INFO: Processing graphite_web_config[/opt/graphite/webapp/graphite/local_settings.py] action create (asy-graphite::graphite line 77)
[2015-09-09T10:58:36+00:00] INFO: Processing file[/opt/graphite/webapp/graphite/local_settings.py] action create (/var/chef/cache/cookbooks/graphite/providers/web_config.rb line 38)
[2015-09-09T10:58:36+00:00] INFO: Running queued delayed notifications before re-raising exception
[2015-09-09T10:58:36+00:00] INFO: Running queued delayed notifications before re-raising exception
[2015-09-09T10:58:36+00:00] INFO: template[/etc/init.d/chef-client] sending restart action to service[chef-client] (delayed)
[2015-09-09T10:58:36+00:00] INFO: Processing service[chef-client] action restart (chef-client::init_service line 32)
[2015-09-09T10:58:36+00:00] FATAL: SIGTERM received, stopping
[2015-09-09T10:58:38+00:00] INFO: Daemonizing..
[2015-09-09T10:58:38+00:00] INFO: Forked, in 15337. Privileges: 0 0
[2015-09-09T10:58:38+00:00] INFO: service[chef-client] restarted
[2015-09-09T10:58:38+00:00] INFO: apt_package[sensu] sending create action to ruby_block[sensu_service_trigger] (delayed)
[2015-09-09T10:58:38+00:00] INFO: Processing ruby_block[sensu_service_trigger] action create (sensu::default line 20)
[2015-09-09T10:58:38+00:00] INFO: ruby_block[sensu_service_trigger] called
[2015-09-09T10:58:38+00:00] INFO: cookbook_file[/etc/ntp.leapseconds] sending restart action to service[ntp] (delayed)
[2015-09-09T10:58:38+00:00] INFO: Processing service[ntp] action restart (ntp::default line 112)
[2015-09-09T10:58:40+00:00] INFO: service[ntp] restarted
[2015-09-09T10:58:40+00:00] INFO: cookbook_file[/etc/apparmor.d/usr.sbin.ntpd] sending restart action to service[apparmor] (delayed)
[2015-09-09T10:58:40+00:00] INFO: Processing service[apparmor] action restart (ntp::apparmor line 20)
[2015-09-09T10:58:41+00:00] INFO: service[apparmor] restarted
[2015-09-09T10:58:41+00:00] INFO: ruby_block[sensu_service_trigger] sending restart action to service[sensu-client] (delayed)
[2015-09-09T10:58:41+00:00] INFO: Processing service[sensu-client] action restart (/var/chef/cache/cookbooks/sensu/providers/service.rb line 47)
[2015-09-09T10:58:45+00:00] INFO: service[sensu-client] restarted
[2015-09-09T10:58:45+00:00] ERROR: Running exception handlers
[2015-09-09T10:58:45+00:00] ERROR: Exception handlers complete
[2015-09-09T10:58:45+00:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
[2015-09-09T10:58:45+00:00] ERROR: graphite_web_config[/opt/graphite/webapp/graphite/local_settings.py] (asy-graphite::graphite line 77) had an error: Chef::Exceptions::EnclosingDirectoryDoesNotExist: file[/opt/graphite/webapp/graphite/local_settings.py] (/var/chef/cache/cookbooks/graphite/providers/web_config.rb line 38) had an error: Chef::Exceptions::EnclosingDirectoryDoesNotExist: Parent directory /opt/graphite/webapp/graphite does not exist.
[2015-09-09T10:58:45+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

My cookbooks look like this:

• base (includes chef_client recipe, adds ssh users, intalls ntp, ...)
• application (graphite)

My role looks like this:

• base, application

The intial chef run of the role always fails. If I run the base recipe only, the chef log looks good. After that I can sucessfuly run the application cookbook.

[dheerajd-msys]

@jperville Could you please check with latest cookbook. chef-client service is properly running after changing attribute interval from 1800 to 1500 .

Platform & Chef client version

Ubuntu 14.04 / Chef Client v13.12.14

In order to verify I've followed the steps below :

1. Run chef-client with recipe chef-client::upstart Service is running

   
   azure@dh-ubu14:~/chef-repo/.chef$ ps aux | grep chef
   root       7321  2.0  0.6 147812 49460 ?        Ssl  10:09   0:01 /opt/chef/embedded/bin/ruby --disable-gems /usr/bin/chef-client -i **1800** -s 300
   azure      7386  0.0  0.0  10480  2176 pts/0    S+   10:10   0:00 grep --color=auto chef

2. Updated attribute 

azure@dh-ubu14:~/chef-repo/.chef$ cat attributes.json { "chef_client":{"interval":"1500"} }

3. Run chef-client 
Service is running with  interval updated.

azure@dh-ubu14:~/chef-repo/.chef$ ps aux | grep chef root 8759 2.0 0.5 146764 48404 ? Ssl 10:31 0:01 /opt/chef/embedded/bin/ruby --disable-gems /usr/bin/chef-client -i 1500 -s 300 azure 8816 0.0 0.0 10480 2184 pts/0 S+ 10:32 0:00 grep --color=auto chef azure@dh-ubu14:~/chef-repo/.chef$

[dheerajd-msys]

Closing for now, please let us know if there is anything.