Failures on RHEL 7

[redhatromero]

Stuck with rabbitmq failing....

Before I get into the details, a couple of things I needed to do on a fresh Centos 7 install:

1. I had only done a base install, but "private-chef-ctl reconfigure" failed until I installed httpd, then ran a system update and rebooted.
2. I was then getting a hanging behavior here:

* link[/opt/opscode/init/couchdb] action create
    - create symlink at /opt/opscode/init/couchdb to /opt/opscode/embedded/bin/sv
    - configure service runit_service[couchdb]
* link[/opt/opscode/service/couchdb] action create
    - create symlink at /opt/opscode/service/couchdb to /opt/opscode/sv/couchdb

1. After some digging, it became apparent that I was missing the "erlang" package, which was not available at the CentOS 7 repository. I added the EPEL 7 repo file, then installed "erlang".... that seemed to get me past this portion.
2. Now I'm stuck here:

Recipe: private-chef::rabbitmq
  * runit_service[rabbitmq] action enableRecipe: <Dynamically Defined Resource>
  * directory[/opt/opscode/sv/rabbitmq] action create (up to date)
  * template[/opt/opscode/sv/rabbitmq/run] action create (up to date)
  * directory[/opt/opscode/sv/rabbitmq/log] action create (up to date)
  * directory[/opt/opscode/sv/rabbitmq/log/main] action create (up to date)
  * template[/opt/opscode/sv/rabbitmq/log/run] action create (up to date)
  * template[/opt/opscode/sv/rabbitmq/log/config] action create (up to date)
  * link[/opt/opscode/init/rabbitmq] action create (up to date)

    - configure service runit_service[rabbitmq]

Recipe: private-chef::rabbitmq
  * execute[/opt/opscode/bin/private-chef-ctl start rabbitmq] action run
================================================================================
Error executing action `run` on resource 'execute[/opt/opscode/bin/private-chef-ctl start rabbitmq]'
================================================================================

Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Expected process to exit with [0], but received '1'
---- Begin output of /opt/opscode/bin/private-chef-ctl start rabbitmq ----
STDOUT: warning: rabbitmq: unable to open supervise/ok: file does not exist
STDERR:
---- End output of /opt/opscode/bin/private-chef-ctl start rabbitmq ----
Ran /opt/opscode/bin/private-chef-ctl start rabbitmq returned 1

Resource Declaration:
---------------------
# In /opt/opscode/embedded/cookbooks/private-chef/recipes/rabbitmq.rb

 79:   execute "#{opc_ctl} start rabbitmq" do
 80:     retries 20
 81:   end
 82:

Compiled Resource:
------------------
# Declared in /opt/opscode/embedded/cookbooks/private-chef/recipes/rabbitmq.rb:79:in `from_file'

execute("/opt/opscode/bin/private-chef-ctl start rabbitmq") do
  action "run"
  retries 0
  retry_delay 2
  command "/opt/opscode/bin/private-chef-ctl start rabbitmq"
  backup 5
  returns 0
  cookbook_name :"private-chef"
  recipe_name "rabbitmq"
end

Running handlers:
[2014-11-21T12:15:45-05:00] ERROR: Running exception handlers
Running handlers complete

[2014-11-21T12:15:45-05:00] ERROR: Exception handlers complete
[2014-11-21T12:15:45-05:00] FATAL: Stacktrace dumped to /opt/opscode/embedded/cookbooks/cache/chef-stacktrace.out
Chef Client failed. 2 resources updated in 51.528969427 seconds
[2014-11-21T12:15:45-05:00] ERROR: execute[/opt/opscode/bin/private-chef-ctl start rabbitmq] (private-chef::rabbitmq line 79) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of /opt/opscode/bin/private-chef-ctl start rabbitmq ----
STDOUT: warning: rabbitmq: unable to open supervise/ok: file does not exist
STDERR:
---- End output of /opt/opscode/bin/private-chef-ctl start rabbitmq ----
Ran /opt/opscode/bin/private-chef-ctl start rabbitmq returned 1
[2014-11-21T12:15:45-05:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

1. I installed "librabbitmq", "librabbitmq-devel", and "rabbitmq-server", rebooted, then reinstalled "private-chef (11.1.6). Ditched this version and downloaded 12, since I am getting to the same place.
2. Version 12 is the same.... hangs here:

* directory[/opt/opscode/sv/rabbitmq/log/main] action create
    - create new directory /opt/opscode/sv/rabbitmq/log/main
    - change mode from '' to '0755'

  * template[/opt/opscode/sv/rabbitmq/log/run] action create
    - create new file /opt/opscode/sv/rabbitmq/log/run
    - update content in file /opt/opscode/sv/rabbitmq/log/run from none to 0e09b8
        --- /opt/opscode/sv/rabbitmq/log/run    2014-11-21 13:07:03.303442999 -0500
        +++ /tmp/chef-rendered-template20141121-1933-1l5gq9c    2014-11-21 13:07:03.305442999 -0500
        @@ -1 +1,4 @@
        +#!/bin/sh
        +exec chpst -U opscode -u opscode \
        +  svlogd -tt /var/log/opscode/rabbitmq
    - change mode from '' to '0755'

  * template[/opt/opscode/sv/rabbitmq/log/config] action create
    - create new file /opt/opscode/sv/rabbitmq/log/config
    - update content in file /opt/opscode/sv/rabbitmq/log/config from none to e3b0c4
        (no diff)
    - change mode from '' to '0644'

  * link[/opt/opscode/init/rabbitmq] action create
    - create symlink at /opt/opscode/init/rabbitmq to /opt/opscode/embedded/bin/sv

    - configure service runit_service[rabbitmq]  * link[/opt/opscode/service/rabbitmq] action create
    - create symlink at /opt/opscode/service/rabbitmq to /opt/opscode/sv/rabbitmq

Just hangs right here....

[redhatromero]

Sorry about the formatting...

[redhatromero]

So... I tried forcing "rabbitmq-server" into action:

[root@chefent ent-chef]# service rabbitmq-server status
Redirecting to /bin/systemctl status  rabbitmq-server.service
rabbitmq-server.service - RabbitMQ broker
   Loaded: loaded (/usr/lib/systemd/system/rabbitmq-server.service; disabled)
   Active: active (running) since Fri 2014-11-21 15:04:12 EST; 31s ago
  Process: 2819 ExecStartPost=/usr/lib/rabbitmq/bin/rabbitmqctl wait /var/run/rabbitmq/pid (code=exited, status=0/SUCCESS)
  Process: 2792 ExecStartPre=/bin/sh -c /usr/lib/rabbitmq/bin/rabbitmqctl status > /dev/null 2>&1 (code=exited, status=2)
 Main PID: 2818 (beam)
   CGroup: /system.slice/rabbitmq-server.service
           ??2818 /usr/lib64/erlang/erts-5.10.4/bin/beam -W w -K true -A30 -P 1048576 -- -root /usr/lib64/erlang -progname erl -- -home /var/lib/ra...
           ??2913 inet_gethost 4
           ??2914 inet_gethost 4

Nov 21 15:04:10 chefent.jordanje.mac rabbitmqctl[2819]: pid is 2818 ...
Nov 21 15:04:12 chefent.jordanje.mac rabbitmq-server[2818]: RabbitMQ 3.1.5. Copyright (C) 2007-2013 GoPivotal, Inc.
Nov 21 15:04:12 chefent.jordanje.mac rabbitmq-server[2818]: ##  ##      Licensed under the MPL.  See http://www.rabbitmq.com/
Nov 21 15:04:12 chefent.jordanje.mac rabbitmq-server[2818]: ##  ##
Nov 21 15:04:12 chefent.jordanje.mac rabbitmq-server[2818]: ##########  Logs: /var/log/rabbitmq/rabbit@chefent.log
Nov 21 15:04:12 chefent.jordanje.mac rabbitmq-server[2818]: ######  ##        /var/log/rabbitmq/rabbit@chefent-sasl.log
Nov 21 15:04:12 chefent.jordanje.mac rabbitmq-server[2818]: ##########
Nov 21 15:04:12 chefent.jordanje.mac rabbitmq-server[2818]: Starting broker... completed with 0 plugins.
Nov 21 15:04:12 chefent.jordanje.mac rabbitmqctl[2819]: ...done.
Nov 21 15:04:12 chefent.jordanje.mac systemd[1]: Started RabbitMQ broker.

BTW: I reached this page from here: https://tickets.opscode.com/browse/CHEF-3838 His file... /opt/chef-server/embedded/cookbooks/runit/recipes/default.rb in Chef12, is... /opt/opscode/embedded/cookbooks/runit/recipes/default.rb. In this file I did:

case node['platform_family']
when 'rhel', 'redhat', 'centos'

and

else
    include_recipe 'build-essential'
    # `rpmdevtools` is in EPEL repo in EL <= 5
    include_recipe 'yum-epel' if node['platform_version'].to_i == 7

    packages = %w{rpm-build rpmdevtools tar gzip}
    packages.each do |p|
      package p
    end

    if node['platform_version'].to_i >= 7
      package 'glibc-static'

Still fails at trying to start "rabbitmq"

[redhatromero]

Ran:

yum install glibc-static rpmdevtools

No change...

[redhatromero]

looks to me like it IS running:

rabbitmq 2818 1 0 15:04 ? 00:00:04 /usr/lib64/erlang/erts-5.10.4/bin/beam -W w -K true -A30 -P 1048576 -- -root /usr/lib64/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.1.5/sbin/../ebin -noshell -noinput -s rabbit boot -sname rabbit@chefent -boot start_sasl -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/var/log/rabbitmq/rabbit@chefent.log"} -rabbit sasl_error_logger {file,"/var/log/rabbitmq/rabbit@chefent-sasl.log"} -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/lib/rabbitmq_server-3.1.5/sbin/../plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit@chefent-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@chefent" root 5396 1860 0 15:42 pts/0 00:00:00 /bin/bash /usr/bin/chef-server-ctl reconfigure root 5399 5397 0 15:42 pts/0 00:00:04 /opt/opscode/embedded/bin/ruby /opt/opscode/embedded/bin/chef-solo -c /opt/opscode/embedded/cookbooks/solo.rb -j /opt/opscode/embedded/cookbooks/dna.json

[redhatromero]

I give for now... I will back the system down to 6.5 or 6.6.

[kbaltrinic]

I am having the exact same set of issues trying to install Jenkins on Centos 7 using the Ops code Jenkins cookbook with the 'war' option (which uses runit to run the war). Similarly The ops-code chef server cookbook, though it appears to use its own runit cookbook, has a detailed post on the same problem Chef Server runit install fails on CentOS 7 #1955.

The short of it being that from 6.x to 7 Centos switched from using upstart to systemd as its service manager. So it seems that the solution is to update the runit cookbook configure runit to use systemd on Cento >= 7.

[redhatromero]

Dang it...!!! I would have never thought to check for that, I guess because it appeared to be alive (??? have to look back at my notes).

Thanks though, for the feedback. It will be a little while before I try it again being that I could afford more time since I was being pressed for a working solution.

[jtimberman]

While we do need to get systemd support added to this cookbook so runit can work properly on CentOS/RHEL 7, this is being addressed in the Chef Server's cookbooks under this PR.

[slyness]

Since the PR referenced (https://github.com/opscode-cookbooks/enterprise-chef-common/pull/25) above is merged has this issue been resolved?

[kbaltrinic]

@slyness No I do not think that this issue is resolved. The enterprise-chef-common PR is specific to enterprise-chef which does not use the runit cookbook (but does much the same thing). The runit cookbook itself appears to still broken vis-a-vis Centos 7. I am still unable to install Jenkins on Centos 7 via the jenkins cookbook due to this issue, and I imagine the original reporter is still unable to install RabbitMQ for the same reason.

[marcusl33beach]

Just tried install chef on CentOS7 and ran into the same issue. Has there been any progress on this ?

[ephracis]

I have this issue on Ubuntu 14.04.

[virtuman]

has anyone been able to get passed this point? seem to have this issue in clean centos 7.0 install in Docker, but looks like it might be working correctly if install directly on Host. Tried 12.4.1 and 12.5.0. Same result. I think i tried everything i could think of at this point.

[mmukherjee]

I am experiencing a similar issue. The chef 'artifactory' cookbook depends upon this 'runit' cookbook and I get this error -

• yum_package[runit] action install

  • install version 2.1.2-3.el7.centos of package runit * service[runsvdir-start] action start * service[runsvdir-start]: No custom command for start specified and unable to locate the init.d script!

    Error executing action start on resource 'service[runsvdir-start]'

    Chef::Exceptions::Service

    service[runsvdir-start]: No custom command for start specified and unable to locate the init.d script! Resource Declaration:

    In /tmp/kitchen/cache/cookbooks/runit/recipes/default.rb

    43: service 'runsvdir-start' do 44: action [:start, :enable] 45: only_if { node['platform_version'].to_i == 7 } 46: end 47:

Is there a workaround/fix?

[sdemura]

@mmukherjee -- I just went through a similar exercise. Are you running Chef 11? If so, Chef 11 defaults assume CentOS is still using sysvinit. I got around the issue by changing the service provider:

Chef::Platform.set :platform => :redhat, :resource => :service, :provider => Chef::Provider::Service::Systemd```