search

sensu / sensu-chef

Sensu Chef cookbook.
https://supermarket.chef.io/cookbooks/sensu
Apache License 2.0
222 stars 283 forks source link

rabbitmq-server Fails to Start After Updating Erlang Packages #619

Open SeanSith opened 5 years ago

SeanSith commented 5 years ago

Current Behavior

Following an apt-get dist-upgrade, the packages 'erlang-mode' and 'esl-erlang' were upgraded above the highest version which supports the available version of 'rabbitmq-server', causing RabbitMQ to crash on startup. No logs were produced.

Possible Solution

Ensure appropriately versioned Erlang packages are installed along side their corresponding RabbitMQ server.

Steps to Reproduce (for bugs)

  1. Existing Ubuntu 16.04 server with the following packages installed:
    • rabbitmq-server=3.6.15-1
    • esl-erlang=1:20.3
    • erlang-mode=1:20.3-1
  2. apt-get dist-upgrade which took versions to the following:
    • esl-erlang=1:21.0.5
    • erlang-mode=1:21.0.5-1
  3. Attempt to start rabbitmq-server service. Failure.
  4. Downgrade 'esl-erlang' and 'erlang-mode' to versions listed in Step 1
  5. rabbitmq-server service starts appropriately.

Context

Merely attempting to patch the system with updated packages from Ubuntu.

Your Environment

Tagging @majormoses per thread in Sensu Community Slack #help channel from 29 August 2018 @ 21:27 UTC.

majormoses commented 5 years ago

This link may not work in the future (as we have limited retention on free slack) but here is the discussion: https://sensucommunity.slack.com/archives/C68LV5M9U/p1535578029000100

SeanSith commented 5 years ago

From a question in the Slack channel: The rabbitmq cookbook pulled in:

rabbitmq (5.6.1)
    dpkg_autostart (>= 0.0.0)
    erlang (>= 0.0.0)
    logrotate (>= 0.0.0)
    yum-epel (>= 0.0.0)
    yum-erlang_solutions (>= 0.0.0)
majormoses commented 5 years ago

Sorry what I needed was what version of the rabbitmq cookbook got pulled in, based on what I see (the version of rabbitmq itself) you are on at least 5.x but I'd like to confirm.

SeanSith commented 5 years ago

Sorry, for whatever reason, when I put that in, the code block wasn't formatted properly and the rabbitmq line didn't show up. Fixed it.

majormoses commented 5 years ago

Can you confirm if you modified this attribute? https://github.com/rabbitmq/chef-cookbook/blob/v5.x/attributes/default.rb#L6 from what I can tell from the defaults this should not have happened via an apt upgrade unless that was changed. You can set node['esl']['version'] similar to this: https://github.com/rabbitmq/chef-cookbook/blob/v5.x/.kitchen.yml#L29

SeanSith commented 5 years ago

I did not modify that attribute, no. Other than enabling TLS and setting a hostname and credentials using the attributes in this cookbook, I did not modify the RabbitMQ setup. I felt that the least amount that I could do to it would be best as I don't understand RabbitMQ well enough, and Redis wasn't going to be secure enough for my deployment pattern.

majormoses commented 5 years ago

I opened a PR on the upstream erlang cookbook which might help your scenerio: https://github.com/chef-cookbooks/erlang/pull/65

majormoses commented 5 years ago

@SeanSith you could try pulling in erlang cookbook 7.0.0 I think that should work, if you have a server to test with you can remove your hold and see if this prevents the issue from manifesting? The fix I made only fix debian based distros as you need some non standard repos to pin versions with yum.