search

sous-chefs / consul

Development repository for the consul cookbook
https://supermarket.chef.io/cookbooks/consul
Apache License 2.0
192 stars 244 forks source link

Default recipe starts consul with `server: true` #423

Closed vsudilov closed 7 years ago

vsudilov commented 7 years ago

seemingly, the root cause is that consul version at least > 0.7.0 ships with server: true as a default in consul.json;

this can be seen by changing node['consul']['version'] = 0.7.0 in the latest cookbook version, and noticing no server: true line in said file as compared to the cookbook's default consul version.

The docs are out of date WRT to this change in behaviour.

legal90 commented 7 years ago

Hi, @vsudilov.

Yes, as I can see, we have a default value for server option set to true: https://github.com/johnbellone/consul-cookbook/blame/7b9b75c86ba4963dce6bebd94f289b258942062a/libraries/consul_config.rb#L92

FYI: It's not related to the Consul version, it is always true. That behavior is here since the beginning, from the first version of this cookbook. And personally, I agree that it is not expected. Especially, "README.md" promises that we bootstrap consul agent in a client mode:

Out of the box the default recipe installs and configures the Consul agent to run as a service in client mode.

I think, we should fix that by removing the default value on this option from the resource definition. @johnbellone What do you think about it?

ghost commented 7 years ago

Please consider backporting this fix to the 2.x release set. I'm glad this change made it into the repo, but I managed to get bitten by this anyways.

I was upgrading from consul==1.5.0 and I thought "better to do one major version at a time" so I ended up going with 2.3.0. This particular doozy of an issue (defaults to server=true in 2.3.0) was mentioned in the changelog... but only for 3.0.0 which I didn't read, because I wasn't upgrading to that version. Now I'm left with a few dozen nodes in a few datacenters that all think they are consul servers. I guess I'll have to demote them all carefully.

legal90 commented 7 years ago

Hi @yardinicwaller You can just set the attribute explicitly in your wrapper cookbook, role, environment, or Puppetfile (depending on the pattern you are using):

node.default['consul']['config']['server'] = false

Nodes converged with this attribute will start in a "client" mode. P.s. You might need to restart the daemon manually to get this configuration change applied.

ghost commented 7 years ago

Yes, thank you @legal90, that's exactly what I did.

However, that doesn't address my concern. The point is that this version 2.x of this cookbook has a radically different default behaviour than the 1.x version of the cookbook, and this change in behaviour is not documented as a breaking change. This means that other people who are upgrading from 1.x to 2.x are likely to fall into the same trap that I did.

Demoting consul servers is a tedious process, and potentially risky because if too many servers are demoted at once then there's a possibility of losing quorum and needing to re-bootstrap the cluster.

I believe this cookbook should do at least one of these two things:

  1. Backport this fix to the 2.x line and release 2.3.1 to Chef Supermarket. This will help protect anybody else who is upgrading from 1.x and decides to go with the 2.x line because it seems like it should be more stable and mature, or because they don't want to do a double-major version upgrade.
  2. Amend the CHANGELOG for the version that stopped providing server=false as default to indicate a breaking change.
lock[bot] commented 4 years ago

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.