Consul protocol version does not upgrade as documented

jakubgs commented 3 years ago

Overview of the Issue

According to the documentation the agents should upgrade to >2 protocol version automatically:

-protocol - The Consul protocol version to use. Consul agents speak protocol 2 by default, however agents will automatically use protocol > 2 when speaking to compatible agents. This should be set only when upgrading. You can view the protocol versions supported by Consul by running consul -v.

https://www.consul.io/docs/agent/options#_protocol

But I've upgraded my whole cluster to 1.9.3 today and forced use of version 3 using protocol parameter in my JSON config, expecting it to stick. But when I attempt to remove the protocol config value set to 3 from a single host it reverts to using verison 2 after restart:

admin@node-01.consul.example.org:~ % consul members
Node                            Address            Status  Type    Build  Protocol  DC       Segment
node-01.consul.example.org      11.22.33.44:8301   alive   server  1.9.3  2         ab-cde3  <default>

But the docs clearly state that "This should be set only when upgrading.", so why is it reverting?

Reproduction Steps

Switch all nodes - both servers and agents - to protocol version 3
Try to remove the protocol: 3 the config on one of the hosts
See how it switched back to using version 3 with consul members

Consul info for both Client and Server

Client info

``` agent: check_monitors = 2 check_ttls = 0 checks = 6 services = 7 build: prerelease = revision = f55da930 version = 1.9.3 consul: acl = disabled known_servers = 3 server = false runtime: arch = amd64 cpu_count = 1 goroutines = 53 max_procs = 1 os = linux version = go1.15.6 serf_lan: coordinate_resets = 0 encrypted = true event_queue = 0 event_time = 240 failed = 0 health_score = 0 intent_queue = 0 left = 0 member_time = 5611 members = 39 query_queue = 0 query_time = 10 ```

Server info

``` agent: check_monitors = 1 check_ttls = 0 checks = 4 services = 4 build: prerelease = revision = f55da930 version = 1.9.3 consul: acl = disabled bootstrap = false known_datacenters = 4 leader = false leader_addr = 35.184.154.56:8300 server = true raft: applied_index = 25028587 commit_index = 25028587 fsm_pending = 0 last_contact = 49.00668ms last_log_index = 25028587 last_log_term = 1308 last_snapshot_index = 25016263 last_snapshot_term = 1302 latest_configuration = [{Suffrage:Voter ID:2ec484a1-76c3-e368-cc83-2557bd6e32f7 Address:35.225.159.96:8300} {Suffrage:Voter ID:93086e4f-f4ca-aa40-bd58-602db5eca99e Address:104.197.126.101:8300} {Suffrage:Voter ID:a920fe0f-af66-4984-e67b-bb32630d9f8f Address:35.184.154.56:8300}] latest_configuration_index = 0 num_peers = 2 protocol_version = 3 protocol_version_max = 3 protocol_version_min = 0 snapshot_version_max = 1 snapshot_version_min = 0 state = Follower term = 1308 runtime: arch = amd64 cpu_count = 1 goroutines = 225 max_procs = 1 os = linux version = go1.15.6 serf_lan: coordinate_resets = 0 encrypted = true event_queue = 0 event_time = 240 failed = 0 health_score = 0 intent_queue = 0 left = 0 member_time = 5611 members = 39 query_queue = 0 query_time = 10 serf_wan: coordinate_resets = 0 encrypted = true event_queue = 0 event_time = 1 failed = 0 health_score = 0 intent_queue = 0 left = 0 member_time = 1516 members = 12 query_queue = 0 query_time = 1 ```

Operating system and Environment details

Ubuntu 20.04.1, amd64

jakubgs commented 3 years ago

Should I just keep protocol: 3 in my config despite docs stating that it "should be set only when upgrading"?

jkirschner-hashicorp commented 3 years ago

@jakubgs: thank you for bringing this to our attention. In short: your agents are already using protocol 3, but the consul members output doesn't make that clear.

After looking into it, here's our understanding.

The members command shows what is configured, which is 2 unless overridden to 3. But what protocol is being used can be different: agents will automatically use 3 when speaking to compatible agents. We could change the output of the members CLI command to make that clearer.
It seems that the protocol config option can be set all the time; it doesn't need to be removed after upgrading.

jakubgs commented 3 years ago

I think it would make more sense for the members command to show the actually used protocol, rather than the configured.

Also, I prefer to keep my configuration files as small as possible to make them easier to read. This is especially useful when dealing with JSON configuration files that have no comments. The more defaults I can use the better.

Thanks for explaining.

ihard commented 1 year ago

Are there any changes to be made? Version 1.14.1 still shows the protocol of the second version.

hashicorp / consul