After upgrading consul from 1.10.12 to 1.11.0 getting error="Unknown check ID

[ikrammohammad]

After upgrading consul from 1.10.12 to 1.11.0 getting error="Unknown check ID

Overview of the Issue

After upgrading consul from 1.10.12 to 1.11.0 getting below error message for all microservices

2023-03-06T11:30:52.149-0700 [ERROR] agent.http: Request error: method=PUT url="/v1/agent/check/pass/service:service-api?note=procmanager+heartbeat" from=127.0.0.1:63118 error="Unknown check ID \"service:service-api\". Ensure that the check ID is passed, not the check name."

---

Reproduction Steps

Steps to reproduce this issue, eg:

1. Create a cluster with 1 client node and 1 server node with a windows service name "service-registry" using consul version as 1.10.12
2. Open http://localhost:8500/ui/dc1/services and all services will come online if we restart "service-registry"
3. Upgrade "service-registry" with consul version as 1.11.0
4. As soon as "service-registry" is restarted all the services goes offline in consul UI and will see errors like below 2023-03-06T11:30:52.149-0700 [ERROR] agent.http: Request error: method=PUT url="/v1/agent/check/pass/service:service-api?note=procmanager+heartbeat" from=127.0.0.1:63118 error="Unknown check ID \"service:service-api\". Ensure that the check ID is passed, not the check name."

-->

Consul info for both Client and Server

Client info

``` Output from client 'consul info' command here on version 1.9.4 ``` C:\Program Files\LogRhythm\LogRhythm Common\LogRhythm Service Registry\dependencies\consul>consul info agent: check_monitors = 0 check_ttls = 24 checks = 24 services = 24 build: prerelease = revision = 10bb6cb3 version = 1.9.4 consul: acl = disabled bootstrap = true known_datacenters = 1 leader = true leader_addr = 10.4.2.99:8300 server = true raft: applied_index = 36058 commit_index = 36058 fsm_pending = 0 last_contact = 0 last_log_index = 36058 last_log_term = 17 last_snapshot_index = 32778 last_snapshot_term = 17 latest_configuration = [{Suffrage:Voter ID:9f578a39-4995-c4e5-19fd-7d109500cca1 Address:10.4.2.99:8300}] latest_configuration_index = 0 num_peers = 0 protocol_version = 3 protocol_version_max = 3 protocol_version_min = 0 snapshot_version_max = 1 snapshot_version_min = 0 state = Leader term = 17 runtime: arch = amd64 cpu_count = 8 goroutines = 590 max_procs = 8 os = windows version = go1.15.8 serf_lan: coordinate_resets = 0 encrypted = true event_queue = 1 event_time = 17 failed = 0 health_score = 0 intent_queue = 1 left = 0 member_time = 17 members = 1 query_queue = 0 query_time = 1 serf_wan: coordinate_resets = 0 encrypted = true event_queue = 0 event_time = 1 failed = 0 health_score = 0 intent_queue = 0 left = 0 member_time = 1 members = 1 query_queue = 0 query_time = 1 ``` Client agent HCL config ```

Server info

``` Output from server 'consul info' command here ``` ``` Server agent HCL config ```

Operating system and Environment details

2023-03-06T11:30:52.149-0700 [ERROR] agent.http: Request error: method=PUT url="/v1/agent/check/pass/service:service-api?note=procmanager+heartbeat" from=127.0.0.1:63118 error="Unknown check ID \"service:service-api\". Ensure that the check ID is passed, not the check name."

[ikrammohammad]

Hi,

I am still waiting for someone to reply on this. Please help.

Thanks Ikram

[jkirschner-hashicorp]

Upgrade "service-registry" with consul version as 1.11.0

Can you clarify what you mean? Did you upgrade the Consul server agent to 1.11.0 first, then upgrade the client agent to 1.11.0? (Tangential: Why upgrade to 1.11.0 specifically rather than the latest 1.11.x release?)

And can you share your service definition for service-registry? Does it have a check with ID service:service-api?