search

hashicorp / consul

Consul is a distributed, highly available, and data center aware solution to connect and configure applications across dynamic, distributed infrastructure.
https://www.consul.io
Other
28.22k stars 4.41k forks source link

Consul lock should not accept to run on nodes with critical node checks #16107

Open baby-gnu opened 1 year ago

baby-gnu commented 1 year ago

Overview of the Issue

I'm trying to use consul lock to execute a service on a single node but when that node is shutdown (to simulate a failure), the lock is taken on another node where there is a node check in critical state.

I tried to ask information on the forum.

According to session documentation, the session should be invalidated if Any of the health checks go to the critical state but I do not have this behaviour.

Reproduction Steps

Steps to reproduce this issue, eg:

  1. Create a cluster with 5 client/server nodes
  2. Create a node check that is always critical on node2 
    check = {
 name = "Always fails"
 args = ["/bin/bash", "-c", "exit 3"]
 interval = "10s"
 timeout = "1s"
}
  3. Run on a first node consul lock -name simple-loop -verbose service/simple-loop/leader "/bin/bash -c 'while true; do echo "$(date --rfc-3339=seconds): OK"; sleep 5; done'", messages are displayed every 5 seconds
  4. Run on a second node consul lock -name simple-loop -verbose service/simple-loop/leader "/bin/bash -c 'while true; do echo "$(date --rfc-3339=seconds): OK"; sleep 5; done'", the command is waiting on Attempting lock acquisition
  5. On the first node, press Control-C
  6. On the second node, messages are displayed every 5 seconds

Consul info for both Client and Server

Client info ``` agent: check_monitors = 0 check_ttls = 0 checks = 0 services = 0 build: prerelease = revision = bd257019 version = 1.14.3 version_metadata = consul: acl = disabled bootstrap = false known_datacenters = 1 leader = false leader_addr = 10.0.254.103:8300 server = true raft: applied_index = 347455 commit_index = 347455 fsm_pending = 0 last_contact = 90.181604ms last_log_index = 347455 last_log_term = 11 last_snapshot_index = 344141 last_snapshot_term = 9 latest_configuration = [{Suffrage:Voter ID:f6a3e39e-3f09-18cd-f753-e02d1a47022d Address:10.0.254.104:8300} {Suffrage:Voter ID:4acfff4c-891f-c212-af76-4ac7a6c3b735 Address:10.0.254.102:8300} {Suffrage:Voter ID:603d79f4-8408-9040-b2ad-0ed9566063a9 Address:10.0.254.101:8300} {Suffrage:Voter ID:8d5ab5dc-f221-ce2f-284e-6aaef2fadf31 Address:10.0.254.100:8300} {Suffrage:Voter ID:6e29d865-fa16-f9f0-d80d-7ec908ee6752 Address:10.0.254.103:8300}] latest_configuration_index = 0 num_peers = 4 protocol_version = 3 protocol_version_max = 3 protocol_version_min = 0 snapshot_version_max = 1 snapshot_version_min = 0 state = Follower term = 11 runtime: arch = amd64 cpu_count = 4 goroutines = 140 max_procs = 4 os = linux version = go1.19.4 serf_lan: coordinate_resets = 0 encrypted = true event_queue = 0 event_time = 9 failed = 0 health_score = 0 intent_queue = 0 left = 0 member_time = 39 members = 5 query_queue = 0 query_time = 1 serf_wan: coordinate_resets = 0 encrypted = true event_queue = 0 event_time = 1 failed = 0 health_score = 0 intent_queue = 0 left = 0 member_time = 43 members = 5 query_queue = 0 query_time = 1 ```

Server info are the same.

Operating system and Environment details

OS, Architecture, and any other information you can provide about the environment.

5 identical Debian Bullseye virtual machines:

lsb_release -a ``` Distributor ID: Debian Description: Debian GNU/Linux 11 (bullseye) Release: 11 Codename: bullseye ```

Log Fragments

Include appropriate Client or Server log fragments. If the log is longer than a few dozen lines, please include the URL to the gist of the log instead of posting it in the issue. Use -log-level=TRACE on the client and server to capture the maximum log detail.

consul logs ``` janv. 30 15:38:29 moosefs-2 consul[536650]: 2023-01-30T15:38:29.222+0100 [WARN] agent: Check is now critical: check="Always fails" janv. 30 15:38:29 moosefs-2 consul[536650]: 2023-01-30T15:38:29.472+0100 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false janv. 30 15:38:29 moosefs-2 consul[536650]: 2023-01-30T15:38:29.636+0100 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false janv. 30 15:38:29 moosefs-2 consul[536650]: 2023-01-30T15:38:29.642+0100 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false janv. 30 15:38:29 moosefs-2 consul[536650]: 2023-01-30T15:38:29.680+0100 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false janv. 30 15:38:29 moosefs-2 consul[536650]: 2023-01-30T15:38:29.688+0100 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false janv. 30 15:38:31 moosefs-2 consul[536650]: 2023-01-30T15:38:31.472+0100 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false janv. 30 15:38:31 moosefs-2 consul[536650]: 2023-01-30T15:38:31.637+0100 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false janv. 30 15:38:31 moosefs-2 consul[536650]: 2023-01-30T15:38:31.642+0100 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false janv. 30 15:38:31 moosefs-2 consul[536650]: 2023-01-30T15:38:31.679+0100 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false janv. 30 15:38:31 moosefs-2 consul[536650]: 2023-01-30T15:38:31.687+0100 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false janv. 30 15:38:32 moosefs-2 consul[536650]: 2023-01-30T15:38:32.737+0100 [TRACE] agent.server: rpc_server_call: method=Session.Renew errored=false request_type=read rpc_type=net/rpc leader=false allow_stale=false blocking=false target_datacenter=dijon locality=local janv. 30 15:38:32 moosefs-2 consul[536650]: 2023-01-30T15:38:32.737+0100 [DEBUG] agent.http: Request finished: method=PUT url=/v1/session/renew/a3eb647d-596e-06fa-f99d-7774393590f9 from=127.0.0.1:44802 latency=1.270284ms janv. 30 15:38:33 moosefs-2 consul[536650]: 2023-01-30T15:38:33.386+0100 [TRACE] agent.server: rpc_server_call: method=KVS.Get errored=false request_type=read rpc_type=net/rpc leader=false allow_stale=false blocking=true target_datacenter=dijon locality=local janv. 30 15:38:33 moosefs-2 consul[536650]: 2023-01-30T15:38:33.387+0100 [DEBUG] agent.http: Request finished: method=GET url="/v1/kv/service/moosefs/leader/.lock?index=347413&wait=15000ms" from=127.0.0.1:33182 latency=15.656219509s janv. 30 15:38:33 moosefs-2 consul[536650]: 2023-01-30T15:38:33.472+0100 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false janv. 30 15:38:33 moosefs-2 consul[536650]: 2023-01-30T15:38:33.636+0100 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false janv. 30 15:38:33 moosefs-2 consul[536650]: 2023-01-30T15:38:33.642+0100 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false janv. 30 15:38:33 moosefs-2 consul[536650]: 2023-01-30T15:38:33.680+0100 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false janv. 30 15:38:33 moosefs-2 consul[536650]: 2023-01-30T15:38:33.688+0100 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false janv. 30 15:38:35 moosefs-2 consul[536650]: 2023-01-30T15:38:35.463+0100 [TRACE] agent.server.usage_metrics: Starting usage run janv. 30 15:38:35 moosefs-2 consul[536650]: 2023-01-30T15:38:35.472+0100 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false janv. 30 15:38:35 moosefs-2 consul[536650]: 2023-01-30T15:38:35.636+0100 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false janv. 30 15:38:35 moosefs-2 consul[536650]: 2023-01-30T15:38:35.642+0100 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false janv. 30 15:38:35 moosefs-2 consul[536650]: 2023-01-30T15:38:35.679+0100 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false janv. 30 15:38:35 moosefs-2 consul[536650]: 2023-01-30T15:38:35.688+0100 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false janv. 30 15:38:37 moosefs-2 consul[536650]: 2023-01-30T15:38:37.472+0100 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false janv. 30 15:38:37 moosefs-2 consul[536650]: 2023-01-30T15:38:37.637+0100 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false janv. 30 15:38:37 moosefs-2 consul[536650]: 2023-01-30T15:38:37.642+0100 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false janv. 30 15:38:37 moosefs-2 consul[536650]: 2023-01-30T15:38:37.680+0100 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false janv. 30 15:38:37 moosefs-2 consul[536650]: 2023-01-30T15:38:37.688+0100 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false janv. 30 15:38:39 moosefs-2 consul[536650]: 2023-01-30T15:38:39.227+0100 [TRACE] agent: Check output: check="Always fails" output="" janv. 30 15:38:39 moosefs-2 consul[536650]: 2023-01-30T15:38:39.227+0100 [WARN] agent: Check is now critical: check="Always fails" janv. 30 15:38:39 moosefs-2 consul[536650]: 2023-01-30T15:38:39.472+0100 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false janv. 30 15:38:39 moosefs-2 consul[536650]: 2023-01-30T15:38:39.636+0100 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false janv. 30 15:38:39 moosefs-2 consul[536650]: 2023-01-30T15:38:39.643+0100 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false janv. 30 15:38:39 moosefs-2 consul[536650]: 2023-01-30T15:38:39.680+0100 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false janv. 30 15:38:39 moosefs-2 consul[536650]: 2023-01-30T15:38:39.688+0100 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false janv. 30 15:38:40 moosefs-2 consul[536650]: 2023-01-30T15:38:40.239+0100 [TRACE] agent.server: rpc_server_call: method=Session.Renew errored=false request_type=read rpc_type=net/rpc leader=false allow_stale=false blocking=false target_datacenter=dijon locality=local janv. 30 15:38:40 moosefs-2 consul[536650]: 2023-01-30T15:38:40.239+0100 [DEBUG] agent.http: Request finished: method=PUT url=/v1/session/renew/a3eb647d-596e-06fa-f99d-7774393590f9 from=127.0.0.1:44802 latency=1.376331ms janv. 30 15:38:40 moosefs-2 consul[536650]: 2023-01-30T15:38:40.839+0100 [DEBUG] agent.http: Request finished: method=GET url=/v1/agent/self from=127.0.0.1:59588 latency=1.019438ms janv. 30 15:38:40 moosefs-2 consul[536650]: 2023-01-30T15:38:40.849+0100 [TRACE] agent.server: rpc_server_call: method=Session.Apply errored=false request_type=write rpc_type=net/rpc leader=false target_datacenter=dijon locality=local janv. 30 15:38:40 moosefs-2 consul[536650]: 2023-01-30T15:38:40.849+0100 [DEBUG] agent.http: Request finished: method=PUT url=/v1/session/create from=127.0.0.1:59588 latency=8.170415ms janv. 30 15:38:40 moosefs-2 consul[536650]: 2023-01-30T15:38:40.850+0100 [TRACE] agent.server: rpc_server_call: method=KVS.Get errored=false request_type=read rpc_type=net/rpc leader=false allow_stale=false blocking=false target_datacenter=dijon locality=local janv. 30 15:38:40 moosefs-2 consul[536650]: 2023-01-30T15:38:40.850+0100 [DEBUG] agent.http: Request finished: method=GET url=/v1/kv/service/simple-loop/leader/.lock?wait=15000ms from=127.0.0.1:59588 latency="425.397µs" janv. 30 15:38:41 moosefs-2 consul[536650]: 2023-01-30T15:38:41.471+0100 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false janv. 30 15:38:41 moosefs-2 consul[536650]: 2023-01-30T15:38:41.636+0100 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false janv. 30 15:38:41 moosefs-2 consul[536650]: 2023-01-30T15:38:41.642+0100 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false janv. 30 15:38:41 moosefs-2 consul[536650]: 2023-01-30T15:38:41.681+0100 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false janv. 30 15:38:41 moosefs-2 consul[536650]: 2023-01-30T15:38:41.688+0100 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false janv. 30 15:38:43 moosefs-2 consul[536650]: 2023-01-30T15:38:43.472+0100 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false janv. 30 15:38:43 moosefs-2 consul[536650]: 2023-01-30T15:38:43.636+0100 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false janv. 30 15:38:43 moosefs-2 consul[536650]: 2023-01-30T15:38:43.643+0100 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false janv. 30 15:38:43 moosefs-2 consul[536650]: 2023-01-30T15:38:43.680+0100 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false janv. 30 15:38:43 moosefs-2 consul[536650]: 2023-01-30T15:38:43.687+0100 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false janv. 30 15:38:44 moosefs-2 consul[536650]: 2023-01-30T15:38:44.463+0100 [TRACE] agent.server.usage_metrics: Starting usage run janv. 30 15:38:45 moosefs-2 consul[536650]: 2023-01-30T15:38:45.472+0100 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false janv. 30 15:38:45 moosefs-2 consul[536650]: 2023-01-30T15:38:45.637+0100 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false janv. 30 15:38:45 moosefs-2 consul[536650]: 2023-01-30T15:38:45.642+0100 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false janv. 30 15:38:45 moosefs-2 consul[536650]: 2023-01-30T15:38:45.680+0100 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false janv. 30 15:38:45 moosefs-2 consul[536650]: 2023-01-30T15:38:45.688+0100 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false janv. 30 15:38:46 moosefs-2 consul[536650]: 2023-01-30T15:38:46.276+0100 [TRACE] agent.server: rpc_server_call: method=KVS.Get errored=false request_type=read rpc_type=net/rpc leader=false allow_stale=false blocking=true target_datacenter=dijon locality=local janv. 30 15:38:46 moosefs-2 consul[536650]: 2023-01-30T15:38:46.276+0100 [DEBUG] agent.http: Request finished: method=GET url="/v1/kv/service/simple-loop/leader/.lock?index=347549&wait=15000ms" from=127.0.0.1:59588 latency=5.42554313s janv. 30 15:38:46 moosefs-2 consul[536650]: 2023-01-30T15:38:46.286+0100 [TRACE] agent.server: rpc_server_call: method=KVS.Apply errored=false request_type=write rpc_type=net/rpc leader=false target_datacenter=dijon locality=local janv. 30 15:38:46 moosefs-2 consul[536650]: 2023-01-30T15:38:46.286+0100 [DEBUG] agent.http: Request finished: method=PUT url="/v1/kv/service/simple-loop/leader/.lock?acquire=cb51912b-c3bc-06c5-a6e1-3b6de0734aa6&flags=3304740253564472344" from=127.0.0.1:59588 latency=9.215896ms janv. 30 15:38:46 moosefs-2 consul[536650]: 2023-01-30T15:38:46.287+0100 [TRACE] agent.server: rpc_server_call: method=KVS.Get errored=false request_type=read rpc_type=net/rpc leader=false allow_stale=false blocking=false target_datacenter=dijon locality=local janv. 30 15:38:46 moosefs-2 consul[536650]: 2023-01-30T15:38:46.287+0100 [DEBUG] agent.http: Request finished: method=GET url=/v1/kv/service/simple-loop/leader/.lock?consistent= from=127.0.0.1:59588 latency="681.303µs" janv. 30 15:38:47 moosefs-2 consul[536650]: 2023-01-30T15:38:47.376+0100 [TRACE] agent.server: rpc_server_call: method=Coordinate.Update errored=false request_type=write rpc_type=net/rpc leader=false target_datacenter=dijon locality=local janv. 30 15:38:47 moosefs-2 consul[536650]: 2023-01-30T15:38:47.472+0100 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false janv. 30 15:38:47 moosefs-2 consul[536650]: 2023-01-30T15:38:47.637+0100 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false janv. 30 15:38:47 moosefs-2 consul[536650]: 2023-01-30T15:38:47.643+0100 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false janv. 30 15:38:47 moosefs-2 consul[536650]: 2023-01-30T15:38:47.680+0100 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false janv. 30 15:38:47 moosefs-2 consul[536650]: 2023-01-30T15:38:47.688+0100 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false janv. 30 15:38:47 moosefs-2 consul[536650]: 2023-01-30T15:38:47.742+0100 [TRACE] agent.server: rpc_server_call: method=Session.Renew errored=false request_type=read rpc_type=net/rpc leader=false allow_stale=false blocking=false target_datacenter=dijon locality=local janv. 30 15:38:47 moosefs-2 consul[536650]: 2023-01-30T15:38:47.742+0100 [DEBUG] agent.http: Request finished: method=PUT url=/v1/session/renew/a3eb647d-596e-06fa-f99d-7774393590f9 from=127.0.0.1:44802 latency=1.095246ms janv. 30 15:38:48 moosefs-2 consul[536650]: 2023-01-30T15:38:48.352+0100 [TRACE] agent.server: rpc_server_call: method=Session.Renew errored=false request_type=read rpc_type=net/rpc leader=false allow_stale=false blocking=false target_datacenter=dijon locality=local janv. 30 15:38:48 moosefs-2 consul[536650]: 2023-01-30T15:38:48.352+0100 [DEBUG] agent.http: Request finished: method=PUT url=/v1/session/renew/cb51912b-c3bc-06c5-a6e1-3b6de0734aa6 from=127.0.0.1:59592 latency="969.717µs" janv. 30 15:38:48 moosefs-2 consul[536650]: 2023-01-30T15:38:48.391+0100 [TRACE] agent.server: rpc_server_call: method=KVS.Get errored=false request_type=read rpc_type=net/rpc leader=false allow_stale=false blocking=true target_datacenter=dijon locality=local janv. 30 15:38:48 moosefs-2 consul[536650]: 2023-01-30T15:38:48.391+0100 [DEBUG] agent.http: Request finished: method=GET url="/v1/kv/service/moosefs/leader/.lock?index=347413&wait=15000ms" from=127.0.0.1:33182 latency=15.003206691s janv. 30 15:38:49 moosefs-2 consul[536650]: 2023-01-30T15:38:49.112+0100 [TRACE] agent.server: rpc_server_call: method=KVS.Apply errored=false request_type=write rpc_type=net/rpc leader=false target_datacenter=dijon locality=local janv. 30 15:38:49 moosefs-2 consul[536650]: 2023-01-30T15:38:49.112+0100 [DEBUG] agent.http: Request finished: method=PUT url="/v1/kv/service/simple-loop/leader/.lock?flags=3304740253564472344&release=cb51912b-c3bc-06c5-a6e1-3b6de0734aa6" from=127.0.0.1:59592 latency=9.333512ms janv. 30 15:38:49 moosefs-2 consul[536650]: 2023-01-30T15:38:49.112+0100 [TRACE] agent.server: rpc_server_call: method=KVS.Get errored=false request_type=read rpc_type=net/rpc leader=false allow_stale=false blocking=true target_datacenter=dijon locality=local janv. 30 15:38:49 moosefs-2 consul[536650]: 2023-01-30T15:38:49.112+0100 [DEBUG] agent.http: Request finished: method=GET url="/v1/kv/service/simple-loop/leader/.lock?consistent=&index=347556" from=127.0.0.1:59588 latency=2.824764577s janv. 30 15:38:49 moosefs-2 consul[536650]: 2023-01-30T15:38:49.113+0100 [TRACE] agent.server: rpc_server_call: method=KVS.Get errored=false request_type=read rpc_type=net/rpc leader=false allow_stale=false blocking=false target_datacenter=dijon locality=local janv. 30 15:38:49 moosefs-2 consul[536650]: 2023-01-30T15:38:49.113+0100 [DEBUG] agent.http: Request finished: method=GET url=/v1/kv/service/simple-loop/leader/.lock from=127.0.0.1:59592 latency="500.67µs" janv. 30 15:38:49 moosefs-2 consul[536650]: 2023-01-30T15:38:49.121+0100 [TRACE] agent.server: rpc_server_call: method=Session.Apply errored=false request_type=write rpc_type=net/rpc leader=false target_datacenter=dijon locality=local janv. 30 15:38:49 moosefs-2 consul[536650]: 2023-01-30T15:38:49.121+0100 [DEBUG] agent.http: Request finished: method=PUT url=/v1/session/destroy/cb51912b-c3bc-06c5-a6e1-3b6de0734aa6 from=127.0.0.1:59594 latency=8.289163ms janv. 30 15:38:49 moosefs-2 consul[536650]: 2023-01-30T15:38:49.123+0100 [TRACE] agent.server: rpc_server_call: method=KVS.Apply errored=false request_type=write rpc_type=net/rpc leader=false target_datacenter=dijon locality=local janv. 30 15:38:49 moosefs-2 consul[536650]: 2023-01-30T15:38:49.123+0100 [DEBUG] agent.http: Request finished: method=DELETE url=/v1/kv/service/simple-loop/leader/.lock?cas=347557 from=127.0.0.1:59592 latency=10.286325ms janv. 30 15:38:49 moosefs-2 consul[536650]: 2023-01-30T15:38:49.231+0100 [TRACE] agent: Check output: check="Always fails" output="" janv. 30 15:38:49 moosefs-2 consul[536650]: 2023-01-30T15:38:49.231+0100 [WARN] agent: Check is now critical: check="Always fails" janv. 30 15:38:49 moosefs-2 consul[536650]: 2023-01-30T15:38:49.471+0100 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false janv. 30 15:38:49 moosefs-2 consul[536650]: 2023-01-30T15:38:49.636+0100 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false janv. 30 15:38:49 moosefs-2 consul[536650]: 2023-01-30T15:38:49.643+0100 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false janv. 30 15:38:49 moosefs-2 consul[536650]: 2023-01-30T15:38:49.679+0100 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false janv. 30 15:38:49 moosefs-2 consul[536650]: 2023-01-30T15:38:49.688+0100 [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false ```
baby-gnu commented 6 months ago

Still an issue with v1.17.3.

Regards.

baby-gnu commented 5 months ago

@huikang any idea if this will be addressed?