Closed su-thomas closed 2 years ago
This is the discussion thread in Discord
@su-thomas that times are really high. Why 500ms? That is ~10 times the average to complete the request
/tmp $ ./hey_linux_amd64 -n 300 -m GET http://localhost:8086/ready
Summary:
Total: 0.4252 secs
Slowest: 0.2342 secs
Fastest: 0.0164 secs
Average: 0.0567 secs
Requests/sec: 705.4881
Total data: 900 bytes
Size/request: 3 bytes
Response time histogram:
0.016 [1] |
0.038 [46] |■■■■■■■■■■■
0.060 [162] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
0.082 [63] |■■■■■■■■■■■■■■■■
0.104 [9] |■■
0.125 [12] |■■■
0.147 [0] |
0.169 [5] |■
0.191 [0] |
0.212 [1] |
0.234 [1] |
Latency distribution:
10% in 0.0337 secs
25% in 0.0426 secs
50% in 0.0528 secs
75% in 0.0613 secs
90% in 0.0772 secs
95% in 0.1094 secs
99% in 0.1674 secs
Details (average, fastest, slowest):
DNS+dialup: 0.0007 secs, 0.0164 secs, 0.2342 secs
DNS-lookup: 0.0005 secs, 0.0000 secs, 0.0039 secs
req write: 0.0001 secs, 0.0000 secs, 0.0018 secs
resp wait: 0.0559 secs, 0.0164 secs, 0.2341 secs
resp read: 0.0001 secs, 0.0000 secs, 0.0022 secs
Status code distribution:
[200] 300 responses
I suggest you check the CPU utilization (I/O in general) in the node where you see this behavior.
Edit: this is from a gitpod.io production node
This is the discussion thread in Discord
@MrSimonEmms I get Messages failed to load
@aledbf I think you'll need to sign-up for the self-hosted part of the Discord community to view that
I suggest you check the CPU utilization (I/O in general) in the node where you see this behavior.
I work with @su-thomas and initially built the cluster. It is running with the exact eksctl commands as specified in the single-cluster reference implementation, and the container he launches is in the same pod, so same machine. It would appear that on every connection to the readiness probe, there are lots of checks happening, and that's part of the problem.
It's worth noting that this is running the exact same EC2 instance type as recommended, with 2 nodegroups of 2 nodes each, running nothing other than gitpod, in the ap-southeast-2 (Sydney) region -- so it's entirely possible that if there's a call via the RBAC that AWS are proxying that call to the master back to the US. That might explain the latency.
Either way, a 500ms timeout seems incredibly low for such a check, considering the service takes many seconds to come up in the first place. Even in your tests the timeout is only twice the maximum observed duration!
@aledbf, would I be correct in understanding that when a GET
request is made (by the node-labeler
) for http://127.0.0.1:8086/ready
(served by the registry-facade
) that then this will call all the readiness checks defined? (source code of heptiolabs/healthcheck
here where each check is called by the handler).
There are 3 readiness checks for the registry-facade
here. Interestingly, each one of those tests is given 1 second timeout each (here, and here).
It seems contradictory that the node-labeler
(client) is hardcoded for a time out for 500ms for the response, whilst the checks (triggered by the request) on the server-side are given 1 second time out (each). Would I be wrong in saying that the maximum time a valid health check could take is upwards of 3 seconds (assuming this source code is running each check after the next without concurrency)? So why is the client (node-labeler
) being given only 500ms to get a response from registry-facade
?
Again, is there any reason to have a very short timeout fixed for the node-labeler
? Is there any reason against letting the node-labeler
continue to increase its http.Client
timeout on each failed tick? This way if it fails at 500ms, it could next check (for example) 750ms, then 1000ms, so on... up to a defined threshold.
It's possible that the health checks for eu.gcr.io
are taking longer for our deployment in ap-southeast-2 (Sydney) region.
:wave: @su-thomas do you have a workaround, or, are you unable to use Gitpod at the moment? I assume unable, but, wanted to double check.
that then this will call all the readiness checks defined?
That's correct.
It seems contradictory that the node-labeler (client) is hardcoded for a time out for 500ms for the response, whilst the checks (triggered by the request) on the server-side are given 1 second time out (each)
Not really. The node-labeler waitForURLToBeReachable
has a default timeout of 60s
It's possible that the health checks for eu.gcr.io are taking longer for our deployment in ap-southeast-2 (Sydney) region.
That could be the issue, but then you need to start increasing the timeout of the regisdtry-facade probes timeout.
We removed the internal timeout, using only one. Please check https://github.com/gitpod-io/gitpod/pull/12272. This change will be included in the next self-hosted release.
👋 @su-thomas do you have a workaround, or, are you unable to use Gitpod at the moment? I assume unable, but, wanted to double check.
Hi @kylos101, from my initial post:
As a manual work around, I removed the problematic
node-labeler
container from theregistry-facade
daemonset, and ran theready-probe-labeler
(with 1000mstick
modification) in my alpine container to apply the node label. After doing this, my GitPod deployment seems to function fine, and theregistry-facade
pod is working as expected with noCrashLoopBackOff
.
@aledbf
That could be the issue, but then you need to start increasing the timeout of the registry-facade probes timeout.
The registry-facade
probes' timeouts were fine, as you can see in my initial post, I could get a 200
response if I curl
http://localhost:8086/ready
, and the entire health check was responding in ~800ms. The registry-facade
's health probes timeout already allowed up to a total 3 seconds to still be valid, because each of the 3 checks had 1 second each. So these timeouts were fine as is.
Not really. The node-labeler
waitForURLToBeReachable
has a default timeout of 60s
Yes, a total duration of 60s, but thats not the issue. The issue is that on each attempt within the 60s duration, the http.Client
was only given 500ms timeout as per the quoted waitForURLToBeReachable
source code in my initial post:
...
tick := 500 * time.Millisecond
...
client := &http.Client{
Timeout: tick,
}
...
...therefore it didn't matter that there was 60s duration given for the attempts, each attempt was failing, because the http.Client
had been limited to 500ms on each attempt.
We removed the internal timeout, using only one. Please check https://github.com/gitpod-io/gitpod/pull/12272. This change will be included in the next self-hosted release.
This looks like it should fix this issue, thanks! 😊
Bug description
tl;dr: The
ready-probe-labeler
has a hardcodedtimeout
of 500ms set for thehttp.Client
that performs the/ready
probe check. In my case, theregistry-facade
response for the/ready
endpoint takes ~800ms. Theready-probe-labeler
exits with an error because it never sees the200
response status code due to the premature timeout. This causes the pod to sufferCrashLoopBackOff
.I followed the GitPod Single-Cluster Reference Architecture installation guide for AWS.
After deploying GitPod through the kots installer, I noticed the
registry-facade
pods were inCrashLoopBackOff
status.Looking at the k8s events, I noticed:
Additionally, the
node-labeler
container logs showed that it wasn't seeing a200
response status code from the/ready
endpoint:This initially led me to believe there was a problem with the
/ready
endpoint on theregistry-facade
container. So to investigate further, I setup an alpine container in theregistry-facade
daemonset.I verified the
/ready
endpoint was actually working fine within the pod:So, why wasn't the
node-labeler
getting a200
response status code?After looking through the GitPod source code, I came across the function responsible for checking the probe:
Notably, there is a hardcoded timeout of 500ms set for the
http.Client
.Adding some additional logging to this function...:
...and running it in the pod:
...it appears the
http.Client
is getting interrupted before it receives the headers. This error continues everytick
until thetimeout
duration ends the loop, then the program exits1
. This error exit is likely what is causing theCrashLoopBackOff
due to the defaultrestartPolicy
of the pod(?).Benchmarking the
/ready
endpoint withhey
, its clear that it doesn't respond within 500ms (at least in my environment).If I set the
tick
to a longer duration such as 1000ms...:...then it works successfully:
As a manual work around, I removed the problematic
node-labeler
container from theregistry-facade
daemonset, and ran theready-probe-labeler
(with 1000mstick
modification) in my alpine container to apply the node label. After doing this, my GitPod deployment seems to function fine, and theregistry-facade
pod is working as expected with noCrashLoopBackOff
.Steps to reproduce
registry-facade
pods.Workspace affected
No response
Expected behavior
node-labeler
should receive a200
response status code from theregistry-facade
/ready
probe.Example repository
No response
Anything else?
This issue raises some questions:
http.Client()
in theready-probe-labeler
?registry-facade
actually being healthy) because theexit 1
from thenode-labeler
container causes aCrashLoopBackOff
for the pod. Perhaps thetick
could increase in length each loop up to a certain threshold, rather than a fixed length that may be too short./ready
endpoint taking ~800ms to respond?ready-probe-labeler
), it would seem that its assumed it can respond in <500ms.