cloudbustinguk
commented
1 year ago Extra context in case it''s helpful.
The below run is not the same run as my initial post.. but all exhibit the same output anyway.
During the "wait" period, the runner logs are effectively spinning on the following:
[WORKER 2023-06-04 15:11:59Z INFO HostContext] Well known directory 'Bin': '/home/runner/bin'
[WORKER 2023-06-04 15:11:59Z INFO HostContext] Well known directory 'Root': '/home/runner'
[WORKER 2023-06-04 15:11:59Z INFO HostContext] Well known directory 'Work': '/runner/_work'
[RUNNER 2023-06-04 15:12:02Z INFO JobDispatcher] Successfully renew job request 59, job is valid till 06/04/2023 15:22:02
[WORKER 2023-06-04 15:12:09Z INFO HostContext] Well known directory 'Bin': '/home/runner/bin'
[WORKER 2023-06-04 15:12:09Z INFO HostContext] Well known directory 'Root': '/home/runner'
[WORKER 2023-06-04 15:12:09Z INFO HostContext] Well known directory 'Work': '/runner/_work'
[WORKER 2023-06-04 15:12:19Z INFO HostContext] Well known directory 'Bin': '/home/runner/bin'
[WORKER 2023-06-04 15:12:19Z INFO HostContext] Well known directory 'Root': '/home/runner'
[WORKER 2023-06-04 15:12:19Z INFO HostContext] Well known directory 'Work': '/runner/_work'
[WORKER 2023-06-04 15:12:29Z INFO HostContext] Well known directory 'Bin': '/home/runner/bin'
[WORKER 2023-06-04 15:12:29Z INFO HostContext] Well known directory 'Root': '/home/runner'
[WORKER 2023-06-04 15:12:29Z INFO HostContext] Well known directory 'Work': '/runner/_work'
[WORKER 2023-06-04 15:12:39Z INFO HostContext] Well known directory 'Bin': '/home/runner/bin'
..... before the time is up and the rest of the process continues:
...
[WORKER 2023-06-04 15:15:29Z INFO HostContext] Well known directory 'Root': '/home/runner'
[WORKER 2023-06-04 15:15:29Z INFO HostContext] Well known directory 'Work': '/runner/_work'
[WORKER 2023-06-04 15:15:31Z INFO JobServerQueue] Try to append 1 batches web console lines for record '1ab5e0cd-4ace-4d35-b32b-2935204705d0', success rate: 1/1.
[WORKER 2023-06-04 15:15:34Z INFO ProcessInvokerWrapper] STDOUT/STDERR stream read finished.
[WORKER 2023-06-04 15:15:34Z INFO ProcessInvokerWrapper] STDOUT/STDERR stream read finished.
[WORKER 2023-06-04 15:15:34Z INFO ProcessInvokerWrapper] Finished process 130 with exit code 0, and elapsed time 00:13:45.5155178.
[WORKER 2023-06-04 15:15:34Z INFO JobServerQueue] Try to append 1 batches web console lines for record '1ab5e0cd-4ace-4d35-b32b-2935204705d0', success rate: 1/1.
[WORKER 2023-06-04 15:15:34Z INFO ContainerHookManager] Response file for the hook script at '/runner/k8s/index.js' running command 'PrepareJob' was processed successfully
[WORKER 2023-06-04 15:15:34Z INFO ContainerHookManager] Response file for the hook script at '/runner/k8s/index.js' running command 'PrepareJob' was deleted successfully
[WORKER 2023-06-04 15:15:34Z INFO ContainerHookManager] Global variable 'ContainerHookState' updated successfully for 'PrepareJob' with data found in 'state' property of the response file.
[WORKER 2023-06-04 15:15:34Z INFO StepsRunner] Step result:
[WORKER 2023-06-04 15:15:34Z INFO ExecutionContext] Publish step telemetry for current step {
[WORKER 2023-06-04 15:15:34Z INFO ExecutionContext] "action": "Pre Job Hook",
...
nikola-jokic
nixlim
anlesk
Hey there,
Just looking for a bit of a pointer here, and to ask whether you've seen this behaviour before.
One of my customers is an early adopter of EKS-Anywhere (
v1.23.15-eks-69f0cbfwith CRI-O + containerd), currently on vSphere 7.x, and they havecontainerMode: kubernetessomewhat working using aRunnerDeploymenton the current releast of ARC (v0.27.4), the runner isv2.304.0runner image. Hooks is0.3.2. The job container image isubi8/ubi-minimal:8.8The "somewhat" part is that every job, no matter the size of the job container, takes almost exactly 14 minutes to get past theInitialize containersstep within the workflow, but then does go on to succeed very swiftly (just a "hello world" step next). It's perhaps prudent to note that this is their first attempt at using ARC on this cluster at all; it's still a PoC for them.The
RunnerDeployment'simagePullPolicyisIfNotPresent, and in the case where it's not, I've debugged this on the nodes themselves, and I see intcpdumpthat a connection to their Artifact store isn't actually attempted until ~14 minutes elapses. In the case where the image is already present, of course no pull is attempted, but the same duration elapses before the job ultimately succeeds.The (two) k8s worker nodes have previously been 2 vCPU and 8 GB, so I bumped resources to 8 vCPU and 32GB, and there has been no change in outcome.. I still see no evidence on the nodes or otherwise that a resource shortfall is in play here.
The CSI I am using is NetApp Trident, and PVCs get an PV and go bound within a few seconds - no issues there.
Here is a redacted shot of a typical workflow. I can confirm that the image - pulled manually on the worker nodes - does indeed pull down within a few seconds (
time crictl pull ...), so I am not convinced this is a network issue.The screenshot doesn't show it, but line 10 (
##[debug]/home/runner/externals/node16/bin/node /runner/k8s/index.js) is where the 14 mins is spent - see further for the raw log counterpart which does show it.f
Here is the raw log excerpt which shows the ~14m delay between the stages:
The processes on the runner themselves are sleeping during this time, and strace (outpt not included for brevity's sake) shows a typical non-blocking I/O cycle of sleep, wake, check futex, yield and go back to sleep. See runner processes sleeping - a state they remain for almost the entire 14m period:
As mentioned, as soon as the 14 min has elapsed, the workflow pod comes up within seconds, and the job moves on to successful completion.
Thanks for any pointers or ideas!