search

falldamagestudio / UE-Jenkins-BuildSystem

Build Unreal Engine & games with Jenkins on GKE/GCE
MIT License
46 stars 11 forks source link

Error when starting pod, "Unable to attach or mount volumes ... timed out waiting for the condition" #23

Closed Kalmalyzer closed 3 years ago

Kalmalyzer commented 3 years ago

This happened for the first time when launching a job to build the game, after the previous job likely managed to downsync an UE build to the workspace.

2021-01-24 18:27:31  [Pipeline] Start of Pipeline
2021-01-24 18:27:32  [Pipeline] podTemplate
2021-01-24 18:27:32  [Pipeline] {
2021-01-24 18:27:32  [Pipeline] node
2021-01-24 18:27:35  Created Pod: kubernetes default/build-game-windows-4-98x8l-nm6jn-fshw5
2021-01-24 18:27:35  [Normal][default/build-game-windows-4-98x8l-nm6jn-fshw5][Scheduled] Successfully assigned default/build-game-windows-4-98x8l-nm6jn-fshw5 to gke-jenkins-jenkins-agent-windows-nod-5255f5cd-qmkp
2021-01-24 18:27:46  [Normal][default/build-game-windows-4-98x8l-nm6jn-fshw5][SuccessfulAttachVolume] AttachVolume.Attach succeeded for volume "pvc-23e11cd9-e8bb-496c-86ac-4c0f624a67dc" 
2021-01-24 18:27:47  Still waiting to schedule task
2021-01-24 18:27:47  ‘build-game-windows-4-98x8l-nm6jn-fshw5’ is offline
2021-01-24 18:29:38  [Warning][default/build-game-windows-4-98x8l-nm6jn-fshw5][FailedMount] Unable to attach or mount volumes: unmounted volumes=[workspace-volume], unattached volumes=[default-token-8rgbw workspace-volume]: timed out waiting for the condition
2021-01-24 18:31:53  [Warning][default/build-game-windows-4-98x8l-nm6jn-fshw5][FailedMount] Unable to attach or mount volumes: unmounted volumes=[workspace-volume], unattached volumes=[workspace-volume default-token-8rgbw]: timed out waiting for the condition
2021-01-24 18:34:05  Created Pod: kubernetes default/build-game-windows-4-98x8l-nm6jn-plmhx
2021-01-24 18:34:05  [Warning][default/build-game-windows-4-98x8l-nm6jn-plmhx][FailedScheduling] 0/2 nodes are available: 1 node(s) didn't match node selector, 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't satisfy existing pods anti-affinity rules.
2021-01-24 18:34:07  [Normal][default/build-game-windows-4-98x8l-nm6jn-plmhx][Scheduled] Successfully assigned default/build-game-windows-4-98x8l-nm6jn-plmhx to gke-jenkins-jenkins-agent-windows-nod-5255f5cd-qmkp
2021-01-24 18:34:10  [Warning][default/build-game-windows-4-98x8l-nm6jn-fshw5][FailedMount] Unable to attach or mount volumes: unmounted volumes=[default-token-8rgbw workspace-volume], unattached volumes=[default-token-8rgbw workspace-volume]: timed out waiting for the condition
2021-01-24 18:36:10  [Warning][default/build-game-windows-4-98x8l-nm6jn-plmhx][FailedMount] Unable to attach or mount volumes: unmounted volumes=[workspace-volume], unattached volumes=[workspace-volume default-token-8rgbw]: timed out waiting for the condition
2021-01-24 18:38:28  [Warning][default/build-game-windows-4-98x8l-nm6jn-plmhx][FailedMount] Unable to attach or mount volumes: unmounted volumes=[workspace-volume], unattached volumes=[default-token-8rgbw workspace-volume]: timed out waiting for the condition
2021-01-24 18:40:44  [Warning][default/build-game-windows-4-98x8l-nm6jn-plmhx][FailedMount] Unable to attach or mount volumes: unmounted volumes=[workspace-volume], unattached volumes=[workspace-volume default-token-8rgbw]: timed out waiting for the condition
2021-01-24 18:41:05  Created Pod: kubernetes default/build-game-windows-4-98x8l-nm6jn-r6pmz
2021-01-24 18:41:05  [Warning][default/build-game-windows-4-98x8l-nm6jn-r6pmz][FailedScheduling] 0/2 nodes are available: 1 node(s) didn't match node selector, 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't satisfy existing pods anti-affinity rules.
2021-01-24 18:41:05  [Warning][default/build-game-windows-4-98x8l-nm6jn-r6pmz][FailedScheduling] 0/2 nodes are available: 1 node(s) didn't match node selector, 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't satisfy existing pods anti-affinity rules.
2021-01-24 18:41:08  [Normal][default/build-game-windows-4-98x8l-nm6jn-r6pmz][Scheduled] Successfully assigned default/build-game-windows-4-98x8l-nm6jn-r6pmz to gke-jenkins-jenkins-agent-windows-nod-5255f5cd-qmkp
2021-01-24 18:43:02  [Warning][default/build-game-windows-4-98x8l-nm6jn-plmhx][FailedMount] Unable to attach or mount volumes: unmounted volumes=[workspace-volume default-token-8rgbw], unattached volumes=[workspace-volume default-token-8rgbw]: timed out waiting for the condition
2021-01-24 18:43:11  [Warning][default/build-game-windows-4-98x8l-nm6jn-r6pmz][FailedMount] Unable to attach or mount volumes: unmounted volumes=[workspace-volume], unattached volumes=[workspace-volume default-token-8rgbw]: timed out waiting for the condition
2021-01-24 18:45:28  [Warning][default/build-game-windows-4-98x8l-nm6jn-r6pmz][FailedMount] Unable to attach or mount volumes: unmounted volumes=[workspace-volume], unattached volumes=[workspace-volume default-token-8rgbw]: timed out waiting for the condition
Kalmalyzer commented 3 years ago

It's likely associated with this: https://github.com/jenkinsci/helm-charts/tree/main/charts/jenkins#long-volume-attachmount-times

Kalmalyzer commented 3 years ago

... well, the problem and associated solutions posted online all talk about Linux-based filesystems and Kubernetes wanting to change user of all files within the mounted filesystem, one by one.

I haven't seen this happen in a while now; closing.