Deploy agones failed in Tencent Auto Scaling Kubernetes cluster, failed to generate container agones-allocator

kaiiak commented 3 years ago

What happened:

kubectl get pods --all-namespaces
NAMESPACE       NAME                                 READY   STATUS                 RESTARTS   AGE
agones-system   agones-allocator-657975759d-jmz7t    0/1     CreateContainerError   0          78s
agones-system   agones-allocator-657975759d-qbxzh    0/1     CreateContainerError   0          78s
agones-system   agones-allocator-657975759d-tnshh    0/1     CreateContainerError   0          78s
agones-system   agones-controller-67488bcf5f-5ksfv   0/1     RunContainerError      0          78s
agones-system   agones-ping-745d58fb77-72p9r         0/1     RunContainerError      0          78s
agones-system   agones-ping-745d58fb77-77fvj         0/1     RunContainerError      0          78s

`failed to mkdir "": mkdir : no such file or directory`

What you expected to happen: Deploy success How to reproduce it (as minimally and precisely as possible): I follow the Documentation, install agones use yaml Anything else we need to know?:

Environment:

Agones version: 1.12.0

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:50:19Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.9-eks.3", GitCommit:"9ccc3f97803ee00eec9de336cad54b854f45db0c", GitTreeState:"clean", BuildDate:"2021-01-07T02:47:35Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider or hardware configuration: Tecent Auto Scaling Kubernetes cluster
Install method (yaml/helm): yaml

Troubleshooting guide log(s):


kubectl describe --namespace=agones-system pods/agones-allocator-657975759d-tnshh

Name: agones-allocator-657975759d-tnshh Namespace: agones-system Priority: 0 Node: cls-5ke8ljh3-virtual-kubelet-subnet-ejptgbn5-0/ Start Time: Thu, 18 Feb 2021 03:12:01 +0800 Labels: app=agones heritage=Helm multicluster.agones.dev/role=allocator pod-template-hash=657975759d release=agones-manual Annotations: prometheus.io/path: /metrics prometheus.io/port: 8080 prometheus.io/scrape: true Status: Pending IP: 10.0.32.6 IPs: IP: 10.0.32.6 Controlled By: ReplicaSet/agones-allocator-657975759d Containers: agones-allocator: Container ID:
Image: registry.cn-hangzhou.aliyuncs.com/kaiiak/agones-allocator:1.12.0 Image ID:
Port: 8443/TCP Host Port: 0/TCP State: Waiting Reason: CreateContainerError Ready: False Restart Count: 0 Liveness: http-get http://:8080/live delay=3s timeout=1s period=3s #success=1 #failure=3 Readiness: http-get http://:8080/ready delay=0s timeout=1s period=10s #success=1 #failure=3 Environment: API_SERVER_QPS: 400 API_SERVER_QPS_BURST: 500 PROMETHEUS_EXPORTER: true STACKDRIVER_EXPORTER: false GCP_PROJECT_ID:
STACKDRIVER_LABELS:
DISABLE_MTLS: false DISABLE_TLS: false REMOTE_ALLOCATION_TIMEOUT: 10s TOTAL_REMOTE_ALLOCATION_TIMEOUT: 30s POD_NAME: agones-allocator-657975759d-tnshh (v1:metadata.name) POD_NAMESPACE: agones-system (v1:metadata.namespace) CONTAINER_NAME: agones-allocator LOG_LEVEL: info FEATURE_GATES:
Mounts: /home/allocator/client-ca from client-ca (ro) /home/allocator/tls from tls (ro) /var/run/secrets/kubernetes.io/serviceaccount from agones-allocator-token-cfs8w (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: tls: Type: Secret (a volume populated by a Secret) SecretName: allocator-tls Optional: false client-ca: Type: Secret (a volume populated by a Secret) SecretName: allocator-client-ca Optional: false agones-allocator-token-cfs8w: Type: Secret (a volume populated by a Secret) SecretName: agones-allocator-token-cfs8w Optional: false QoS Class: BestEffort Node-Selectors: Tolerations: agones.dev/agones-system=true:NoExecute node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message

Normal Scheduled 3m44s default-scheduler Successfully assigned agones-system/agones-allocator-657975759d-tnshh to cls-5ke8ljh3-virtual-kubelet-subnet-ejptgbn5-0 Normal Pulling 3m35s eklet Pulling image "registry.cn-hangzhou.aliyuncs.com/kaiiak/agones-allocator:1.12.0" Normal Started 3m35s eklet pod sandbox started Normal Pulled 3m35s eklet Successfully pulled image "registry.cn-hangzhou.aliyuncs.com/kaiiak/agones-allocator:1.12.0" Warning Failed 3m35s eklet Error: failed to generate container "741c42936ad63c5a83463372991c6281331d2e8d1e529243b79bd662100ee425" spec: failed to set OCI bind mounts [&Mount{ContainerPath:/dev/shm,HostPath:/run/containerd/io.containerd.grpc.v1.cri/sandboxes/5b179fc56c9f14f064247301750b302ed64301839f6a0342b183a6be41c3b8bd/shm,Readonly:false,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,} &Mount{ContainerPath:/etc/hostname,HostPath:/var/lib/containerd/io.containerd.grpc.v1.cri/sandboxes/5b179fc56c9f14f064247301750b302ed64301839f6a0342b183a6be41c3b8bd/hostname,Readonly:false,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,} &Mount{ContainerPath:/etc/hosts,HostPath:/var/lib/containerd/io.containerd.grpc.v1.cri/sandboxes/5b179fc56c9f14f064247301750b302ed64301839f6a0342b183a6be41c3b8bd/hosts,Readonly:false,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,} &Mount{ContainerPath:/etc/resolv.conf,HostPath:/var/lib/containerd/io.containerd.grpc.v1.cri/sandboxes/5b179fc56c9f14f064247301750b302ed64301839f6a0342b183a6be41c3b8bd/resolv.conf,Readonly:false,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,} &Mount{ContainerPath:/home/allocator/client-ca,HostPath:/kata/vk-cri/eks-rfvgzkgp/volumes/secrets/Opaque/client-ca,Readonly:true,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,} &Mount{ContainerPath:/home/allocator/tls,HostPath:,Readonly:true,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,} &Mount{ContainerPath:/var/run/secrets/kubernetes.io/serviceaccount,HostPath:/kata/vk-cri/eks-rfvgzkgp/volumes/secrets/kubernetes.io/service-account-token/agones-allocator-token-cfs8w,Readonly:true,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,}]: failed to mkdir "": mkdir : no such file or directory Normal Created 3m34s eklet Sync volumes Warning Failed 3m21s eklet Error: failed to generate container "ee827a8b26bb9d36e9557f3e65f34d76d5ff2e3d543b142a400c8d7537af0c8d" spec: failed to set OCI bind mounts [&Mount{ContainerPath:/dev/shm,HostPath:/run/containerd/io.containerd.grpc.v1.cri/sandboxes/5b179fc56c9f14f064247301750b302ed64301839f6a0342b183a6be41c3b8bd/shm,Readonly:false,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,} &Mount{ContainerPath:/etc/hostname,HostPath:/var/lib/containerd/io.containerd.grpc.v1.cri/sandboxes/5b179fc56c9f14f064247301750b302ed64301839f6a0342b183a6be41c3b8bd/hostname,Readonly:false,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,} &Mount{ContainerPath:/etc/hosts,HostPath:/var/lib/containerd/io.containerd.grpc.v1.cri/sandboxes/5b179fc56c9f14f064247301750b302ed64301839f6a0342b183a6be41c3b8bd/hosts,Readonly:false,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,} &Mount{ContainerPath:/etc/resolv.conf,HostPath:/var/lib/containerd/io.containerd.grpc.v1.cri/sandboxes/5b179fc56c9f14f064247301750b302ed64301839f6a0342b183a6be41c3b8bd/resolv.conf,Readonly:false,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,} &Mount{ContainerPath:/home/allocator/client-ca,HostPath:/kata/vk-cri/eks-rfvgzkgp/volumes/secrets/Opaque/client-ca,Readonly:true,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,} &Mount{ContainerPath:/home/allocator/tls,HostPath:,Readonly:true,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,} &Mount{ContainerPath:/var/run/secrets/kubernetes.io/serviceaccount,HostPath:/kata/vk-cri/eks-rfvgzkgp/volumes/secrets/kubernetes.io/service-account-token/agones-allocator-token-cfs8w,Readonly:true,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,}]: failed to mkdir "": mkdir : no such file or directory Warning Failed 3m7s eklet Error: failed to generate container "be21eae072843da837cb93c9bb0cb45a62b839e96f933ddfbe776f49d1df516c" spec: failed to set OCI bind mounts [&Mount{ContainerPath:/dev/shm,HostPath:/run/containerd/io.containerd.grpc.v1.cri/sandboxes/5b179fc56c9f14f064247301750b302ed64301839f6a0342b183a6be41c3b8bd/shm,Readonly:false,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,} &Mount{ContainerPath:/etc/hostname,HostPath:/var/lib/containerd/io.containerd.grpc.v1.cri/sandboxes/5b179fc56c9f14f064247301750b302ed64301839f6a0342b183a6be41c3b8bd/hostname,Readonly:false,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,} &Mount{ContainerPath:/etc/hosts,HostPath:/var/lib/containerd/io.containerd.grpc.v1.cri/sandboxes/5b179fc56c9f14f064247301750b302ed64301839f6a0342b183a6be41c3b8bd/hosts,Readonly:false,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,} &Mount{ContainerPath:/etc/resolv.conf,HostPath:/var/lib/containerd/io.containerd.grpc.v1.cri/sandboxes/5b179fc56c9f14f064247301750b302ed64301839f6a0342b183a6be41c3b8bd/resolv.conf,Readonly:false,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,} &Mount{ContainerPath:/home/allocator/client-ca,HostPath:/kata/vk-cri/eks-rfvgzkgp/volumes/secrets/Opaque/client-ca,Readonly:true,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,} &Mount{ContainerPath:/home/allocator/tls,HostPath:,Readonly:true,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,} &Mount{ContainerPath:/var/run/secrets/kubernetes.io/serviceaccount,HostPath:/kata/vk-cri/eks-rfvgzkgp/volumes/secrets/kubernetes.io/service-account-token/agones-allocator-token-cfs8w,Readonly:true,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,}]: failed to mkdir "": mkdir : no such file or directory Warning Failed 2m44s eklet Error: failed to generate container "b0cab77cd11d3fe0a949e35928db4cfb7797f8b5cee000fcf849128ba0ccbdd0" spec: failed to set OCI bind mounts [&Mount{ContainerPath:/dev/shm,HostPath:/run/containerd/io.containerd.grpc.v1.cri/sandboxes/5b179fc56c9f14f064247301750b302ed64301839f6a0342b183a6be41c3b8bd/shm,Readonly:false,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,} &Mount{ContainerPath:/etc/hostname,HostPath:/var/lib/containerd/io.containerd.grpc.v1.cri/sandboxes/5b179fc56c9f14f064247301750b302ed64301839f6a0342b183a6be41c3b8bd/hostname,Readonly:false,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,} &Mount{ContainerPath:/etc/hosts,HostPath:/var/lib/containerd/io.containerd.grpc.v1.cri/sandboxes/5b179fc56c9f14f064247301750b302ed64301839f6a0342b183a6be41c3b8bd/hosts,Readonly:false,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,} &Mount{ContainerPath:/etc/resolv.conf,HostPath:/var/lib/containerd/io.containerd.grpc.v1.cri/sandboxes/5b179fc56c9f14f064247301750b302ed64301839f6a0342b183a6be41c3b8bd/resolv.conf,Readonly:false,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,} &Mount{ContainerPath:/home/allocator/client-ca,HostPath:/kata/vk-cri/eks-rfvgzkgp/volumes/secrets/Opaque/client-ca,Readonly:true,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,} &Mount{ContainerPath:/home/allocator/tls,HostPath:,Readonly:true,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,} &Mount{ContainerPath:/var/run/secrets/kubernetes.io/serviceaccount,HostPath:/kata/vk-cri/eks-rfvgzkgp/volumes/secrets/kubernetes.io/service-account-token/agones-allocator-token-cfs8w,Readonly:true,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,}]: failed to mkdir "": mkdir : no such file or directory Warning Failed 2m1s eklet Error: failed to generate container "2c844248d91f10d55a949a4954ecca64523d6fe6124c82ad7736ba96a5513cc9" spec: failed to set OCI bind mounts [&Mount{ContainerPath:/dev/shm,HostPath:/run/containerd/io.containerd.grpc.v1.cri/sandboxes/5b179fc56c9f14f064247301750b302ed64301839f6a0342b183a6be41c3b8bd/shm,Readonly:false,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,} &Mount{ContainerPath:/etc/hostname,HostPath:/var/lib/containerd/io.containerd.grpc.v1.cri/sandboxes/5b179fc56c9f14f064247301750b302ed64301839f6a0342b183a6be41c3b8bd/hostname,Readonly:false,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,} &Mount{ContainerPath:/etc/hosts,HostPath:/var/lib/containerd/io.containerd.grpc.v1.cri/sandboxes/5b179fc56c9f14f064247301750b302ed64301839f6a0342b183a6be41c3b8bd/hosts,Readonly:false,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,} &Mount{ContainerPath:/etc/resolv.conf,HostPath:/var/lib/containerd/io.containerd.grpc.v1.cri/sandboxes/5b179fc56c9f14f064247301750b302ed64301839f6a0342b183a6be41c3b8bd/resolv.conf,Readonly:false,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,} &Mount{ContainerPath:/home/allocator/client-ca,HostPath:/kata/vk-cri/eks-rfvgzkgp/volumes/secrets/Opaque/client-ca,Readonly:true,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,} &Mount{ContainerPath:/home/allocator/tls,HostPath:,Readonly:true,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,} &Mount{ContainerPath:/var/run/secrets/kubernetes.io/serviceaccount,HostPath:/kata/vk-cri/eks-rfvgzkgp/volumes/secrets/kubernetes.io/service-account-token/agones-allocator-token-cfs8w,Readonly:true,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,}]: failed to mkdir "": mkdir : no such file or directory Normal Pulled 30s (x5 over 3m21s) eklet Container image "registry.cn-hangzhou.aliyuncs.com/kaiiak/agones-allocator:1.12.0" already present on machine Warning Failed 30s eklet Error: failed to generate container "194ab8bd9bdc923767e5d348ff25beed51838c1c808c408b638537706cd43fd6" spec: failed to set OCI bind mounts [&Mount{ContainerPath:/dev/shm,HostPath:/run/containerd/io.containerd.grpc.v1.cri/sandboxes/5b179fc56c9f14f064247301750b302ed64301839f6a0342b183a6be41c3b8bd/shm,Readonly:false,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,} &Mount{ContainerPath:/etc/hostname,HostPath:/var/lib/containerd/io.containerd.grpc.v1.cri/sandboxes/5b179fc56c9f14f064247301750b302ed64301839f6a0342b183a6be41c3b8bd/hostname,Readonly:false,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,} &Mount{ContainerPath:/etc/hosts,HostPath:/var/lib/containerd/io.containerd.grpc.v1.cri/sandboxes/5b179fc56c9f14f064247301750b302ed64301839f6a0342b183a6be41c3b8bd/hosts,Readonly:false,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,} &Mount{ContainerPath:/etc/resolv.conf,HostPath:/var/lib/containerd/io.containerd.grpc.v1.cri/sandboxes/5b179fc56c9f14f064247301750b302ed64301839f6a0342b183a6be41c3b8bd/resolv.conf,Readonly:false,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,} &Mount{ContainerPath:/home/allocator/client-ca,HostPath:/kata/vk-cri/eks-rfvgzkgp/volumes/secrets/Opaque/client-ca,Readonly:true,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,} &Mount{ContainerPath:/home/allocator/tls,HostPath:,Readonly:true,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,} &Mount{ContainerPath:/var/run/secrets/kubernetes.io/serviceaccount,HostPath:/kata/vk-cri/eks-rfvgzkgp/volumes/secrets/kubernetes.io/service-account-token/agones-allocator-token-cfs8w,Readonly:true,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,}]: failed to mkdir "": mkdir : no such file or directory

markmandel commented 3 years ago

If you install on a Kubernetes cluster other than Tecent (Tencent?) Kubernetes - does it work? Say even Minikube or Kind?

kaiiak commented 3 years ago

If you install on a Kubernetes cluster other than Tecent (Tencent?) Kubernetes - does it work? Say even Minikube or Kind?

I succeed in installing on other Kind Kubernetes cluster on Tencent cloud, agones worked. This is a bug of Tencent Auto Scaling Kubernetes cluster.

googleforgames / agones

Deploy agones failed in Tencent Auto Scaling Kubernetes cluster, failed to generate container agones-allocator #1997