search

apecloud / kubeblocks

KubeBlocks is an open-source control plane software that runs and manages databases, message queues and other stateful applications on K8s.
https://kubeblocks.io
GNU Affero General Public License v3.0
2.22k stars 184 forks source link

[BUG] doris cluster component fe create failed on GKE #7215

Closed JashBook closed 5 months ago

JashBook commented 10 months ago

Describe the bug Error: failed to generate container "6ed13e4ae34749b117f8ddd25923ff47f09139f56ec68306976651d8014b3b5d" spec: failed to generate spec: failed to mkdir "/root/.kube/config": mkdir /root/.kube: read-only file system

To Reproduce Steps to reproduce the behavior:

  1. create cluster 
    apiVersion: apps.kubeblocks.io/v1alpha1
kind: Cluster
metadata:
name: doris-cluster
namespace: default
spec:
clusterDefinitionRef: doris
clusterVersionRef: doris-2.0.3
terminationPolicy: Delete
componentSpecs:
- name: doris-fe
  componentDefRef: doris-fe
  replicas: 3
  resources:
    requests:
      cpu: 100m
      memory: 0.5Gi
    limits:
      cpu: 100m
      memory: 0.5Gi
  volumeClaimTemplates:
    - name: data
      spec:
        storageClassName:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
- componentDefRef: doris-be
  name: doris-be
  replicas: 3
  resources:
    requests:
      cpu: 100m
      memory: 0.5Gi
    limits:
      cpu: 100m
      memory: 0.5Gi
  volumeClaimTemplates:
    - name: data
      spec:
        storageClassName:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
- componentDefRef: doris-cn
  name: doris-cn
  replicas: 3
  resources:
    requests:
      cpu: 100m
      memory: 0.5Gi
    limits:
      cpu: 100m
      memory: 0.5Gi

  2. See error

    kubectl get pod       
NAME                                              READY   STATUS                 RESTARTS          AGE
doris-cluster-doris-be-0                          1/1     Running                0                 3m44s
doris-cluster-doris-be-1                          1/1     Running                0                 3m44s
doris-cluster-doris-be-2                          1/1     Running                0                 3m44s
doris-cluster-doris-cn-0                          1/1     Running                0                 3m46s
doris-cluster-doris-cn-1                          1/1     Running                0                 3m46s
doris-cluster-doris-cn-2                          1/1     Running                0                 3m46s
doris-cluster-doris-fe-0                          2/3     CreateContainerError   0                 3m46s
doris-cluster-doris-fe-1                          2/3     CreateContainerError   0                 3m46s
doris-cluster-doris-fe-2                          2/3     CreateContainerError   0                 3m46s

    describe pod 

    kubectl describe pod doris-cluster-doris-fe-0  
Name:         doris-cluster-doris-fe-0
Namespace:    default
Priority:     0
Node:         gke-infracreate-gke-default-pool-3154c215-ht4s/10.10.0.66
Start Time:   Thu, 01 Feb 2024 18:50:38 +0800
Labels:       app.kubernetes.io/component=doris-fe
          app.kubernetes.io/instance=doris-cluster
          app.kubernetes.io/managed-by=kubeblocks
          app.kubernetes.io/name=doris
          app.kubernetes.io/version=
          apps.kubeblocks.io/component-name=doris-fe
          controller-revision-hash=doris-cluster-doris-fe-5dfcccc76b
          statefulset.kubernetes.io/pod-name=doris-cluster-doris-fe-0
Annotations:  apps.kubeblocks.io/component-replicas: 3
Status:       Pending
IP:           10.128.4.9
IPs:
IP:           10.128.4.9
Controlled By:  StatefulSet/doris-cluster-doris-fe
Containers:
doris-fe:
Container ID:   
Image:          registry.cn-hangzhou.aliyuncs.com/yq_doris/apache-doris:2.0.3-fe
Image ID:       
Ports:          8030/TCP, 9020/TCP, 9030/TCP, 9010/TCP
Host Ports:     0/TCP, 0/TCP, 0/TCP, 0/TCP
State:          Waiting
  Reason:       CreateContainerError
Ready:          False
Restart Count:  0
Limits:
  cpu:     100m
  memory:  512Mi
Requests:
  cpu:     100m
  memory:  512Mi
Environment Variables from:
  doris-cluster-doris-fe-env      ConfigMap  Optional: false
  doris-cluster-doris-fe-rsm-env  ConfigMap  Optional: false
Environment:
  KB_POD_NAME:          doris-cluster-doris-fe-0 (v1:metadata.name)
  KB_POD_UID:            (v1:metadata.uid)
  KB_NAMESPACE:         default (v1:metadata.namespace)
  KB_SA_NAME:            (v1:spec.serviceAccountName)
  KB_NODENAME:           (v1:spec.nodeName)
  KB_HOST_IP:            (v1:status.hostIP)
  KB_POD_IP:             (v1:status.podIP)
  KB_POD_IPS:            (v1:status.podIPs)
  KB_HOSTIP:             (v1:status.hostIP)
  KB_PODIP:              (v1:status.podIP)
  KB_PODIPS:             (v1:status.podIPs)
  KB_POD_FQDN:          $(KB_POD_NAME).doris-cluster-doris-fe-headless.$(KB_NAMESPACE).svc
  APP_NAMESPACE:        $(KB_NAMESPACE)
  BUILD_TYPE:           k8s
  FE_INIT_NUMBER:       3
  CN_SERVICE:           $(KB_CLUSTER_NAME)-doris-cn-headless
  CN_STATEFULSET:       $(KB_CLUSTER_NAME)-doris-cn
  BE_SERVICE:           $(KB_CLUSTER_NAME)-doris-be-headless
  BE_STATEFULSET:       $(KB_CLUSTER_NAME)-doris-be
  FE_SERVICE:           $(KB_CLUSTER_NAME)-doris-fe-headless
  FE_STATEFULSET:       $(KB_CLUSTER_NAME)-doris-fe
  MYSQL_ROOT_PASSWORD:  <set to the key 'password' in secret 'doris-cluster-conn-credential'>  Optional: false
Mounts:
  /opt/apache-doris/fe/bin/mysql_root.sh from update-root (rw,path="mysql_root.sh")
  /opt/apache-doris/fe/conf from conf (rw)
  /opt/apache-doris/fe/doris-meta from data (rw)
  /root/.kube/config from kube (ro)
  /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-8bl6m (ro)
mysql-root-script:
Container ID:  containerd://1c688f80f9ce3db049be5ce6ba5719ad1830636c28cd990a19c8938addceae23
Image:         registry.cn-hangzhou.aliyuncs.com/yq_doris/apache-doris:2.0.3-fe
Image ID:      registry.cn-hangzhou.aliyuncs.com/yq_doris/apache-doris@sha256:65618f340a647eb0d781e7a4052992a42d52b07d0e15c2143aed7a25880da624
Port:          <none>
Host Port:     <none>
Command:
  sh
  -c
  /opt/apache-doris/fe/bin/mysql_root.sh > /opt/apache-doris/fe/bin/update_root.log 2>&1 &
  while true; do sleep 1; done

State:          Running
  Started:      Thu, 01 Feb 2024 18:50:48 +0800
Ready:          True
Restart Count:  0
Limits:
  cpu:     1
  memory:  2G
Requests:
  cpu:     500m
  memory:  1G
Environment Variables from:
  doris-cluster-doris-fe-env      ConfigMap  Optional: false
  doris-cluster-doris-fe-rsm-env  ConfigMap  Optional: false
Environment:
  KB_POD_NAME:          doris-cluster-doris-fe-0 (v1:metadata.name)
  KB_POD_UID:            (v1:metadata.uid)
  KB_NAMESPACE:         default (v1:metadata.namespace)
  KB_SA_NAME:            (v1:spec.serviceAccountName)
  KB_NODENAME:           (v1:spec.nodeName)
  KB_HOST_IP:            (v1:status.hostIP)
  KB_POD_IP:             (v1:status.podIP)
  KB_POD_IPS:            (v1:status.podIPs)
  KB_HOSTIP:             (v1:status.hostIP)
  KB_PODIP:              (v1:status.podIP)
  KB_PODIPS:             (v1:status.podIPs)
  KB_POD_FQDN:          $(KB_POD_NAME).doris-cluster-doris-fe-headless.$(KB_NAMESPACE).svc
  MYSQL_ROOT_PASSWORD:  <set to the key 'password' in secret 'doris-cluster-conn-credential'>  Optional: false
Mounts:
  /opt/apache-doris/fe/bin/mysql_root.sh from update-root (rw,path="mysql_root.sh")
  /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-8bl6m (ro)
config-manager:
Container ID:  containerd://99c3bdc3da5939684708289bc2b1384a588591e1e25212b16d19cbb9faf2d29d
Image:         docker.io/apecloud/kubeblocks-tools:0.8.2-beta.4
Image ID:      docker.io/apecloud/kubeblocks-tools@sha256:d4ced3a9c566b41b8df8c3eba89a3f8c5625fd6a6d8095561b13562f49a34b40
Port:          <none>
Host Port:     <none>
Command:
  env
Args:
  PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$(TOOLS_PATH)
  /bin/reloader
  --log-level
  info
  --operator-update-enable
  --tcp
  9901
  --config
  /opt/config-manager/config-manager.yaml
State:          Running
  Started:      Thu, 01 Feb 2024 18:50:48 +0800
Ready:          True
Restart Count:  0
Limits:
  cpu:     0
  memory:  0
Requests:
  cpu:     0
  memory:  0
Environment Variables from:
  doris-cluster-doris-fe-env      ConfigMap  Optional: false
  doris-cluster-doris-fe-rsm-env  ConfigMap  Optional: false
Environment:
  KB_POD_NAME:            doris-cluster-doris-fe-0 (v1:metadata.name)
  KB_POD_UID:              (v1:metadata.uid)
  KB_NAMESPACE:           default (v1:metadata.namespace)
  KB_SA_NAME:              (v1:spec.serviceAccountName)
  KB_NODENAME:             (v1:spec.nodeName)
  KB_HOST_IP:              (v1:status.hostIP)
  KB_POD_IP:               (v1:status.podIP)
  KB_POD_IPS:              (v1:status.podIPs)
  KB_HOSTIP:               (v1:status.hostIP)
  KB_PODIP:                (v1:status.podIP)
  KB_PODIPS:               (v1:status.podIPs)
  KB_POD_FQDN:            $(KB_POD_NAME).doris-cluster-doris-fe-headless.$(KB_NAMESPACE).svc
  CONFIG_MANAGER_POD_IP:   (v1:status.podIP)
  DB_TYPE:                doris-fe
  TOOLS_PATH:             /opt/kb-tools/reload/doris-fe-conf-configuration:/opt/config-manager
Mounts:
  /opt/apache-doris/fe/conf from conf (rw)
  /opt/config-manager from config-manager-config (rw)
  /opt/kb-tools/reload/doris-fe-conf-configuration from cm-script-doris-fe-conf-configuration (rw)
  /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-8bl6m (ro)
Conditions:
Type              Status
Initialized       True 
Ready             False 
ContainersReady   False 
PodScheduled      True 
Volumes:
data:
Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName:  data-doris-cluster-doris-fe-0
ReadOnly:   false
kube:
Type:          HostPath (bare host directory volume)
Path:          /root/.kube/config
HostPathType:  
conf:
Type:      ConfigMap (a volume populated by a ConfigMap)
Name:      doris-cluster-doris-fe-doris-fe-conf-configuration
Optional:  false
update-root:
Type:      ConfigMap (a volume populated by a ConfigMap)
Name:      doris-cluster-doris-fe-doris-fe-root-script
Optional:  false
cm-script-doris-fe-conf-configuration:
Type:      ConfigMap (a volume populated by a ConfigMap)
Name:      sidecar-doris-tools-script-doris-cluster
Optional:  false
config-manager-config:
Type:      ConfigMap (a volume populated by a ConfigMap)
Name:      sidecar-doris-cluster-doris-fe-config-manager-config
Optional:  false
kube-api-access-8bl6m:
Type:                    Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds:  3607
ConfigMapName:           kube-root-ca.crt
ConfigMapOptional:       <nil>
DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 kb-data=true:NoSchedule
                         node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                         node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type     Reason                  Age                     From                     Message
----     ------                  ----                    ----                     -------
Normal   Scheduled               4m27s                   default-scheduler        Successfully assigned default/doris-cluster-doris-fe-0 to gke-infracreate-gke-default-pool-3154c215-ht4s
Normal   SuccessfulAttachVolume  4m20s                   attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-40d97094-0e6d-4f82-89b5-b7eb526481c7"
Warning  Failed                  4m18s                   kubelet                  Error: failed to generate container "1b7fd46e44134a89e944c1859882e96db772d75ffaf14d55c134762e67c47f54" spec: failed to generate spec: failed to mkdir "/root/.kube/config": mkdir /root/.kube: read-only file system
Normal   Pulled                  4m18s                   kubelet                  Container image "registry.cn-hangzhou.aliyuncs.com/yq_doris/apache-doris:2.0.3-fe" already present on machine
Normal   Created                 4m17s                   kubelet                  Created container config-manager
Normal   Created                 4m17s                   kubelet                  Created container mysql-root-script
Normal   Started                 4m17s                   kubelet                  Started container mysql-root-script
Normal   Pulled                  4m17s                   kubelet                  Container image "docker.io/apecloud/kubeblocks-tools:0.8.2-beta.4" already present on machine
Normal   Started                 4m17s                   kubelet                  Started container config-manager
Warning  Failed                  4m16s                   kubelet                  Error: failed to generate container "6f841e1c059030b532e8af4b61e4833226a95206faf365bc50fbec7551934275" spec: failed to generate spec: failed to mkdir "/root/.kube/config": mkdir /root/.kube: read-only file system
Warning  Failed                  4m15s                   kubelet                  Error: failed to generate container "5fe445da971ca1f9084b474746624e70063035253764a8473d17f85afca1978b" spec: failed to generate spec: failed to mkdir "/root/.kube/config": mkdir /root/.kube: read-only file system
Warning  Failed                  4m3s                    kubelet                  Error: failed to generate container "e4871b1c58f89d657f57f145df1a0580509424657cf5ee64610c39a9d62907ea" spec: failed to generate spec: failed to mkdir "/root/.kube/config": mkdir /root/.kube: read-only file system
Warning  Failed                  3m50s                   kubelet                  Error: failed to generate container "7cdab9cfa2587df08889fb1440374da0ae7d9341b43b4c5090173c15af730400" spec: failed to generate spec: failed to mkdir "/root/.kube/config": mkdir /root/.kube: read-only file system
Warning  Failed                  3m37s                   kubelet                  Error: failed to generate container "ea16e5febc287d27f44a343b4cc900e998c9f3509cd0bde65cc5dc5d25c0c087" spec: failed to generate spec: failed to mkdir "/root/.kube/config": mkdir /root/.kube: read-only file system
Warning  Failed                  3m24s                   kubelet                  Error: failed to generate container "dc76c483b6ed365202eaa8d83c10e2763fa88b10bacd2c95eefa8a03306c53c4" spec: failed to generate spec: failed to mkdir "/root/.kube/config": mkdir /root/.kube: read-only file system
Warning  Failed                  3m9s                    kubelet                  Error: failed to generate container "c61a34d9e508fee53a9caacb7745d490e48d996d8648ea8fc61ed1617be1c20c" spec: failed to generate spec: failed to mkdir "/root/.kube/config": mkdir /root/.kube: read-only file system
Warning  Failed                  2m58s                   kubelet                  Error: failed to generate container "6ed13e4ae34749b117f8ddd25923ff47f09139f56ec68306976651d8014b3b5d" spec: failed to generate spec: failed to mkdir "/root/.kube/config": mkdir /root/.kube: read-only file system
Normal   Pulled                  2m44s (x10 over 4m18s)  kubelet                  Container image "registry.cn-hangzhou.aliyuncs.com/yq_doris/apache-doris:2.0.3-fe" already present on machine
➜  ~

Expected behavior A clear and concise description of what you expected to happen.

Screenshots If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

Additional context Add any other context about the problem here.

ahjing99 commented 5 months ago

still failed on 0.9.0-beta.41

shanshanying commented 5 months ago

Doris FrontEnd tries to mount hostpath to access K8s resources. Optimized the addon and picked to Release 0.8, should work now.