search

kubernetes / minikube

Run Kubernetes locally
https://minikube.sigs.k8s.io/
Apache License 2.0
29.5k stars 4.89k forks source link

Resolved CrashLoopBackOff Issue (driver = docker) #19481

Open asusikai opened 2 months ago

asusikai commented 2 months ago

What Happened?

I was trying to use Jenkins with Helm, but the pod kept entering a CrashLoopBackOff state. The issue was resolved by changing the permissions of the directory /tmp/hostpath-provisioner/default/[helm-chart-dir] on the pod from 755 to 777.

I tried this on both Mac and Windows.

I believe this issue could be resolved by modifying the directory creation permissions from 0755 to 0777 in the following file:

https://github.com/kubernetes/minikube/blob/master/cmd/storage-provisioner/main.go

Attach the log file

dongjun@andongjun-ui-MacBookAir ~ % helm repo add jenkinsci https://charts.jenkins.io/ "jenkinsci" has been added to your repositories dongjun@andongjun-ui-MacBookAir ~ % helm install my-jenkins jenkinsci/jenkins --version 5.5.6 NAME: my-jenkins LAST DEPLOYED: Mon Aug 19 16:29:37 2024 NAMESPACE: default STATUS: deployed REVISION: 1 NOTES:

  1. Get your 'admin' user password by running: kubectl exec --namespace default -it svc/my-jenkins -c jenkins -- /bin/cat /run/secrets/additional/chart-admin-password && echo

  2. Get the Jenkins URL to visit by running these commands in the same shell: echo http://127.0.0.1:8080 kubectl --namespace default port-forward svc/my-jenkins 8080:8080

  3. Login with the password from step 1 and the username: admin

  4. Configure security realm and authorization strategy

  5. Use Jenkins Configuration as Code by specifying configScripts in your values.yaml file, see documentation: http://127.0.0.1:8080/configuration-as-code and examples: https://github.com/jenkinsci/configuration-as-code-plugin/tree/master/demos

For more information on running Jenkins on Kubernetes, visit: https://cloud.google.com/solutions/jenkins-on-container-engine

For more information about Jenkins Configuration as Code, visit: https://jenkins.io/projects/jcasc/

NOTE: Consider using a custom image with pre-installed plugins dongjun@andongjun-ui-MacBookAir ~ % kubectl get nodes NAME STATUS ROLES AGE VERSION kube-test Ready control-plane 15m v1.28.1 kube-test-m02 Ready 14m v1.28.1 kube-test-m03 Ready 14m v1.28.1 kube-test-m04 Ready 14m v1.28.1 dongjun@andongjun-ui-MacBookAir ~ % kubectl get pods NAME READY STATUS RESTARTS AGE my-jenkins-0 0/2 Init:1/2 0 9s dongjun@andongjun-ui-MacBookAir ~ % kubectl get pods NAME READY STATUS RESTARTS AGE my-jenkins-0 0/2 Init:1/2 0 11s dongjun@andongjun-ui-MacBookAir ~ % watch -n1 kubectl get pods zsh: command not found: watch dongjun@andongjun-ui-MacBookAir ~ % kubectl get pods
NAME READY STATUS RESTARTS AGE my-jenkins-0 0/2 Init:CrashLoopBackOff 1 (2s ago) 23s dongjun@andongjun-ui-MacBookAir ~ % kubectl get pods NAME READY STATUS RESTARTS AGE my-jenkins-0 0/2 Init:CrashLoopBackOff 1 (4s ago) 25s dongjun@andongjun-ui-MacBookAir ~ % kubectl get pods NAME READY STATUS RESTARTS AGE my-jenkins-0 0/2 Init:CrashLoopBackOff 1 (7s ago) 28s dongjun@andongjun-ui-MacBookAir ~ % kubectl get pods NAME READY STATUS RESTARTS AGE my-jenkins-0 0/2 Init:CrashLoopBackOff 4 (61s ago) 2m54s dongjun@andongjun-ui-MacBookAir ~ % kubectl describe pod my-jenikns-0 Error from server (NotFound): pods "my-jenikns-0" not found dongjun@andongjun-ui-MacBookAir ~ % kubectl describe pod my-jenkins-0 Name: my-jenkins-0 Namespace: default Priority: 0 Service Account: my-jenkins Node: kube-test-m04/192.168.49.5 Start Time: Mon, 19 Aug 2024 16:29:37 +0900 Labels: app.kubernetes.io/component=jenkins-controller app.kubernetes.io/instance=my-jenkins app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=jenkins apps.kubernetes.io/pod-index=0 controller-revision-hash=my-jenkins-7dd4db4cd5 statefulset.kubernetes.io/pod-name=my-jenkins-0 Annotations: checksum/config: bf5c41547ad32f6807e04f82058cd4540d545e6b947509422beff6f7e4491ff0 Status: Pending IP: 10.244.3.3 IPs: IP: 10.244.3.3 Controlled By: StatefulSet/my-jenkins Init Containers: config-reload-init: Container ID: docker://964d6d4718be0f8c99663902e8f481905c40fb491d1aff83964784de137f2484 Image: docker.io/kiwigrid/k8s-sidecar:1.27.5 Image ID: docker-pullable://kiwigrid/k8s-sidecar@sha256:1fc88232e223a6977c626510372a74ca1f73af073e3c361719ccf02f223c8a12 Port: Host Port: State: Terminated Reason: Completed Exit Code: 0 Started: Mon, 19 Aug 2024 16:29:43 +0900 Finished: Mon, 19 Aug 2024 16:29:45 +0900 Ready: True Restart Count: 0 Environment: POD_NAME: my-jenkins-0 (v1:metadata.name) LABEL: my-jenkins-jenkins-config FOLDER: /var/jenkins_home/casc_configs NAMESPACE: default METHOD: LIST Mounts: /var/jenkins_home from jenkins-home (rw) /var/jenkins_home/casc_configs from sc-config-volume (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-ckzfj (ro) init: Container ID: docker://cfbfab6b35185d24a1934d95d65c6634046ebd06a30ae28c340a934ad2db4a43 Image: docker.io/jenkins/jenkins:2.462.1-jdk17 Image ID: docker-pullable://jenkins/jenkins@sha256:2bafb1fb2d6489bccadc1b7c172937e9b56a888ed77e625a4ebe59a6b038221e Port: Host Port: Command: sh /var/jenkins_config/apply_config.sh State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Error Exit Code: 2 Started: Mon, 19 Aug 2024 16:37:13 +0900 Finished: Mon, 19 Aug 2024 16:37:13 +0900 Ready: False Restart Count: 5 Limits: cpu: 2 memory: 4Gi Requests: cpu: 50m memory: 256Mi Environment: Mounts: /tmp from tmp-volume (rw) /usr/share/jenkins/ref/plugins from plugins (rw) /var/jenkins_config from jenkins-config (rw) /var/jenkins_home from jenkins-home (rw) /var/jenkins_plugins from plugin-dir (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-ckzfj (ro) Containers: jenkins: Container ID:
Image: docker.io/jenkins/jenkins:2.462.1-jdk17 Image ID:
Ports: 8080/TCP, 50000/TCP Host Ports: 0/TCP, 0/TCP Args: --httpPort=8080 State: Waiting Reason: PodInitializing Ready: False Restart Count: 0 Limits: cpu: 2 memory: 4Gi Requests: cpu: 50m memory: 256Mi Liveness: http-get http://:http/login delay=0s timeout=5s period=10s #success=1 #failure=5 Readiness: http-get http://:http/login delay=0s timeout=5s period=10s #success=1 #failure=3 Startup: http-get http://:http/login delay=0s timeout=5s period=10s #success=1 #failure=12 Environment: SECRETS: /run/secrets/additional POD_NAME: my-jenkins-0 (v1:metadata.name) JAVA_OPTS: -Dcasc.reload.token=$(POD_NAME) JENKINS_OPTS: --webroot=/var/jenkins_cache/war JENKINS_SLAVE_AGENT_PORT: 50000 CASC_JENKINS_CONFIG: /var/jenkins_home/casc_configs Mounts: /run/secrets/additional from jenkins-secrets (ro) /tmp from tmp-volume (rw) /usr/share/jenkins/ref/plugins/ from plugin-dir (rw) /var/jenkins_cache from jenkins-cache (rw) /var/jenkins_config from jenkins-config (ro) /var/jenkins_home from jenkins-home (rw) /var/jenkins_home/casc_configs from sc-config-volume (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-ckzfj (ro) config-reload: Container ID:
Image: docker.io/kiwigrid/k8s-sidecar:1.27.5 Image ID:
Port: Host Port: State: Waiting Reason: PodInitializing Ready: False Restart Count: 0 Environment: POD_NAME: my-jenkins-0 (v1:metadata.name) LABEL: my-jenkins-jenkins-config FOLDER: /var/jenkins_home/casc_configs NAMESPACE: default REQ_URL: http://localhost:8080/reload-configuration-as-code/?casc-reload-token=$(POD_NAME) REQ_METHOD: POST REQ_RETRY_CONNECT: 10 Mounts: /var/jenkins_home from jenkins-home (rw) /var/jenkins_home/casc_configs from sc-config-volume (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-ckzfj (ro) Conditions: Type Status Initialized False Ready False ContainersReady False PodScheduled True Volumes: plugins: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium:
SizeLimit: jenkins-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: my-jenkins Optional: false plugin-dir: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium:
SizeLimit: jenkins-secrets: Type: Projected (a volume that contains injected data from multiple sources) SecretName: my-jenkins SecretOptionalName: jenkins-cache: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium:
SizeLimit: jenkins-home: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: my-jenkins ReadOnly: false sc-config-volume: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium:
SizeLimit: tmp-volume: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium:
SizeLimit: kube-api-access-ckzfj: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message

Normal Scheduled 7m50s default-scheduler Successfully assigned default/my-jenkins-0 to kube-test-m04 Normal Pulling 7m49s kubelet Pulling image "docker.io/kiwigrid/k8s-sidecar:1.27.5" Normal Pulled 7m44s kubelet Successfully pulled image "docker.io/kiwigrid/k8s-sidecar:1.27.5" in 5.707s (5.707s including waiting) Normal Created 7m44s kubelet Created container config-reload-init Normal Started 7m44s kubelet Started container config-reload-init Normal Pulled 7m32s kubelet Successfully pulled image "docker.io/jenkins/jenkins:2.462.1-jdk17" in 10.113s (10.113s including waiting) Normal Pulled 7m29s kubelet Successfully pulled image "docker.io/jenkins/jenkins:2.462.1-jdk17" in 1.827s (1.827s including waiting) Normal Pulled 7m11s kubelet Successfully pulled image "docker.io/jenkins/jenkins:2.462.1-jdk17" in 1.898s (1.898s including waiting) Normal Pulling 6m42s (x4 over 7m42s) kubelet Pulling image "docker.io/jenkins/jenkins:2.462.1-jdk17" Normal Created 6m41s (x4 over 7m32s) kubelet Created container init Normal Pulled 6m41s kubelet Successfully pulled image "docker.io/jenkins/jenkins:2.462.1-jdk17" in 1.798s (1.798s including waiting) Normal Started 6m40s (x4 over 7m32s) kubelet Started container init Warning BackOff 6m40s (x5 over 7m29s) kubelet Back-off restarting failed container init in pod my-jenkins-0_default(d7efc991-9c1e-4321-ad3b-8a88fa283b08)

Operating System

macOS (Default)

Driver

Docker

asusikai commented 2 months ago

@gnufied could you please add ev/CrashLoopBackoff label to this issue? thx