Add error logs about the problem here (operator logs and Kubernetes events).
Events from describe pod:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 9s default-scheduler Successfully assigned nampespace-gitlab/jenkins-jenkins to kube-node-k98kjpr6
Normal SuccessfulAttachVolume 9s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-80969eac-2a73-4090-b2a8-bf06c4221470"
Normal Pulling 6s kubelet Pulling image "jenkins/jenkins:2.414.1-lts"
Normal Pulled 5s kubelet Successfully pulled image "jenkins/jenkins:2.414.1-lts" in 824.642378ms
Normal Created 5s kubelet Created container jenkins-master
Normal Started 5s kubelet Started container jenkins-master
Normal Pulled 5s kubelet Container image "quay.io/jenkins-kubernetes-operator/backup-pvc:v0.2.6" already present on machine
Normal Created 5s kubelet Created container backup
Normal Started 5s kubelet Started container backup
Normal Killing 5s kubelet Stopping container jenkins-master
Normal Killing 5s kubelet Stopping container backup
Jenkins-master logs (tail from the logs):
Returning cached value for: update-center-2.414.1
Returning cached value for: experimental-update-center-2.414.1
Returning cached value for: plugin-versions
Done
+ echo 'Installing plugins required by user - end'
Installing plugins required by user - end
Running from: /usr/share/jenkins/jenkins.war
webroot: /var/lib/jenkins/war
2024-02-06 12:31:38.516+0000 [id=1] INFO winstone.Logger#logInternal: Beginning extraction from war file
2024-02-06 12:31:40.096+0000 [id=1] WARNING o.e.j.s.handler.ContextHandler#setContextPath: Empty contextPath
2024-02-06 12:31:40.215+0000 [id=1] INFO org.eclipse.jetty.server.Server#doStart: jetty-10.0.15; built: 2023-04-11T17:25:14.480Z; git: 68017dbd00236bb7e187330d7585a059610f661d; jvm 11.0.20+8
2024-02-06 12:31:40.895+0000 [id=1] INFO o.e.j.w.StandardDescriptorProcessor#visitServlet: NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet
2024-02-06 12:31:41.019+0000 [id=1] INFO o.e.j.s.s.DefaultSessionIdManager#doStart: Session workerName=node0
2024-02-06 12:31:42.228+0000 [id=1] INFO hudson.WebAppMain#contextInitialized: Jenkins home directory: /var/lib/jenkins found at: EnvVars.masterEnvVars.get("JENKINS_HOME")
jenkins backup logs:
Trimming to only 3 recent backups in preparation for new backup
Trimming to only 3 recent backups in preparation for new backup
Trimming to only 3 recent backups in preparation for new backup
operator logs
2024-02-06T14:20:05.422Z INFO cmd Git commit: a6e32fde
2024-02-06T14:20:05.422Z INFO cmd Go Version: go1.15.6
2024-02-06T14:20:05.422Z INFO cmd Go OS/Arch: linux/amd64
2024-02-06T14:20:05.422Z INFO cmd Watch namespace: namespace-gitlab
I0206 14:20:06.472842 1 request.go:655] Throttling request took 1.038320305s, request: GET:https://10.233.0.1:443/apis/vault.banzaicloud.com/v1alpha1?timeout=32s
2024-02-06T14:20:06.879Z INFO controller-runtime.metrics metrics server is starting to listen {"addr": "0.0.0.0:8383"}
2024-02-06T14:20:06.885Z INFO cmd starting manager
I0206 14:20:06.886076 1 leaderelection.go:243] attempting to acquire leader lease namespace-gitlab/c674355f.jenkins.io...
2024-02-06T14:20:06.886Z INFO controller-runtime.manager starting metrics server {"path": "/metrics"}
I0206 14:20:21.994302 1 leaderelection.go:253] successfully acquired lease namespace-gitlab/c674355f.jenkins.io
2024-02-06T14:20:21.994Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"ConfigMap","namespace":"namespace-gitlab","name":"c674355f.jenkins.io","uid":"bd3e1c6f-b3f9-40fc-8042-c2aa430cb850","apiVersion":"v1","resourceVersion":"121927203"}, "reason": "LeaderElection", "message": "jenkins-operator-77c57cb46c-6pwzj_6346bbc8-fe20-417c-84a1-ffe4d6fefb50 became leader"}
2024-02-06T14:20:21.994Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"Lease","namespace":"namespace-gitlab","name":"c674355f.jenkins.io","uid":"7a99fa4c-294f-479d-8667-7ace2b934890","apiVersion":"coordination.k8s.io/v1","resourceVersion":"121927205"}, "reason": "LeaderElection", "message": "jenkins-operator-77c57cb46c-6pwzj_6346bbc8-fe20-417c-84a1-ffe4d6fefb50 became leader"}
2024-02-06T14:20:21.994Z INFO controller-runtime.manager.controller.jenkins Starting EventSource {"reconciler group": "jenkins.io", "reconciler kind": "Jenkins", "source": "kind source: jenkins.io/v1alpha2, Kind=Jenkins"}
2024-02-06T14:20:22.095Z INFO controller-runtime.manager.controller.jenkins Starting EventSource {"reconciler group": "jenkins.io", "reconciler kind": "Jenkins", "source": "kind source: /, Kind="}
2024-02-06T14:20:22.195Z INFO controller-runtime.manager.controller.jenkins Starting EventSource {"reconciler group": "jenkins.io", "reconciler kind": "Jenkins", "source": "kind source: /, Kind="}
2024-02-06T14:20:22.296Z INFO controller-runtime.manager.controller.jenkins Starting EventSource {"reconciler group": "jenkins.io", "reconciler kind": "Jenkins", "source": "kind source: /, Kind="}
2024-02-06T14:20:22.397Z INFO controller-runtime.manager.controller.jenkins Starting EventSource {"reconciler group": "jenkins.io", "reconciler kind": "Jenkins", "source": "kind source: core/v1, Kind=Secret"}
2024-02-06T14:20:22.397Z INFO controller-runtime.manager.controller.jenkins Starting EventSource {"reconciler group": "jenkins.io", "reconciler kind": "Jenkins", "source": "kind source: core/v1, Kind=ConfigMap"}
2024-02-06T14:20:22.397Z INFO controller-runtime.manager.controller.jenkins Starting EventSource {"reconciler group": "jenkins.io", "reconciler kind": "Jenkins", "source": "kind source: jenkins.io/v1alpha2, Kind=Jenkins"}
2024-02-06T14:20:22.397Z INFO controller-runtime.manager.controller.jenkins Starting Controller {"reconciler group": "jenkins.io", "reconciler kind": "Jenkins"}
2024-02-06T14:20:22.397Z INFO controller-runtime.manager.controller.jenkins Starting workers {"reconciler group": "jenkins.io", "reconciler kind": "Jenkins", "worker count": 1}
2024-02-06T14:20:22.397Z INFO controller-jenkins *v1alpha2.Jenkins/jenkins was created {"cr": "jenkins"}
2024-02-06T14:20:22.397Z DEBUG controller-jenkins Reconciling Jenkins {"cr": "jenkins"}
2024-02-06T14:20:22.397Z INFO controller-jenkins Setting default Jenkins container command {"cr": "jenkins"}
2024-02-06T14:20:22.397Z INFO controller-jenkins Setting default Jenkins container JAVA_OPTS environment variable {"cr": "jenkins"}
2024-02-06T14:20:22.397Z INFO controller-jenkins Setting default operator plugins {"cr": "jenkins"}
2024-02-06T14:20:22.397Z INFO controller-jenkins Setting default Jenkins master service {"cr": "jenkins"}
2024-02-06T14:20:22.397Z INFO controller-jenkins Setting default Jenkins slave service {"cr": "jenkins"}
2024-02-06T14:20:22.436Z INFO controller-jenkins *v1alpha2.Jenkins/jenkins has been updated {"cr": "jenkins"}
2024-02-06T14:20:22.436Z DEBUG controller-jenkins Reconciling Jenkins {"cr": "jenkins"}
2024-02-06T14:20:22.557Z DEBUG controller-jenkins Operator credentials secret is present {"cr": "jenkins"}
2024-02-06T14:20:22.577Z DEBUG controller-jenkins Scripts config map is present {"cr": "jenkins"}
2024-02-06T14:20:22.592Z DEBUG controller-jenkins Init configuration config map is present {"cr": "jenkins"}
2024-02-06T14:20:22.639Z DEBUG controller-jenkins Base configuration config map is present {"cr": "jenkins"}
2024-02-06T14:20:22.639Z DEBUG controller-jenkins GroovyScripts Secret and ConfigMap added watched labels {"cr": "jenkins"}
2024-02-06T14:20:22.639Z DEBUG controller-jenkins ConfigurationAsCode Secret and ConfigMap added watched labels {"cr": "jenkins"}
2024-02-06T14:20:22.740Z DEBUG controller-jenkins createServiceAccount with annotations map[] {"cr": "jenkins"}
2024-02-06T14:20:22.805Z DEBUG controller-jenkins Service account, role and role binding are present {"cr": "jenkins"}
2024-02-06T14:20:22.905Z DEBUG controller-jenkins Extra role bindings are present {"cr": "jenkins"}
2024-02-06T14:20:23.028Z DEBUG controller-jenkins Jenkins HTTP Service is present {"cr": "jenkins"}
2024-02-06T14:20:23.069Z DEBUG controller-jenkins Jenkins slave Service is present {"cr": "jenkins"}
2024-02-06T14:20:23.069Z DEBUG controller-jenkins Kubernetes resources are present {"cr": "jenkins"}
2024-02-06T14:20:23.069Z INFO controller-jenkins Creating a new Jenkins Master Pod namespace-gitlab/jenkins-jenkins {"cr": "jenkins"}
2024-02-06T14:20:23.161Z DEBUG controller-jenkins Reconciling Jenkins {"cr": "jenkins"}
2024-02-06T14:20:23.161Z DEBUG controller-jenkins Operator credentials secret is present {"cr": "jenkins"}
2024-02-06T14:20:23.241Z DEBUG controller-jenkins Scripts config map is present {"cr": "jenkins"}
2024-02-06T14:20:23.295Z DEBUG controller-jenkins Init configuration config map is present {"cr": "jenkins"}
2024-02-06T14:20:23.372Z DEBUG controller-jenkins Base configuration config map is present {"cr": "jenkins"}
2024-02-06T14:20:23.372Z DEBUG controller-jenkins GroovyScripts Secret and ConfigMap added watched labels {"cr": "jenkins"}
2024-02-06T14:20:23.372Z DEBUG controller-jenkins ConfigurationAsCode Secret and ConfigMap added watched labels {"cr": "jenkins"}
2024-02-06T14:20:23.372Z DEBUG controller-jenkins createServiceAccount with annotations map[] {"cr": "jenkins"}
2024-02-06T14:20:23.522Z DEBUG controller-jenkins Service account, role and role binding are present {"cr": "jenkins"}
2024-02-06T14:20:23.522Z DEBUG controller-jenkins Extra role bindings are present {"cr": "jenkins"}
2024-02-06T14:20:23.532Z DEBUG controller-jenkins Jenkins HTTP Service is present {"cr": "jenkins"}
2024-02-06T14:20:23.547Z DEBUG controller-jenkins Jenkins slave Service is present {"cr": "jenkins"}
2024-02-06T14:20:23.547Z DEBUG controller-jenkins Kubernetes resources are present {"cr": "jenkins"}
2024-02-06T14:20:23.547Z INFO controller-jenkins Jenkins master pod restarted by operator: {"cr": "jenkins"}
2024-02-06T14:20:23.547Z INFO controller-jenkins Jenkins Operator version has changed, actual '' new 'v0.8.0' {"cr": "jenkins"}
2024-02-06T14:20:23.547Z INFO controller-jenkins Jenkins priorityClassName has changed, actual 'default-nonpreempting' required '' {"cr": "jenkins"}
2024-02-06T14:20:23.547Z INFO controller-jenkins Jenkins CR has been replaced {"cr": "jenkins"}
2024-02-06T14:20:23.628Z DEBUG controller-jenkins Reconciling Jenkins {"cr": "jenkins"}
2024-02-06T14:20:23.629Z DEBUG controller-jenkins Operator credentials secret is present {"cr": "jenkins"}
2024-02-06T14:20:23.711Z DEBUG controller-jenkins Scripts config map is present {"cr": "jenkins"}
2024-02-06T14:20:23.758Z DEBUG controller-jenkins Init configuration config map is present {"cr": "jenkins"}
2024-02-06T14:20:23.815Z DEBUG controller-jenkins Base configuration config map is present {"cr": "jenkins"}
2024-02-06T14:20:23.815Z DEBUG controller-jenkins GroovyScripts Secret and ConfigMap added watched labels {"cr": "jenkins"}
2024-02-06T14:20:23.815Z DEBUG controller-jenkins ConfigurationAsCode Secret and ConfigMap added watched labels {"cr": "jenkins"}
2024-02-06T14:20:23.815Z DEBUG controller-jenkins createServiceAccount with annotations map[] {"cr": "jenkins"}
2024-02-06T14:20:23.956Z DEBUG controller-jenkins Service account, role and role binding are present {"cr": "jenkins"}
2024-02-06T14:20:23.957Z DEBUG controller-jenkins Extra role bindings are present {"cr": "jenkins"}
2024-02-06T14:20:23.966Z DEBUG controller-jenkins Jenkins HTTP Service is present {"cr": "jenkins"}
2024-02-06T14:20:23.977Z DEBUG controller-jenkins Jenkins slave Service is present {"cr": "jenkins"}
2024-02-06T14:20:23.977Z DEBUG controller-jenkins Kubernetes resources are present {"cr": "jenkins"}
2024-02-06T14:20:23.977Z DEBUG controller-jenkins Jenkins master pod is present {"cr": "jenkins"}
2024-02-06T14:20:23.977Z DEBUG controller-jenkins Jenkins master pod is terminating {"cr": "jenkins"}
2024-02-06T14:20:23.977Z DEBUG controller-jenkins Reconciling Jenkins {"cr": "jenkins"}
2024-02-06T14:20:23.978Z DEBUG controller-jenkins Operator credentials secret is present {"cr": "jenkins"}
2024-02-06T14:20:24.038Z DEBUG controller-jenkins Scripts config map is present {"cr": "jenkins"}
2024-02-06T14:20:24.075Z DEBUG controller-jenkins Init configuration config map is present {"cr": "jenkins"}
2024-02-06T14:20:24.145Z DEBUG controller-jenkins Base configuration config map is present {"cr": "jenkins"}
2024-02-06T14:20:24.145Z DEBUG controller-jenkins GroovyScripts Secret and ConfigMap added watched labels {"cr": "jenkins"}
2024-02-06T14:20:24.145Z DEBUG controller-jenkins ConfigurationAsCode Secret and ConfigMap added watched labels {"cr": "jenkins"}
2024-02-06T14:20:24.145Z DEBUG controller-jenkins createServiceAccount with annotations map[] {"cr": "jenkins"}
2024-02-06T14:20:24.264Z DEBUG controller-jenkins Service account, role and role binding are present {"cr": "jenkins"}
2024-02-06T14:20:24.264Z DEBUG controller-jenkins Extra role bindings are present {"cr": "jenkins"}
2024-02-06T14:20:24.273Z DEBUG controller-jenkins Jenkins HTTP Service is present {"cr": "jenkins"}
2024-02-06T14:20:24.283Z DEBUG controller-jenkins Jenkins slave Service is present {"cr": "jenkins"}
2024-02-06T14:20:24.283Z DEBUG controller-jenkins Kubernetes resources are present {"cr": "jenkins"}
2024-02-06T14:20:24.283Z DEBUG controller-jenkins Jenkins master pod is present {"cr": "jenkins"}
2024-02-06T14:20:24.283Z DEBUG controller-jenkins Jenkins master pod is terminating {"cr": "jenkins"}
2024-02-06T14:20:25.753Z DEBUG controller-jenkins Reconciling Jenkins {"cr": "jenkins"}
2024-02-06T14:20:25.754Z DEBUG controller-jenkins Operator credentials secret is present {"cr": "jenkins"}
2024-02-06T14:20:25.826Z DEBUG controller-jenkins Scripts config map is present {"cr": "jenkins"}
2024-02-06T14:20:25.877Z DEBUG controller-jenkins Init configuration config map is present {"cr": "jenkins"}
2024-02-06T14:20:25.939Z DEBUG controller-jenkins Base configuration config map is present {"cr": "jenkins"}
2024-02-06T14:20:25.939Z DEBUG controller-jenkins GroovyScripts Secret and ConfigMap added watched labels {"cr": "jenkins"}
2024-02-06T14:20:25.939Z DEBUG controller-jenkins ConfigurationAsCode Secret and ConfigMap added watched labels {"cr": "jenkins"}
2024-02-06T14:20:25.939Z DEBUG controller-jenkins createServiceAccount with annotations map[] {"cr": "jenkins"}
2024-02-06T14:20:26.100Z DEBUG controller-jenkins Service account, role and role binding are present {"cr": "jenkins"}
2024-02-06T14:20:26.100Z DEBUG controller-jenkins Extra role bindings are present {"cr": "jenkins"}
2024-02-06T14:20:26.111Z DEBUG controller-jenkins Jenkins HTTP Service is present {"cr": "jenkins"}
2024-02-06T14:20:26.125Z DEBUG controller-jenkins Jenkins slave Service is present {"cr": "jenkins"}
2024-02-06T14:20:26.125Z DEBUG controller-jenkins Kubernetes resources are present {"cr": "jenkins"}
2024-02-06T14:20:26.125Z DEBUG controller-jenkins Jenkins master pod is present {"cr": "jenkins"}
2024-02-06T14:20:26.125Z DEBUG controller-jenkins Jenkins master pod is terminating {"cr": "jenkins"}
2024-02-06T14:20:28.977Z DEBUG controller-jenkins Reconciling Jenkins {"cr": "jenkins"}
2024-02-06T14:20:28.978Z DEBUG controller-jenkins Operator credentials secret is present {"cr": "jenkins"}
2024-02-06T14:20:29.059Z DEBUG controller-jenkins Scripts config map is present {"cr": "jenkins"}
2024-02-06T14:20:29.100Z DEBUG controller-jenkins Init configuration config map is present {"cr": "jenkins"}
2024-02-06T14:20:29.170Z DEBUG controller-jenkins Base configuration config map is present {"cr": "jenkins"}
2024-02-06T14:20:29.170Z DEBUG controller-jenkins GroovyScripts Secret and ConfigMap added watched labels {"cr": "jenkins"}
2024-02-06T14:20:29.170Z DEBUG controller-jenkins ConfigurationAsCode Secret and ConfigMap added watched labels {"cr": "jenkins"}
2024-02-06T14:20:29.170Z DEBUG controller-jenkins createServiceAccount with annotations map[] {"cr": "jenkins"}
2024-02-06T14:20:29.287Z DEBUG controller-jenkins Service account, role and role binding are present {"cr": "jenkins"}
2024-02-06T14:20:29.287Z DEBUG controller-jenkins Extra role bindings are present {"cr": "jenkins"}
2024-02-06T14:20:29.297Z DEBUG controller-jenkins Jenkins HTTP Service is present {"cr": "jenkins"}
2024-02-06T14:20:29.309Z DEBUG controller-jenkins Jenkins slave Service is present {"cr": "jenkins"}
2024-02-06T14:20:29.309Z DEBUG controller-jenkins Kubernetes resources are present {"cr": "jenkins"}
2024-02-06T14:20:29.309Z DEBUG controller-jenkins Jenkins master pod is present {"cr": "jenkins"}
2024-02-06T14:20:29.309Z DEBUG controller-jenkins Jenkins master pod is terminating {"cr": "jenkins"}
2024-02-06T14:20:34.309Z DEBUG controller-jenkins Reconciling Jenkins {"cr": "jenkins"}
2024-02-06T14:20:34.310Z DEBUG controller-jenkins Operator credentials secret is present {"cr": "jenkins"}
2024-02-06T14:20:34.379Z DEBUG controller-jenkins Scripts config map is present {"cr": "jenkins"}
2024-02-06T14:20:34.407Z DEBUG controller-jenkins Init configuration config map is present {"cr": "jenkins"}
2024-02-06T14:20:34.468Z DEBUG controller-jenkins Base configuration config map is present {"cr": "jenkins"}
2024-02-06T14:20:34.468Z DEBUG controller-jenkins GroovyScripts Secret and ConfigMap added watched labels {"cr": "jenkins"}
2024-02-06T14:20:34.468Z DEBUG controller-jenkins ConfigurationAsCode Secret and ConfigMap added watched labels {"cr": "jenkins"}
2024-02-06T14:20:34.468Z DEBUG controller-jenkins createServiceAccount with annotations map[] {"cr": "jenkins"}
2024-02-06T14:20:34.595Z DEBUG controller-jenkins Service account, role and role binding are present {"cr": "jenkins"}
2024-02-06T14:20:34.595Z DEBUG controller-jenkins Extra role bindings are present {"cr": "jenkins"}
2024-02-06T14:20:34.605Z DEBUG controller-jenkins Jenkins HTTP Service is present {"cr": "jenkins"}
2024-02-06T14:20:34.620Z DEBUG controller-jenkins Jenkins slave Service is present {"cr": "jenkins"}
2024-02-06T14:20:34.620Z DEBUG controller-jenkins Kubernetes resources are present {"cr": "jenkins"}
2024-02-06T14:20:34.620Z DEBUG controller-jenkins Jenkins master pod is present {"cr": "jenkins"}
2024-02-06T14:20:34.620Z DEBUG controller-jenkins Jenkins master pod is terminating {"cr": "jenkins"}
2024-02-06T14:20:39.620Z DEBUG controller-jenkins Reconciling Jenkins {"cr": "jenkins"}
2024-02-06T14:20:39.621Z DEBUG controller-jenkins Operator credentials secret is present {"cr": "jenkins"}
2024-02-06T14:20:39.699Z DEBUG controller-jenkins Scripts config map is present {"cr": "jenkins"}
2024-02-06T14:20:39.740Z DEBUG controller-jenkins Init configuration config map is present {"cr": "jenkins"}
2024-02-06T14:20:39.807Z DEBUG controller-jenkins Base configuration config map is present {"cr": "jenkins"}
2024-02-06T14:20:39.807Z DEBUG controller-jenkins GroovyScripts Secret and ConfigMap added watched labels {"cr": "jenkins"}
2024-02-06T14:20:39.807Z DEBUG controller-jenkins ConfigurationAsCode Secret and ConfigMap added watched labels {"cr": "jenkins"}
2024-02-06T14:20:39.807Z DEBUG controller-jenkins createServiceAccount with annotations map[] {"cr": "jenkins"}
2024-02-06T14:20:39.959Z DEBUG controller-jenkins Service account, role and role binding are present {"cr": "jenkins"}
2024-02-06T14:20:39.959Z DEBUG controller-jenkins Extra role bindings are present {"cr": "jenkins"}
2024-02-06T14:20:39.968Z DEBUG controller-jenkins Jenkins HTTP Service is present {"cr": "jenkins"}
2024-02-06T14:20:39.978Z DEBUG controller-jenkins Jenkins slave Service is present {"cr": "jenkins"}
2024-02-06T14:20:39.978Z DEBUG controller-jenkins Kubernetes resources are present {"cr": "jenkins"}
2024-02-06T14:20:39.978Z DEBUG controller-jenkins Jenkins master pod is present {"cr": "jenkins"}
2024-02-06T14:20:39.978Z DEBUG controller-jenkins Jenkins master pod is terminating {"cr": "jenkins"}
Describe the bug By simply deploying the helm chart as it is the pods get stuck in terminating:
To Reproduce Just install the helm chart with default values, or same effects if you set backup.enable to false.
Additional information
Kubernetes version: v1.24.10 Jenkins Operator version: 0.8.0
Add error logs about the problem here (operator logs and Kubernetes events).
Events from describe pod:
Jenkins-master logs (tail from the logs):
jenkins backup logs:
operator logs