Closed Harguer closed 1 year ago
brokenpip3
commented
1 year ago Hi @Harguer I'm not able to replicate the issue. Could you please share the version of the lts jenkins image that you are using and also can you try to remove everything you have in the jenkins CRD and let the operator deploy a vanilla version from the latest helm chart? Let me know the results so I can try to help with this issue :)
Harguer
commented
1 year ago Hi @brokenpip3, thanks a lot for looking at it :)
I can replicate it easily with the latest lts jenkins, i tried 2.375.2-lts and 2.375.3-lts. This jenkins has been running for 18 days, I'm going to kill it and paste here the process and the part where it breaks.
Pod running for 18 days
$kubectl get pods
NAME READY STATUS RESTARTS AGE
jenkins-jenkins 2/2 Running 0 18d
jenkins-operator-574c9b5587-cfwll 1/1 Running 0 27d
seed-job-agent-jenkins-5df58cddd7-qrlc9 1/1 Running 0 18d
$configuration of the current jenkins CR, where i have pinned all the plugins (the dependencies too) NOTE: I have retrieved the list of all plugins with this:
From the https://jenkins-url/script
Jenkins.instance.pluginManager.plugins.each{
plugin ->
println ("name: (${plugin.getShortName()}) version: \"${plugin.getVersion()}\"")
}
then grab the list above and from bash command line:
cat ~/temp/plugin_list.txt | sed 's#(##g; s#)##g' | egrep "kubernetes |workflow-job |workflow-aggregator |git |job-dsl |configuration-as-code |kubernetes-credentials-provider " | sed 's#name:#- name:#g; s# version:#\n version:#g'
cat ~/temp/plugin_list.txt | sed 's#(##g; s#)##g' | egrep -v "kubernetes |workflow-job |workflow-aggregator |git |job-dsl |configuration-as-code |kubernetes-credentials-provider " | sed 's#name:#- name:#g; s# version:#\n version:#g'the result is pinned here in the jenkins CR
$ kubectl get jenkins -o yaml
apiVersion: v1
items:
- apiVersion: jenkins.io/v1alpha2
kind: Jenkins
metadata:
annotations:
creationTimestamp: "2023-01-20T16:43:20Z"
generation: 19
name: jenkins
namespace: jenkins
spec:
backup:
action:
exec:
command:
- /home/user/bin/backup.sh
containerName: backup
interval: 3600
makeBackupBeforePodDeletion: true
configurationAsCode:
configurations:
- name: jenkins-core
- name: jenkins-agent
secret:
name: ""
groovyScripts:
configurations: []
secret:
name: ""
jenkinsAPISettings:
authorizationStrategy: createUser
master:
basePlugins:
- name: kubernetes
version: 3802.vb_b_600831fcb_3
- name: job-dsl
version: "1.81"
- name: kubernetes-credentials-provider
version: 1.208.v128ee9800c04
- name: configuration-as-code
version: 1569.vb_72405b_80249
- name: workflow-aggregator
version: 590.v6a_d052e5a_a_b_5
- name: git
version: 5.0.0
- name: workflow-job
version: 1268.v6eb_e2ee1a_85a
containers:
- command:
- bash
- -c
- /var/jenkins/scripts/init.sh && exec /usr/bin/tini -s -- /usr/local/bin/jenkins.sh
env:
- name: JAVA_OPTS
value: -XX:MinRAMPercentage=50.0 -XX:MaxRAMPercentage=80.0 -Djenkins.install.runSetupWizard=false
-Djava.awt.headless=true
image: jenkins/jenkins:2.375.2-lts
imagePullPolicy: Always
livenessProbe:
failureThreshold: 12
httpGet:
path: /login
port: http
scheme: HTTP
initialDelaySeconds: 80
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
name: jenkins-master
readinessProbe:
failureThreshold: 3
httpGet:
path: /login
port: http
scheme: HTTP
initialDelaySeconds: 30
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources:
limits:
cpu: "2"
memory: 4Gi
requests:
cpu: "1"
memory: 1Gi
volumeMounts:
- mountPath: /etc/jenkins-saml-signing
name: jenkins-saml-signing
readOnly: true
- env:
- name: BACKUP_DIR
value: /backup
- name: JENKINS_HOME
value: /jenkins-home
- name: BACKUP_COUNT
value: "60"
image: xxxxxxxxxxxx.dkr.ecr.us-east-2.amazonaws.com/jenkins-operator-backup-pvc:1.0.1
imagePullPolicy: IfNotPresent
name: backup
resources:
limits:
cpu: 1500m
memory: 3Gi
requests:
cpu: 500m
memory: 500Mi
volumeMounts:
- mountPath: /jenkins-home
name: jenkins-home
- mountPath: /backup
name: backup
disableCSRFProtection: false
plugins:
- name: javax-mail-api
version: 1.6.2-5
- name: pipeline-milestone-step
version: 111.v449306f708b_7
- name: jakarta-activation-api
version: 2.0.1-2
- name: snakeyaml-api
version: 1.33-90.v80dcb_3814d35
- name: trilead-api
version: 2.84.v72119de229b_7
- name: commons-text-api
version: 1.10.0-27.vb_fa_3896786a_7
- name: workflow-multibranch
version: 716.vc692a_e52371b_
- name: pipeline-stage-tags-metadata
version: 2.2118.v31fd5b_9944b_5
- name: javax-activation-api
version: 1.2.0-5
- name: git-client
version: 4.1.0
- name: variant
version: 59.vf075fe829ccb
- name: scm-api
version: 631.v9143df5b_e4a_a
- name: kubernetes-client-api
version: 6.3.1-206.v76d3b_6b_14db_b
- name: pipeline-stage-step
version: 305.ve96d0205c1c6
- name: pipeline-model-extensions
version: 2.2118.v31fd5b_9944b_5
- name: pipeline-groovy-lib
version: 629.vb_5627b_ee2104
- name: workflow-support
version: 839.v35e2736cfd5c
- name: mailer
version: 448.v5b_97805e3767
- name: jackson2-api
version: 2.14.2-319.v37853346a_229
- name: workflow-scm-step
version: 400.v6b_89a_1317c9a_
- name: jakarta-mail-api
version: 2.0.1-2
- name: pipeline-model-api
version: 2.2118.v31fd5b_9944b_5
- name: pipeline-input-step
version: 466.v6d0a_5df34f81
- name: metrics
version: 4.2.13-420.vea_2f17932dd6
- name: credentials-binding
version: 523.vd859a_4b_122e6
- name: kubernetes-credentials
version: 0.10.0
- name: jaxb
version: 2.3.8-1
- name: pipeline-build-step
version: "2.18"
- name: credentials
version: 1214.v1de940103927
- name: pipeline-model-definition
version: 2.2118.v31fd5b_9944b_5
- name: caffeine-api
version: 2.9.3-65.v6a_47d0f4d1fe
- name: instance-identity
version: 142.v04572ca_5b_265
- name: workflow-step-api
version: 639.v6eca_cd8c04a_a_
- name: plain-credentials
version: 143.v1b_df8b_d3b_e48
- name: bouncycastle-api
version: "2.27"
- name: ace-editor
version: "1.1"
- name: workflow-durable-task-step
version: 1223.v7f1a_98a_8863e
- name: authentication-tokens
version: "1.4"
- name: ssh-credentials
version: 305.v8f4381501156
- name: cloudbees-folder
version: 6.800.v71307ca_b_986b
- name: durable-task
version: 504.vb10d1ae5ba2f
- name: workflow-basic-steps
version: 994.vd57e3ca_46d24
- name: jsch
version: 0.1.55.61.va_e9ee26616e7
- name: structs
version: 324.va_f5d6774f3a_d
- name: apache-httpcomponents-client-4-api
version: 4.5.13-138.v4e7d9a_7b_a_e61
- name: workflow-cps
version: 3618.v13db_a_21f0fcf
- name: ionicons-api
version: 31.v4757b_6987003
- name: display-url-api
version: 2.3.7
- name: commons-lang3-api
version: 3.12.0-36.vd97de6465d5b_
- name: script-security
version: 1229.v4880b_b_e905a_6
- name: branch-api
version: 2.1071.v1a_188a_562481
- name: workflow-api
version: 1208.v0cc7c6e0da_9e
- name: blueocean-config
version: 1.27.1
- name: mina-sshd-api-core
version: 2.9.2-50.va_0e1f42659a_a
- name: jdk-tool
version: "1.4"
- name: github-branch-source
version: 1701.v00cc8184df93
- name: blueocean-pipeline-api-impl
version: 1.27.1
- name: github
version: 1.36.1
- name: blueocean-personalization
version: 1.27.1
- name: aws-java-sdk-ecs
version: 1.12.287-357.vf82d85a_6eefd
- name: aws-java-sdk-ecr
version: 1.12.287-357.vf82d85a_6eefd
- name: aws-java-sdk-minimal
version: 1.12.287-357.vf82d85a_6eefd
- name: blueocean-web
version: 1.27.1
- name: aws-java-sdk-logs
version: 1.12.287-357.vf82d85a_6eefd
- name: blueocean-events
version: 1.27.1
- name: token-macro
version: 321.vd7cc1f2a_52c8
- name: aws-java-sdk
version: 1.12.80
- name: blueocean-i18n
version: 1.27.1
- name: favorite
version: 2.4.1
- name: okhttp-api
version: 4.9.3-108.v0feda04578cf
- name: blueocean-rest
version: 1.27.1
- name: build-user-vars-plugin
version: "1.9"
- name: blueocean-pipeline-editor
version: 1.27.1
- name: saml
version: 4.385.v4dea_91565e9d
- name: github-api
version: 1.303-400.v35c2d8258028
- name: echarts-api
version: 5.4.0-1
- name: sshd
version: 3.275.v9e17c10f2571
- name: matrix-auth
version: 3.1.6
- name: jjwt-api
version: 0.11.5-77.v646c772fddb_0
- name: blueocean-commons
version: 1.27.1
- name: sse-gateway
version: "1.26"
- name: aws-java-sdk-codebuild
version: 1.12.287-357.vf82d85a_6eefd
- name: blueocean-dashboard
version: 1.27.1
- name: blueocean-core-js
version: 1.27.1
- name: blueocean-github-pipeline
version: 1.27.1
- name: role-strategy
version: 584.vf8e515397ecd
- name: blueocean-jwt
version: 1.27.1
- name: aws-java-sdk-elasticbeanstalk
version: 1.12.287-357.vf82d85a_6eefd
- name: aws-java-sdk-ssm
version: 1.12.287-357.vf82d85a_6eefd
- name: ldap
version: "2.4"
- name: plugin-util-api
version: 2.20.0
- name: font-awesome-api
version: 6.2.1-1
- name: aws-java-sdk-cloudformation
version: 1.12.287-357.vf82d85a_6eefd
- name: checks-api
version: 1.8.1
- name: blueocean-rest-impl
version: 1.27.1
- name: pipeline-utility-steps
version: 2.15.0
- name: jquery3-api
version: 3.6.1-2
- name: git-server
version: "1.11"
- name: blueocean-git-pipeline
version: 1.27.1
- name: ghprb
version: 1.42.2
- name: blueocean-pipeline-scm-api
version: 1.27.1
- name: mina-sshd-api-common
version: 2.9.2-50.va_0e1f42659a_a
- name: command-launcher
version: "1.6"
- name: matrix-project
version: 785.v06b_7f47b_c631
- name: bootstrap5-api
version: 5.2.1-3
- name: popper2-api
version: 2.11.6-2
- name: pipeline-graph-analysis
version: 202.va_d268e64deb_3
- name: junit
version: 1166.va_436e268e972
- name: blueocean-bitbucket-pipeline
version: 1.27.1
- name: blueocean
version: 1.27.0
- name: htmlpublisher
version: "1.31"
- name: aws-java-sdk-iam
version: 1.12.287-357.vf82d85a_6eefd
- name: handy-uri-templates-2-api
version: 2.1.8-22.v77d5b_75e6953
- name: aws-java-sdk-ec2
version: 1.12.287-357.vf82d85a_6eefd
- name: ansicolor
version: 1.0.2
- name: envinject
version: 2.4.0
- name: cloudbees-bitbucket-branch-source
version: 791.vb_eea_a_476405b
- name: envinject-api
version: 1.199.v3ce31253ed13
- name: blueocean-display-url
version: 2.4.1
- name: jenkins-design-language
version: 1.27.1
- name: pubsub-light
version: "1.17"
securityContext:
fsGroup: 1000
runAsUser: 1000
volumes:
- name: backup
persistentVolumeClaim:
claimName: jenkins-backup-pvc
- name: jenkins-saml-signing
secret:
defaultMode: 420
items:
- key: jenkins-saml-signing.jks
path: jenkins-saml-signing.jks
secretName: jenkins-saml-signing
restore:
action:
exec:
command:
- /home/user/bin/restore.sh
containerName: backup
getLatestAction:
exec:
command:
- /home/user/bin/get-latest.sh
seedJobAgentImage: jenkins/inbound-agent:4.10-3
seedJobs:
- additionalClasspath: ""
bitbucketPushTrigger: false
buildPeriodically: ""
description: Jenkins Operator repository
failOnMissingPlugin: false
githubPushTrigger: false
id: jenkins-operator
ignoreMissingFiles: false
pollSCM: ""
repositoryBranch: master
repositoryUrl: https://github.com/jenkinsci/kubernetes-operator.git
targets: cicd/jobs/*.jenkins
unstableOnDeprecation: false
service:
annotations:
external-dns.alpha.kubernetes.io/ttl: "60"
port: 8080
type: ClusterIP
serviceAccount: {}
slaveService:
port: 50000
type: ClusterIP
kind: List
metadata:
resourceVersion: ""
selfLink: ""Now, deleting the pod:
$ kubectl delete pod jenkins-jenkins
pod "jenkins-jenkins" deleted
k get pods
NAME READY STATUS RESTARTS AGE
jenkins-jenkins 1/2 Running 0 5s
jenkins-operator-574c9b5587-cfwll 1/1 Running 0 27d
seed-job-agent-jenkins-5df58cddd7-qrlc9 1/1 Running 0 18dNew pod start crashing in a loop:
$ kubectl get pods -w
NAME READY STATUS RESTARTS AGE
jenkins-jenkins 1/2 Running 0 44s
jenkins-operator-574c9b5587-cfwll 1/1 Running 0 27d
seed-job-agent-jenkins-5df58cddd7-qrlc9 1/1 Running 0 18d
jenkins-jenkins 2/2 Running 0 112s
jenkins-jenkins 2/2 Terminating 0 113s
seed-job-agent-jenkins-5df58cddd7-qrlc9 0/1 Error 0 18d
seed-job-agent-jenkins-5df58cddd7-qrlc9 1/1 Running 1 (2s ago) 18d
seed-job-agent-jenkins-5df58cddd7-qrlc9 0/1 Error 1 (4s ago) 18d
seed-job-agent-jenkins-5df58cddd7-qrlc9 0/1 CrashLoopBackOff 1 (11s ago) 18d
seed-job-agent-jenkins-5df58cddd7-qrlc9 1/1 Running 2 (12s ago) 18d
seed-job-agent-jenkins-5df58cddd7-qrlc9 0/1 Error 2 (14s ago) 18d
jenkins-jenkins 1/2 Terminating 0 2m22s
jenkins-jenkins 0/2 Terminating 0 2m24s
jenkins-jenkins 0/2 Terminating 0 2m24s
jenkins-jenkins 0/2 Terminating 0 2m24s
jenkins-jenkins 0/2 Pending 0 0s
jenkins-jenkins 0/2 Pending 0 0s
jenkins-jenkins 0/2 ContainerCreating 0 0s
jenkins-jenkins 1/2 Running 0 2s
seed-job-agent-jenkins-5df58cddd7-qrlc9 0/1 CrashLoopBackOff 2 (15s ago) 18d
seed-job-agent-jenkins-5df58cddd7-qrlc9 1/1 Running 3 (28s ago) 18d
seed-job-agent-jenkins-5df58cddd7-qrlc9 0/1 Error 3 (30s ago) 18d
seed-job-agent-jenkins-5df58cddd7-qrlc9 0/1 CrashLoopBackOff 3 (13s ago) 18d
jenkins-jenkins 2/2 Running 0 71s
jenkins-jenkins 2/2 Terminating 0 72s
seed-job-agent-jenkins-5df58cddd7-qrlc9 1/1 Running 4 (54s ago) 18d
seed-job-agent-jenkins-5df58cddd7-qrlc9 0/1 Error 4 (55s ago) 18d
^C
$Inspecting the jenkins pods, you can see that there is no reason to have this jenkins terminated, it just die (killed by the operator)
...
...
Downloaded echarts-api from https://ftp-chi.osuosl.org/pub/jenkins/plugins/echarts-api/5.4.0-1/echarts-api.hpi (attempt 1 of 3)
Downloaded and validated plugin echarts-api
Checksum valid for: echarts-api
Downloaded aws-java-sdk from https://ftp-chi.osuosl.org/pub/jenkins/plugins/aws-java-sdk/1.12.80/aws-java-sdk.hpi (attempt 1 of 3)
Downloaded and validated plugin aws-java-sdk
Checksum valid for: aws-java-sdk
Done
+ echo 'Installing plugins required by user - end'
Installing plugins required by user - end
Running from: /usr/share/jenkins/jenkins.war
webroot: EnvVars.masterEnvVars.get("JENKINS_HOME")
2023-02-28 14:46:31.185+0000 [id=1] INFO winstone.Logger#logInternal: Beginning extraction from war file
2023-02-28 14:46:32.228+0000 [id=1] WARNING o.e.j.s.handler.ContextHandler#setContextPath: Empty contextPath
2023-02-28 14:46:32.307+0000 [id=1] INFO org.eclipse.jetty.server.Server#doStart: jetty-10.0.12; built: 2022-09-14T01:54:40.076Z; git: 408d0139887e27a57b54ed52e2d92a36731a7e88; jvm 11.0.18+10
2023-02-28 14:46:32.712+0000 [id=1] INFO o.e.j.w.StandardDescriptorProcessor#visitServlet: NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet
2023-02-28 14:46:32.796+0000 [id=1] INFO o.e.j.s.s.DefaultSessionIdManager#doStart: Session workerName=node0
2023-02-28 14:46:33.357+0000 [id=1] INFO hudson.WebAppMain#contextInitialized: Jenkins home directory: /var/lib/jenkins found at: EnvVars.masterEnvVars.get("JENKINS_HOME")
2023-02-28 14:46:33.537+0000 [id=1] INFO o.e.j.s.handler.ContextHandler#doStart: Started w.@42b28ff1{Jenkins v2.375.2,/,file:///var/lib/jenkins/war/,AVAILABLE}{/var/lib/jenkins/war}
2023-02-28 14:46:33.579+0000 [id=1] INFO o.e.j.server.AbstractConnector#doStart: Started ServerConnector@1755e85b{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}
2023-02-28 14:46:33.592+0000 [id=1] INFO org.eclipse.jetty.server.Server#doStart: Started Server@55342f40{STARTING}[10.0.12,sto=0] @3123ms
2023-02-28 14:46:33.603+0000 [id=23] INFO winstone.Logger#logInternal: Winstone Servlet Engine running: controlPort=disabled
2023-02-28 14:46:33.896+0000 [id=30] INFO jenkins.InitReactorRunner$1#onAttained: Started initialization
2023-02-28 14:46:39.917+0000 [id=29] INFO jenkins.InitReactorRunner$1#onAttained: Listed all plugins
2023-02-28 14:46:45.872+0000 [id=28] INFO jenkins.InitReactorRunner$1#onAttained: Prepared all plugins
2023-02-28 14:46:45.904+0000 [id=29] INFO jenkins.InitReactorRunner$1#onAttained: Started all plugins
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.codehaus.groovy.vmplugin.v7.Java7$1 (file:/var/lib/jenkins/war/WEB-INF/lib/groovy-all-2.4.21.jar) to constructor java.lang.invoke.MethodHandles$Lookup(java.lang.Class,int)
WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.vmplugin.v7.Java7$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2023-02-28 14:46:47.414+0000 [id=28] INFO c.c.j.p.k.KubernetesCredentialProvider#startWatchingForSecrets: retrieving secrets with selector: jenkins.io/credentials-type, LabelSelector(matchExpressions=[], matchLabels={}, additionalProperties={})
2023-02-28 14:46:47.585+0000 [id=31] INFO jenkins.InitReactorRunner$1#onAttained: Augmented all extensions
2023-02-28 14:46:49.772+0000 [id=28] INFO jenkins.InitReactorRunner$1#onAttained: System config loaded
2023-02-28 14:46:49.883+0000 [id=30] INFO jenkins.InitReactorRunner$1#onAttained: System config adapted
2023-02-28 14:46:49.883+0000 [id=29] INFO jenkins.InitReactorRunner$1#onAttained: Loaded all jobs
2023-02-28 14:46:49.888+0000 [id=30] INFO jenkins.InitReactorRunner$1#onAttained: Configuration for all jobs updated
2023-02-28 14:46:49.895+0000 [id=30] INFO j.util.groovy.GroovyHookScript#execute: Executing /var/lib/jenkins/init.groovy.d/createOperatorUser.groovy
2023-02-28 14:46:49.905+0000 [id=59] INFO hudson.util.Retrier#start: Attempt #1 to do the action check updates server
2023-02-28 14:46:51.206+0000 [id=28] INFO jenkins.InitReactorRunner$1#onAttained: Completed initialization
2023-02-28 14:46:51.298+0000 [id=22] INFO hudson.lifecycle.Lifecycle#onReady: Jenkins is fully up and running
2023-02-28 14:47:05.172+0000 [id=59] INFO h.m.DownloadService$Downloadable#load: Obtained the updated data file for hudson.tasks.Maven.MavenInstaller
2023-02-28 14:47:05.643+0000 [id=59] INFO h.m.DownloadService$Downloadable#load: Obtained the updated data file for hudson.tools.JDKInstaller
2023-02-28 14:47:05.644+0000 [id=59] INFO hudson.util.Retrier#start: Performed the action check updates server successfully at the attempt #1
2023-02-28 14:47:10.568+0000 [id=24] INFO winstone.Logger#logInternal: JVM is terminating. Shutting down Jetty
$But from the jenkins operator, you can see the reason, it is because jenkins-operator compares with a fresh list of plugins here https://github.com/jenkinsci/kubernetes-operator/blob/master/pkg/configuration/base/plugin.go#L15-L46
2023-02-28T14:47:09.820Z DEBUG controller-jenkins Kubernetes resources are present {"cr": "jenkins"}
2023-02-28T14:47:09.820Z DEBUG controller-jenkins Jenkins master pod is present {"cr": "jenkins"}
2023-02-28T14:47:09.820Z DEBUG controller-jenkins Jenkins master pod is ready {"cr": "jenkins"}
2023-02-28T14:47:10.423Z DEBUG controller-jenkins Jenkins API client set {"cr": "jenkins"}
2023-02-28T14:47:10.530Z DEBUG controller-jenkins Installed plugins '[kubernetes:3802.vb_b_600831fcb_3 pipeline-milestone-step:111.v449306f708b_7 mina-sshd-api-core:2.9.2-50.va_0e1f42659a_a jakarta-activation-api:2.0.1-3 snakeyaml-api:1.33-95.va_b_a_e3e47b_fa_4 trilead-api:2.84.v72119de229b_7 commons-text-api:1.10.0-27.vb_fa_3896786a_7 workflow-multibranch:733.v109046189126 job-dsl:1.81 pipeline-stage-tags-metadata:2.2118.v31fd5b_9944b_5 javax-activation-api:1.2.0-6 git-client:4.1.0 variant:59.vf075fe829ccb scm-api:631.v9143df5b_e4a_a kubernetes-client-api:6.4.1-208.vfe09a_9362c2c pipeline-stage-step:305.ve96d0205c1c6 pipeline-model-extensions:2.2118.v31fd5b_9944b_5 kubernetes-credentials-provider:1.208.v128ee9800c04 pipeline-groovy-lib:629.vb_5627b_ee2104 configuration-as-code:1569.vb_72405b_80249 workflow-support:839.v35e2736cfd5c mailer:448.v5b_97805e3767 jackson2-api:2.14.2-319.v37853346a_229 workflow-scm-step:400.v6b_89a_1317c9a_ jakarta-mail-api:2.0.1-3 pipeline-model-api:2.2118.v31fd5b_9944b_5 pipeline-input-step:466.v6d0a_5df34f81 metrics:4.2.13-420.vea_2f17932dd6 credentials-binding:523.vd859a_4b_122e6 workflow-aggregator:590.v6a_d052e5a_a_b_5 jaxb:2.3.8-1 kubernetes-credentials:0.10.0 pipeline-build-step:2.18.1 credentials:1214.v1de940103927 pipeline-model-definition:2.2118.v31fd5b_9944b_5 mina-sshd-api-common:2.9.2-50.va_0e1f42659a_a caffeine-api:2.9.3-65.v6a_47d0f4d1fe instance-identity:142.v04572ca_5b_265 workflow-step-api:639.v6eca_cd8c04a_a_ plain-credentials:143.v1b_df8b_d3b_e48 bouncycastle-api:2.27 git:5.0.0 workflow-durable-task-step:1234.v019404b_3832a authentication-tokens:1.4 ssh-credentials:305.v8f4381501156 cloudbees-folder:6.800.v71307ca_b_986b durable-task:504.vb10d1ae5ba2f workflow-job:1268.v6eb_e2ee1a_85a workflow-basic-steps:1010.vf7a_b_98e847c1 structs:324.va_f5d6774f3a_d apache-httpcomponents-client-4-api:4.5.13-138.v4e7d9a_7b_a_e61 workflow-cps:3618.v13db_a_21f0fcf ionicons-api:31.v4757b_6987003 commons-lang3-api:3.12.0-36.vd97de6465d5b_ display-url-api:2.3.7 script-security:1229.v4880b_b_e905a_6 branch-api:2.1071.v1a_188a_562481 workflow-api:1208.v0cc7c6e0da_9e blueocean-config:1.27.1 jdk-tool:1.4 github-branch-source:1701.v00cc8184df93 github:1.36.1 blueocean-pipeline-api-impl:1.27.1 blueocean-personalization:1.27.1 aws-java-sdk-ecs:1.12.287-357.vf82d85a_6eefd aws-java-sdk-ecr:1.12.287-357.vf82d85a_6eefd aws-java-sdk-minimal:1.12.287-357.vf82d85a_6eefd blueocean-web:1.27.1 aws-java-sdk-logs:1.12.287-357.vf82d85a_6eefd blueocean-events:1.27.1 token-macro:321.vd7cc1f2a_52c8 aws-java-sdk:1.12.80 blueocean-i18n:1.27.1 favorite:2.4.1 okhttp-api:4.9.3-108.v0feda04578cf blueocean-rest:1.27.1 build-user-vars-plugin:1.9 ace-editor:1.1 blueocean-pipeline-editor:1.27.1 saml:4.385.v4dea_91565e9d jsch:0.1.55.61.va_e9ee26616e7 github-api:1.303-400.v35c2d8258028 echarts-api:5.4.0-1 sshd:3.275.v9e17c10f2571 matrix-auth:3.1.6 jjwt-api:0.11.5-77.v646c772fddb_0 blueocean-commons:1.27.1 sse-gateway:1.26 aws-java-sdk-codebuild:1.12.287-357.vf82d85a_6eefd blueocean-dashboard:1.27.1 blueocean-core-js:1.27.1 blueocean-github-pipeline:1.27.1 role-strategy:584.vf8e515397ecd blueocean-jwt:1.27.1 aws-java-sdk-elasticbeanstalk:1.12.287-357.vf82d85a_6eefd javax-mail-api:1.6.2-5 aws-java-sdk-ssm:1.12.287-357.vf82d85a_6eefd ldap:2.4 plugin-util-api:2.20.0 font-awesome-api:6.2.1-1 aws-java-sdk-cloudformation:1.12.287-357.vf82d85a_6eefd checks-api:1.8.1 blueocean-rest-impl:1.27.1 pipeline-utility-steps:2.15.0 jquery3-api:3.6.1-2 git-server:1.11 blueocean-git-pipeline:1.27.1 ghprb:1.42.2 blueocean-pipeline-scm-api:1.27.1 command-launcher:1.6 matrix-project:785.v06b_7f47b_c631 bootstrap5-api:5.2.1-3 popper2-api:2.11.6-2 junit:1166.va_436e268e972 pipeline-graph-analysis:202.va_d268e64deb_3 blueocean-bitbucket-pipeline:1.27.1 blueocean:1.27.0 htmlpublisher:1.31 aws-java-sdk-iam:1.12.287-357.vf82d85a_6eefd handy-uri-templates-2-api:2.1.8-22.v77d5b_75e6953 aws-java-sdk-ec2:1.12.287-357.vf82d85a_6eefd ansicolor:1.0.2 envinject:2.4.0 cloudbees-bitbucket-branch-source:791.vb_eea_a_476405b blueocean-display-url:2.4.1 envinject-api:1.199.v3ce31253ed13 jenkins-design-language:1.27.1 pubsub-light:1.17]' {"cr": "jenkins"}
2023-02-28T14:47:10.530Z WARN controller-jenkins Incompatible plugin '{jakarta-activation-api 2.0.1-2 }' version, actual '2.0.1-3' {"cr": "jenkins"}
github.com/go-logr/zapr.(*zapLogger).Info
/go/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:126
github.com/jenkinsci/kubernetes-operator/pkg/configuration/base.(*JenkinsBaseConfigurationReconciler).verifyPlugins
/workspace/pkg/configuration/base/plugin.go:39
github.com/jenkinsci/kubernetes-operator/pkg/configuration/base.(*JenkinsBaseConfigurationReconciler).Reconcile
/workspace/pkg/configuration/base/reconciler.go:107
github.com/jenkinsci/kubernetes-operator/controllers.(*JenkinsReconciler).reconcile
/workspace/controllers/jenkins_controller.go:252
github.com/jenkinsci/kubernetes-operator/controllers.(*JenkinsReconciler).Reconcile
/workspace/controllers/jenkins_controller.go:136
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:235
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.1
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:198
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:99
2023-02-28T14:47:10.530Z WARN controller-jenkins Incompatible plugin '{snakeyaml-api 1.33-90.v80dcb_3814d35 }' version, actual '1.33-95.va_b_a_e3e47b_fa_4' {"cr": "jenkins"}
github.com/go-logr/zapr.(*zapLogger).Info
/go/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:126
github.com/jenkinsci/kubernetes-operator/pkg/configuration/base.(*JenkinsBaseConfigurationReconciler).verifyPlugins
/workspace/pkg/configuration/base/plugin.go:39
github.com/jenkinsci/kubernetes-operator/pkg/configuration/base.(*JenkinsBaseConfigurationReconciler).Reconcile
/workspace/pkg/configuration/base/reconciler.go:107
github.com/jenkinsci/kubernetes-operator/controllers.(*JenkinsReconciler).reconcile
/workspace/controllers/jenkins_controller.go:252
github.com/jenkinsci/kubernetes-operator/controllers.(*JenkinsReconciler).Reconcile
/workspace/controllers/jenkins_controller.go:136
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:235
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.1
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:198
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:99
2023-02-28T14:47:10.530Z WARN controller-jenkins Incompatible plugin '{workflow-multibranch 716.vc692a_e52371b_ }' version, actual '733.v109046189126' {"cr": "jenkins"}
github.com/go-logr/zapr.(*zapLogger).Info
/go/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:126
github.com/jenkinsci/kubernetes-operator/pkg/configuration/base.(*JenkinsBaseConfigurationReconciler).verifyPlugins
/workspace/pkg/configuration/base/plugin.go:39
github.com/jenkinsci/kubernetes-operator/pkg/configuration/base.(*JenkinsBaseConfigurationReconciler).Reconcile
/workspace/pkg/configuration/base/reconciler.go:107
github.com/jenkinsci/kubernetes-operator/controllers.(*JenkinsReconciler).reconcile
/workspace/controllers/jenkins_controller.go:252
github.com/jenkinsci/kubernetes-operator/controllers.(*JenkinsReconciler).Reconcile
/workspace/controllers/jenkins_controller.go:136
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:235
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.1
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:198
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:99
2023-02-28T14:47:10.530Z WARN controller-jenkins Incompatible plugin '{javax-activation-api 1.2.0-5 }' version, actual '1.2.0-6' {"cr": "jenkins"}
github.com/go-logr/zapr.(*zapLogger).Info
/go/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:126
github.com/jenkinsci/kubernetes-operator/pkg/configuration/base.(*JenkinsBaseConfigurationReconciler).verifyPlugins
/workspace/pkg/configuration/base/plugin.go:39
github.com/jenkinsci/kubernetes-operator/pkg/configuration/base.(*JenkinsBaseConfigurationReconciler).Reconcile
/workspace/pkg/configuration/base/reconciler.go:107
github.com/jenkinsci/kubernetes-operator/controllers.(*JenkinsReconciler).reconcile
/workspace/controllers/jenkins_controller.go:252
github.com/jenkinsci/kubernetes-operator/controllers.(*JenkinsReconciler).Reconcile
/workspace/controllers/jenkins_controller.go:136
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:235
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.1
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:198
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:99
2023-02-28T14:47:10.530Z WARN controller-jenkins Incompatible plugin '{kubernetes-client-api 6.3.1-206.v76d3b_6b_14db_b }' version, actual '6.4.1-208.vfe09a_9362c2c' {"cr": "jenkins"}
github.com/go-logr/zapr.(*zapLogger).Info
/go/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:126
github.com/jenkinsci/kubernetes-operator/pkg/configuration/base.(*JenkinsBaseConfigurationReconciler).verifyPlugins
/workspace/pkg/configuration/base/plugin.go:39
github.com/jenkinsci/kubernetes-operator/pkg/configuration/base.(*JenkinsBaseConfigurationReconciler).Reconcile
/workspace/pkg/configuration/base/reconciler.go:107
github.com/jenkinsci/kubernetes-operator/controllers.(*JenkinsReconciler).reconcile
/workspace/controllers/jenkins_controller.go:252
github.com/jenkinsci/kubernetes-operator/controllers.(*JenkinsReconciler).Reconcile
/workspace/controllers/jenkins_controller.go:136
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:235
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.1
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:198
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:99
2023-02-28T14:47:10.531Z WARN controller-jenkins Incompatible plugin '{jakarta-mail-api 2.0.1-2 }' version, actual '2.0.1-3' {"cr": "jenkins"}
github.com/go-logr/zapr.(*zapLogger).Info
/go/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:126
github.com/jenkinsci/kubernetes-operator/pkg/configuration/base.(*JenkinsBaseConfigurationReconciler).verifyPlugins
/workspace/pkg/configuration/base/plugin.go:39
github.com/jenkinsci/kubernetes-operator/pkg/configuration/base.(*JenkinsBaseConfigurationReconciler).Reconcile
/workspace/pkg/configuration/base/reconciler.go:107
github.com/jenkinsci/kubernetes-operator/controllers.(*JenkinsReconciler).reconcile
/workspace/controllers/jenkins_controller.go:252
github.com/jenkinsci/kubernetes-operator/controllers.(*JenkinsReconciler).Reconcile
/workspace/controllers/jenkins_controller.go:136
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:235
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.1
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:198
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:99
2023-02-28T14:47:10.531Z WARN controller-jenkins Incompatible plugin '{pipeline-build-step 2.18 }' version, actual '2.18.1' {"cr": "jenkins"}
github.com/go-logr/zapr.(*zapLogger).Info
/go/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:126
github.com/jenkinsci/kubernetes-operator/pkg/configuration/base.(*JenkinsBaseConfigurationReconciler).verifyPlugins
/workspace/pkg/configuration/base/plugin.go:39
github.com/jenkinsci/kubernetes-operator/pkg/configuration/base.(*JenkinsBaseConfigurationReconciler).Reconcile
/workspace/pkg/configuration/base/reconciler.go:107
github.com/jenkinsci/kubernetes-operator/controllers.(*JenkinsReconciler).reconcile
/workspace/controllers/jenkins_controller.go:252
github.com/jenkinsci/kubernetes-operator/controllers.(*JenkinsReconciler).Reconcile
/workspace/controllers/jenkins_controller.go:136
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:235
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.1
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:198
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:99
2023-02-28T14:47:10.531Z WARN controller-jenkins Incompatible plugin '{workflow-durable-task-step 1223.v7f1a_98a_8863e }' version, actual '1234.v019404b_3832a' {"cr": "jenkins"}
github.com/go-logr/zapr.(*zapLogger).Info
/go/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:126
github.com/jenkinsci/kubernetes-operator/pkg/configuration/base.(*JenkinsBaseConfigurationReconciler).verifyPlugins
/workspace/pkg/configuration/base/plugin.go:39
github.com/jenkinsci/kubernetes-operator/pkg/configuration/base.(*JenkinsBaseConfigurationReconciler).Reconcile
/workspace/pkg/configuration/base/reconciler.go:107
github.com/jenkinsci/kubernetes-operator/controllers.(*JenkinsReconciler).reconcile
/workspace/controllers/jenkins_controller.go:252
github.com/jenkinsci/kubernetes-operator/controllers.(*JenkinsReconciler).Reconcile
/workspace/controllers/jenkins_controller.go:136
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:235
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.1
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:198
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:99
2023-02-28T14:47:10.531Z WARN controller-jenkins Incompatible plugin '{workflow-basic-steps 994.vd57e3ca_46d24 }' version, actual '1010.vf7a_b_98e847c1' {"cr": "jenkins"}
github.com/go-logr/zapr.(*zapLogger).Info
/go/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:126
github.com/jenkinsci/kubernetes-operator/pkg/configuration/base.(*JenkinsBaseConfigurationReconciler).verifyPlugins
/workspace/pkg/configuration/base/plugin.go:39
github.com/jenkinsci/kubernetes-operator/pkg/configuration/base.(*JenkinsBaseConfigurationReconciler).Reconcile
/workspace/pkg/configuration/base/reconciler.go:107
github.com/jenkinsci/kubernetes-operator/controllers.(*JenkinsReconciler).reconcile
/workspace/controllers/jenkins_controller.go:252
github.com/jenkinsci/kubernetes-operator/controllers.(*JenkinsReconciler).Reconcile
/workspace/controllers/jenkins_controller.go:136
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:235
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.1
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:198
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:99
2023-02-28T14:47:10.531Z INFO controller-jenkins Some plugins have changed, restarting Jenkins {"cr": "jenkins"}
2023-02-28T14:47:10.545Z DEBUG controller-jenkins Reconciling Jenkins {"cr": "jenkins"}
2023-02-28T14:47:10.545Z DEBUG controller-jenkins Operator credentials secret is present {"cr": "jenkins"}
2023-02-28T14:47:10.577Z DEBUG controller-jenkins Scripts config map is present {"cr": "jenkins"}
2023-02-28T14:47:10.593Z DEBUG controller-jenkins Init configuration config map is present {"cr": "jenkins"}
2023-02-28T14:47:10.691Z DEBUG controller-jenkins Base configuration config map is present {"cr": "jenkins"}
2023-02-28T14:47:10.691Z DEBUG controller-jenkins GroovyScripts Secret and ConfigMap added watched labels {"cr": "jenkins"}
2023-02-28T14:47:10.691Z DEBUG controller-jenkins ConfigurationAsCode Secret and ConfigMap added watched labels {"cr": "jenkins"}
2023-02-28T14:47:10.691Z DEBUG controller-jenkins createServiceAccount with annotations map[] {"cr": "jenkins"}
2023-02-28T14:47:10.847Z DEBUG controller-jenkins Service account, role and role binding are present {"cr": "jenkins"}
2023-02-28T14:47:10.847Z DEBUG controller-jenkins Extra role bindings are present {"cr": "jenkins"}
2023-02-28T14:47:10.853Z DEBUG controller-jenkins Jenkins HTTP Service is present {"cr": "jenkins"}
2023-02-28T14:47:10.860Z DEBUG controller-jenkins Jenkins slave Service is present {"cr": "jenkins"}
2023-02-28T14:47:10.860Z DEBUG controller-jenkins Kubernetes resources are present {"cr": "jenkins"}
2023-02-28T14:47:10.860Z DEBUG controller-jenkins Jenkins master pod is present {"cr": "jenkins"}
2023-02-28T14:47:10.860Z DEBUG controller-jenkins Jenkins master pod is terminating {"cr": "jenkins"}This is an important issue it could make people to leave using jenkins operator, every time jenkins dies, you need to update the plugins, I have tried pinned all plugins directly into the dockerfile, but same result, jenkins-operator eventually will get restarted and then it could crash because some plugin needs to be updated, it happened during the weekends or at nights with my prod jenkins, and it creates incidents and not good for my team believe me :')
brokenpip3
commented
1 year ago Ok I see several things we can try/improve here :)
configuration of the current jenkins CR, where i have pinned all the plugins (the dependencies too)
Please don't, this is a common mistake, you don't need to pin the dependencies otherwise you are going to have this kind of issue all the time :) Instead try to do this:
blueocean version it's the compatible one with your 2.375.2-lts and try to pin only the the blueocean version in the plugins crd section, not all the dependencies. Inspecting the jenkins pods, you can see that there is no reason to have this jenkins terminated, it just die (killed by the operator)
Can you describe the pods?
But from the jenkins operator, you can see the reason, it is because jenkins-operator compares with a fresh list of plugins here https://github.com/jenkinsci/kubernetes-operator/blob/master/pkg/configuration/base/plugin.go#L15-L46
I don't think this is true, all of those are warnings during a reconcile loop, it's the operator that is killing the pod because its checking the plugins update, looking at the logs mostly likely it's a jvm issue, with all of these plugins only:
resources:
limits:
cpu: "2"
memory: 4Gi
requests:
cpu: "1"
memory: 1Giis not enough, try by doubling the size of limits and request here and let me know
This is an important issue it could make people to leave using jenkins operator, every time jenkins dies, you need to update the plugins, I have tried pinned all plugins directly into the dockerfile, but same result, jenkins-operator eventually will get restarted and then it could crash because some plugin needs to be updated, it happened during the weekends or at nights with my prod jenkins, and it creates incidents and not good for my team believe me :')
Can happen that a new version of jenkins will require some version for some plugins but in general jenkins will not crash because there is a plugin update.
Try these suggestions and let me know :)
rikycaldeira
commented
1 year ago We're hitting this problem as well since we migrated to the operator with tag 60b8ee5. We have all the plugin versions pinned in the CR but a Jenkins restart (for whatever reason) will most likely not start correctly because of an automatic plugin update, entering a restart loop
I don't completely understand the reasoning for the operator to compare with a fresh list of plugins, I would hope for it to respect the CR versions otherwise we can't follow a GitOps approach (jenkins resource declaration in git != what's actually running in prod)
Even if we follow the steps above to get a shorter plugin list to pin is there any guarantee that one of those pinned plugins won't get automatically bumped to the latest version?
brokenpip3
commented
1 year ago hi @rikycaldeira
I don't completely understand the reasoning for the operator to compare with a fresh list of plugins, I would hope for it to respect the CR versions otherwise we can't follow a GitOps approach (jenkins resource declaration in git != what's actually running in prod) Even if we follow the steps above to get a shorter plugin list to pin is there any guarantee that one of those pinned plugins won't get automatically bumped to the latest version?
Nothing will automatically bump the plugin to the latest version.
If you check the code here the operator will just double check if a plugin from the plugins is installed in jenkins and if is compatible vs the current running jenkins version. Nothing more than trowing a warning, it will not perform an update.
Let's take the @Harguer logs/config as example, he has in the base plugin (by default) the kubernetes plugin but he is pinning one of the dependencies: which is the kubernetes-client-api and probably is not compatible with that version as per operator logs:
2023-02-28T14:47:10.530Z WARN controller-jenkins Incompatible plugin '{kubernetes-client-api 6.3.1-206.v76d3b_6b_14db_b }' version, actual '6.4.1-208.vfe09a_9362c2c' {"cr": "jenkins"}but again this is something that should be avoided, you need to pin only the plugins you want to install without touching the dependencies, only adding to the gitops state what you need and do not bother if you see some plugins (not listed in your plugins list) that will require an update.
We're hitting this problem as well since we migrated to the operator with tag https://github.com/jenkinsci/kubernetes-operator/commit/60b8ee56debf840f0f6bee45081f23eacd7d2068.
Can you please share operator/jenkins logs, k8s events of the jenkins pods and the jenkins crd that you are using? so I can take a look and help if needed
rikycaldeira
commented
1 year ago Thank you for the help! I invested some time in understanding how plugin installation is performed, and it seems that this is the default behavior of the plugin-installation-manager-tool that replaced the install-plugins.sh script. When we migrated to use the operator version 60b8ee5 we also updated jenkins to a recent version (currently 2.391), and started using the tool instead of the script
According to the docs by default the tool will take the latest version of the dependencies even if they are pinned in the plugin list that the operator is providing
The version-pinning behavior of this plugin installer has changed compared to the previous Jenkins plugin installer. By default, --latest option defaults to true, which means that even if you pass a list of pinned versions, these may fail to be installed correctly if they or some other dependency has a newer latest version available. In order to use only pinned versions of plugins, you must pass --latest=false. NOTE: When a new dependency is added to a plugin, it won’t get updated until you notice that it’s missing from your plugin list. (Details here: https://github.com/jenkinsci/plugin-installation-manager-tool/issues/250)
This explains the behavior, and I understand that the recommended way to set it up is to not pin dependencies, but I think it's a valid use case to be able to control everything that gets installed in the pod
At the moment there's no way to provide the --latest false option to the tool, maybe that option could be added to the jenkins CR?
The tool is called by the init.sh script created by the operator here https://github.com/jenkinsci/kubernetes-operator/blob/d91a729ccf895a9e10155ccc262e1f1d826ad4af/pkg/configuration/base/resources/scripts_configmap.go#L350
lemeurherve
commented
1 year ago Using latest for plugins is a mistake IMO.
Nothing will automatically bump the plugin to the latest version.
There are tools for that, see for example how we're doing it in jenkins-infra/docker-jenkins-lts and jenkins-infra/docker-jenkins-weekly, ex https://github.com/jenkins-infra/docker-jenkins-lts/blob/main/.github/workflows/update.yaml
brokenpip3
commented
1 year ago Thank you for the help! I invested some time in understanding how plugin installation is performed, and it seems that this is the default behavior of the plugin-installation-manager-tool that replaced the install-plugins.sh script. When we migrated to use the operator version 60b8ee5 we also updated jenkins to a recent version (currently 2.391), and started using the tool instead of the script
According to the docs by default the tool will take the latest version of the dependencies even if they are pinned in the plugin list that the operator is providing
First of all thanks a lot to digging deeper on this :muscle: I didn't realize about this cli behavior change when we switched from the old to the new plugin binary.
This explains the behavior, and I understand that the recommended way to set it up is to not pin dependencies, but I think it's a valid use case to be able to control everything that gets installed in the pod At the moment there's no way to provide the
--latest falseoption to the tool, maybe that option could be added to the jenkins CR? The tool is called by the init.sh script created by the operator here
I did some tests:
$ cat /base-plugins.txt
kubernetes:3883.v4d70a_a_a_df034
kubernetes-credentials-provider:1.209.v862c6e5fb_1ef
workflow-job:1282.ve6d865025906
workflow-aggregator:590.v6a_d052e5a_a_b_5
git:5.0.0
job-dsl:1.81
configuration-as-code:1569.vb_72405b_80249using this ^ simple list of plugins I ran twice the jenkins-plugin-cli:
$ jenkins-plugin-cli --verbose -f base-plugins.txt --no-download -l --latest=false > /tmp/latest-false
$ jenkins-plugin-cli --verbose -f base-plugins.txt --no-download -l --latest=true > /tmp/latest-trueand then compared the result:
$ diff /tmp/latest-false /tmp/latest-true|wc -l
104
$ diff /tmp/latest-false /tmp/latest-true|grep -i kubernetes-client-api
< kubernetes-client-api 6.4.1-208.vfe09a_9362c2c
> kubernetes-client-api 6.4.1-215.v2ed17097a_8e9Indeed we have differences for all the dependencies like for instance the kubernetes-client-api
So I think your suggestion it's correct we should add --latest=false to our code, I will do it later this week after fixing the CI :)
However I still think that pinning the dependencies should be avoided and it's the root cause of the operators warning logs reported in this issue because this jenkins-plugin-cli flow is happening before the jenkins war launch and if will fail will let the pod fail and never reach the situation where the pod is up and the operator is trying to validate if the plugins in the lists are installed and compatible.
Thanks for your contribution! Will let you know when I have the PR ready and tested
brokenpip3
commented
1 year ago Using
latestfor plugins is a mistake IMO.
Thanks for your feedback!
Nothing will automatically bump the plugin to the latest version.
There are tools for that, see for example how we're doing it in jenkins-infra/docker-jenkins-lts and jenkins-infra/docker-jenkins-weekly, ex https://github.com/jenkins-infra/docker-jenkins-lts/blob/main/.github/workflows/update.yaml
By Nothing will automatically bump the plugin to the latest version.I was just trying to emphasize the fact that there is no code in the operator that will automatically bump the plugins listed as code to the latest version without user intervention not that there is no way to achieve this in general :)
However your example was great because I thought about creating a github action that could be used by our community members that follow a gitops flow (like me) to install and configure this jenkins operator. Thanks! :medal_military: https://github.com/jenkinsci/kubernetes-operator/issues/803
brokenpip3
commented
1 year ago Hi @Harguer @rikycaldeira, I think I have a better understanding of the issue now.
We switched from the old jenkins cli plugin sh to the new jenkins-plugin-cli and the new binary is smarter and it will verify while downloading a plugin if it's incompatible with the current version of jenkins:
...
Will install new plugin workflow-job 1282.ve6d865025906
Will use url: https://updates.jenkins.io/download/plugins/workflow-job/1282.ve6d865025906/workflow-job.hpi to download workflow-job plugin
...
Installed version (1282.ve6d865025906) of workflow-job is less than minimum required version of 1284.v2fe8ed4573d4, bundled plugin will be upgraded
Will use url: https://updates.jenkins.io/download/plugins/workflow-job/1284.v2fe8ed4573d4/workflow-job.hpi to download workflow-job pluginand it will download the compatible version. I could be wrong but the old install-plugin bash script was not so smart and jenkins was just not able to boot properly. So the situation where as code we had the incompatible plugin and at runtime the correct one was never possible (in the past).
The operator is reporting a need to restart here while founding a not compatible plugin defined as code and with this jenkins-plugin-cli will basically end up to a restart loop.
In the current "road to 8.0" I changed the behavior, the operator will only report the fact that we specified as code the wrong plugin version. I also added the possibility to switch from "true" to "false" with the --latest argument of jenkins-plugin-cli but I still believe that should remain true, but like I said giving the possibility to the user to change it if thinks will be necessary.
I will merge the PR soon
brokenpip3
commented
1 year ago @Harguer @rikycaldeira could you please try the latest chart from master branch and let me know? thanks!
brokenpip3
commented
1 year ago any luck with the tests? Thanks!
Harguer
commented
1 year ago hey @brokenpip3 , thanks a lot for your help with this, i will try it and i will let you know :)
brokenpip3
commented
1 year ago gently ping :)
brokenpip3
commented
1 year ago yet another gently ping :)
Harguer
commented
1 year ago hey @brokenpip3 , sorry for the delay, i was busy with other things, but now i'm starting taking a look, give a week or so to test. I will get back to you next week. Thanks again for all your support :)
Harguer
commented
1 year ago hi @brokenpip3 , in my test env. I have created new jenkins-operator setup using this image:
quay.io/jenkins-kubernetes-operator/operator:latestI had to install it via CRDs , the helm chart for some reason failed with an error about the name of the backup PVC. Also seems those are out of date, so I ended editing manually the deployment with quay.io/jenkins-kubernetes-operator/operator:latest.
I'm not sure if the updated documentation is some where else out of the jenkins operator web site.
Also using this in my jenkins cr quay.io/jenkins-kubernetes-operator/backup-pvc
I removed all my pinned plugins and trying with the compatibles that jenkins operator pointed out. I will kill this pod in a week or so to see if this time jenkins operator is not complaining and breaking because it needs new plugins...
Thanks again for your support
brokenpip3
commented
1 year ago Hi @Harguer thanks a lot for the feedback.
I had to install it via CRDs
by CRDs you mean by the plain text manifests?
What was exactly the helm install error? Do you still have it somewhere?
Thanks again for your support
yw :)
Harguer
commented
1 year ago Hi @brokenpip3
by CRDs you mean by the plain text manifests?
Yes, the manifest kubectl apply -f https://....
What was exactly the helm install error? Do you still have it somewhere?
This is the error log from jenkins operator, and no jenkins pod showed up, and I have no idea why it was looking for jenkins-backup instead of jenkins-backup-pvc that is how it is setup in my Jenkins CR:
/go/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:126
github.com/jenkinsci/kubernetes-operator/controllers.(*JenkinsReconciler).reconcile
/workspace/controllers/jenkins_controller.go:243
github.com/jenkinsci/kubernetes-operator/controllers.(*JenkinsReconciler).Reconcile
/workspace/controllers/jenkins_controller.go:136
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:235
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.1
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:198
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:99
2023-05-15T21:25:42.707Z WARN controller-jenkins PersistentVolumeClaim 'jenkins-backup' not found for volume '{backup {nil nil nil nil nil nil nil nil nil &PersistentVolumeClaimVolumeSource{ClaimName:jenkins-backup,ReadOnly:false,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}' {"cr": "jenkins"}
github.com/go-logr/zapr.(*zapLogger).Info
/go/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:126I was 100% sure the PVC was working fine because that jenkins was running before i deleted all the CRDs and used helm to install it Jenkins CR Volume section:
volumes:
- name: backup
persistentVolumeClaim:
claimName: jenkins-backup-pvc
containers:
containers:
- image: jenkins/jenkins:2.387.3-lts
...
...
- env:
- name: BACKUP_DIR
value: /backup
- name: JENKINS_HOME
value: /jenkins-home
- name: BACKUP_COUNT
value: "200"
image: quay.io/jenkins-kubernetes-operator/backup-pvc
imagePullPolicy: IfNotPresent
name: backup
resources:
limits:
cpu: 1500m
memory: 3Gi
requests:
cpu: ".5"
memory: 500Mi
volumeMounts:
- mountPath: /jenkins-home
name: jenkins-home
- mountPath: /backup
name: backupmy PVC
$ k get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
jenkins-backup-pvc Bound pvc-08046980-5102-4e69-9fe4-083e0d9b8140 100Gi RWO enc-gp3 371d
$I tried two times installing with helm, and got same weird error about the PVC, so then I just did this using manifiest and worked and my jenkins pod showed up:
$ helm delete jenkins
release "jenkins" uninstalled
$ kubectl apply -f https://raw.githubusercontent.com/jenkinsci/kubernetes-operator/master/config/crd/bases/jenkins.io_jenkins.yaml
Warning: resource customresourcedefinitions/jenkins.jenkins.io is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
customresourcedefinition.apiextensions.k8s.io/jenkins.jenkins.io configured
$ kubectl apply -f https://raw.githubusercontent.com/jenkinsci/kubernetes-operator/master/deploy/all-in-one-v1alpha2.yaml
serviceaccount/jenkins-operator created
role.rbac.authorization.k8s.io/leader-election-role created
rolebinding.rbac.authorization.k8s.io/leader-election-rolebinding created
role.rbac.authorization.k8s.io/jenkins-operator created
rolebinding.rbac.authorization.k8s.io/manager-rolebinding created
deployment.apps/jenkins-operator createdAfter this was installed i edited manually the operator deployment to update the image to the new one on quay.io registry.
Thanks again for looking into it :)
ingineru
commented
1 year ago I stumbled upon the same issue. From what I can tell, whatever value is provided for LatestPlugins, the jenkins-plugin-cli always takes "true" as value for the --latest argument.
brokenpip3
commented
1 year ago yes that's what I modified, if you want to avoid that you need to set latestPlugins to false: https://github.com/jenkinsci/kubernetes-operator/blob/3fe842fff4bb910278a905af8f9171e77a9cd313/chart/jenkins-operator/values.yaml#L120-L124
ingineru
commented
1 year ago This is the change:
defaultlatestPlugin := true
latestP := jenkins.Spec.Master.LatestPlugins
if !latestP {
latestP = defaultlatestPlugin
}This is what happens
defaultlatestPlugin = true
case 1: latestP = false
the if statement is true => latestP = defaultlatestPlugin = true
case 2: latestP = true
the if statement is false => no changes => latestP = trueNo matter which value I specify for latestPlugins, latestP will end up being always true.
Unless if !latestP means something other than "if not(latestP value)" in go (which I do not know).
Harguer
commented
1 year ago Yeah seems that is not working, even when I set this to false, still complaining about new plugins and my jenkins is breaking
$ k get jenkins -o yaml | grep latestPlugins | tail -1
latestPlugins: false
$ kubectl-image | grep jenkins-operator
jenkins-operator-6f8c5f6dbc-4x7f5 quay.io/jenkins-kubernetes-operator/operator:latestAfter this was installed i edited manually the operator deployment to update the image to the new one on quay.io registry.
see https://github.com/jenkinsci/kubernetes-operator/issues/827
Yeah seems that is not working, even when I set this to false, still complaining about new plugins and my jenkins is breaking
What you mean by still complaining? if you see the logs in the operator I changed the logic, it's just a warning without restarting jenkins.
my jenkins is breaking
can you elaborate? what happened? please share the logs, the events, the jenkins crd config etc otherwise will be hard to understand.
brokenpip3
commented
1 year ago No matter which value I specify for latestPlugins, latestP will end up being always true.
Yes you are right, will fix it soon
Harguer
commented
1 year ago Hi @brokenpip3 , thanks for getting back and for your help with this :). I'm sorry I did not include any errors because it seems the same issue i reported before, but here are the latest messages / errors I'm getting Note: I killed on porpuse this jenkins that was running for 20 days, and it could not start, this is the same issue i have been having with jenkins where if it get restartd/killed it won't start untill someone fix the plugin issues, this is a big problem because it this happens at nights or weeknds reaction time is not same as when we are at office.
jenkins configs:
master:
basePlugins:
- name: kubernetes
version: 3923.v294a_d4250b_91
- name: workflow-job
version: 1295.v395eb_7400005
- name: workflow-aggregator
version: 596.v8c21c963d92d
- name: git
version: 5.0.2
- name: job-dsl
version: "1.83"
- name: configuration-as-code
version: 1625.v27444588cc3d
- name: kubernetes-credentials-provider
version: 1.211.vc236a_f5a_2f3c
...
...
...
latestPlugins: false
plugins:
- name: appscan
version: 1.0.9
- name: aws-credentials
version: "1.32"
- name: aws-secrets-manager-credentials-provider
version: 1.202.ve0ec0c17611c
- name: credentials-binding
version: 604.vb_64480b_c56ca_
- name: envinject
version: 2.4.0
- name: github
version: 1.34.1
- name: ldap
version: "2.7"
- name: role-strategy
version: 3.2.0
- name: throttle-concurrents
version: "2.5"
- name: workflow-multibranch
version: 746.v05814d19c001
- name: ws-cleanup
version: "0.39"
- name: kubernetes-client-api
version: 6.4.1-215.v2ed17097a_8e9
- name: kubernetes-credentials
version: 0.10.0
- name: groovy
version: 453.vcdb_a_c5c99890
- name: workflow-cps
version: 3659.v582dc37621d8
- name: pipeline-groovy-lib
version: 656.va_a_ceeb_6ffb_f7
- name: ionicons-api
version: 45.vf54fca_5d2154
- name: ssh
version: 2.6.1
- name: parameterized-trigger
version: 2.43.1
- name: matrix-project
version: 789.v57a_725b_63c79
- name: matrix-auth
version: 2.6.11
- name: aws-java-sdk
version: 1.12.287-357.vf82d85a_6eefd
- name: ansicolor
version: 1.0.1
- name: saml
version: 1.1.8
- name: nexus-artifact-uploader
version: "2.13"
- name: artifactory
version: 3.17.0
- name: splunk-devops
version: 1.10.1
securityContext:
fsGroup: 1000
runAsUser: 1000
...
...Jenkins-master container, it breaks/restarts every 20 seconds or so:
$ k get pods
NAME READY STATUS RESTARTS AGE
jenkins-jenkins 1/2 Terminating 0 21s
jenkins-operator-6f8c5f6dbc-4x7f5 1/1 Running 0 20d
seed-job-agent-jenkins-644f9cb4b-rnwfk 1/1 Running 0 20d
$jenkins-master container:
Will install new plugin display-url-api 2.3.7
Will install new plugin commons-lang3-api 3.12.0-36.vd97de6465d5b_
Will install new plugin script-security 1251.vfe552ed55f8d
Will install new plugin branch-api 2.1109.vdf225489a_16d
Will install new plugin workflow-api 1213.v646def1087f9
io.jenkins.tools.pluginmanager.impl.AggregatePluginPrerequisitesNotMetException: Multiple plugin prerequisites not met:
Plugin workflow-aggregator:596.v8c21c963d92d (via workflow-cps:3691.v28b_14c465a_b_b_) depends on git:5.1.0, but there is an older version defined on the top level - git:5.0.2,
Plugin workflow-aggregator:596.v8c21c963d92d (via workflow-multibranch:756.v891d88f2cd46) depends on workflow-job:1308.v58d48a_763b_31, but there is an older version defined on the top level - workflow-job:1295.v395eb_7400005,
Plugin workflow-aggregator:596.v8c21c963d92d (via workflow-cps:3691.v28b_14c465a_b_b_->git:5.1.0) depends on configuration-as-code:1647.ve39ca_b_829b_42, but there is an older version defined on the top level - configuration-as-code:1625.v27444588cc3d,
Plugin workflow-aggregator:596.v8c21c963d92d (via pipeline-model-definition:2.2141.v5402e818a_779->git-client:4.4.0) depends on configuration-as-code:1647.ve39ca_b_829b_42, but there is an older version defined on the top level - configuration-as-code:1625.v27444588cc3d,
Plugin workflow-aggregator:596.v8c21c963d92d (via pipeline-model-definition:2.2141.v5402e818a_779->pipeline-model-extensions:2.2141.v5402e818a_779) depends on workflow-job:1308.v58d48a_763b_31, but there is an older version defined on the top level - workflow-job:1295.v395eb_7400005,
Plugin git:5.0.2 (via git-client:4.4.0) depends on configuration-as-code:1647.ve39ca_b_829b_42, but there is an older version defined on the top level - configuration-as-code:1625.v27444588cc3d
at io.jenkins.tools.pluginmanager.impl.PluginManager.start(PluginManager.java:240)
at io.jenkins.tools.pluginmanager.impl.PluginManager.start(PluginManager.java:189)
at io.jenkins.tools.pluginmanager.cli.Main.main(Main.java:52)
Suppressed: io.jenkins.tools.pluginmanager.impl.PluginDependencyException: Plugin workflow-aggregator:596.v8c21c963d92d (via workflow-cps:3691.v28b_14c465a_b_b_) depends on git:5.1.0, but there is an older version defined on the top level - git:5.0.2
at io.jenkins.tools.pluginmanager.impl.PluginManager.resolveRecursiveDependencies(PluginManager.java:1122)
at io.jenkins.tools.pluginmanager.impl.PluginManager.findPluginsAndDependencies(PluginManager.java:692)
at io.jenkins.tools.pluginmanager.impl.PluginManager.start(PluginManager.java:232)
... 2 more
Suppressed: io.jenkins.tools.pluginmanager.impl.PluginDependencyException: Plugin workflow-aggregator:596.v8c21c963d92d (via workflow-multibranch:756.v891d88f2cd46) depends on workflow-job:1308.v58d48a_763b_31, but there is an older version defined on the top level - workflow-job:1295.v395eb_7400005
at io.jenkins.tools.pluginmanager.impl.PluginManager.resolveRecursiveDependencies(PluginManager.java:1122)
at io.jenkins.tools.pluginmanager.impl.PluginManager.findPluginsAndDependencies(PluginManager.java:692)
at io.jenkins.tools.pluginmanager.impl.PluginManager.start(PluginManager.java:232)
... 2 more
Suppressed: io.jenkins.tools.pluginmanager.impl.PluginDependencyException: Plugin workflow-aggregator:596.v8c21c963d92d (via workflow-cps:3691.v28b_14c465a_b_b_->git:5.1.0) depends on configuration-as-code:1647.ve39ca_b_829b_42, but there is an older version defined on the top level - configuration-as-code:1625.v27444588cc3d
at io.jenkins.tools.pluginmanager.impl.PluginManager.resolveRecursiveDependencies(PluginManager.java:1122)
at io.jenkins.tools.pluginmanager.impl.PluginManager.findPluginsAndDependencies(PluginManager.java:692)
at io.jenkins.tools.pluginmanager.impl.PluginManager.start(PluginManager.java:232)
... 2 more
Suppressed: io.jenkins.tools.pluginmanager.impl.PluginDependencyException: Plugin workflow-aggregator:596.v8c21c963d92d (via pipeline-model-definition:2.2141.v5402e818a_779->git-client:4.4.0) depends on configuration-as-code:1647.ve39ca_b_829b_42, but there is an older version defined on the top level - configuration-as-code:1625.v27444588cc3d
at io.jenkins.tools.pluginmanager.impl.PluginManager.resolveRecursiveDependencies(PluginManager.java:1122)
at io.jenkins.tools.pluginmanager.impl.PluginManager.findPluginsAndDependencies(PluginManager.java:692)
at io.jenkins.tools.pluginmanager.impl.PluginManager.start(PluginManager.java:232)
... 2 more
Suppressed: io.jenkins.tools.pluginmanager.impl.PluginDependencyException: Plugin workflow-aggregator:596.v8c21c963d92d (via pipeline-model-definition:2.2141.v5402e818a_779->pipeline-model-extensions:2.2141.v5402e818a_779) depends on workflow-job:1308.v58d48a_763b_31, but there is an older version defined on the top level - workflow-job:1295.v395eb_7400005
at io.jenkins.tools.pluginmanager.impl.PluginManager.resolveRecursiveDependencies(PluginManager.java:1122)
at io.jenkins.tools.pluginmanager.impl.PluginManager.findPluginsAndDependencies(PluginManager.java:692)
at io.jenkins.tools.pluginmanager.impl.PluginManager.start(PluginManager.java:232)
... 2 more
Suppressed: io.jenkins.tools.pluginmanager.impl.PluginDependencyException: Plugin git:5.0.2 (via git-client:4.4.0) depends on configuration-as-code:1647.ve39ca_b_829b_42, but there is an older version defined on the top level - configuration-as-code:1625.v27444588cc3d
at io.jenkins.tools.pluginmanager.impl.PluginManager.resolveRecursiveDependencies(PluginManager.java:1122)
at io.jenkins.tools.pluginmanager.impl.PluginManager.findPluginsAndDependencies(PluginManager.java:692)
at io.jenkins.tools.pluginmanager.impl.PluginManager.start(PluginManager.java:232)
... 2 more
Multiple plugin prerequisites not met:
Plugin workflow-aggregator:596.v8c21c963d92d (via workflow-cps:3691.v28b_14c465a_b_b_) depends on git:5.1.0, but there is an older version defined on the top level - git:5.0.2,
Plugin workflow-aggregator:596.v8c21c963d92d (via workflow-multibranch:756.v891d88f2cd46) depends on workflow-job:1308.v58d48a_763b_31, but there is an older version defined on the top level - workflow-job:1295.v395eb_7400005,
Plugin workflow-aggregator:596.v8c21c963d92d (via workflow-cps:3691.v28b_14c465a_b_b_->git:5.1.0) depends on configuration-as-code:1647.ve39ca_b_829b_42, but there is an older version defined on the top level - configuration-as-code:1625.v27444588cc3d,
Plugin workflow-aggregator:596.v8c21c963d92d (via pipeline-model-definition:2.2141.v5402e818a_779->git-client:4.4.0) depends on configuration-as-code:1647.ve39ca_b_829b_42, but there is an older version defined on the top level - configuration-as-code:1625.v27444588cc3d,
Plugin workflow-aggregator:596.v8c21c963d92d (via pipeline-model-definition:2.2141.v5402e818a_779->pipeline-model-extensions:2.2141.v5402e818a_779) depends on workflow-job:1308.v58d48a_763b_31, but there is an older version defined on the top level - workflow-job:1295.v395eb_7400005,
Plugin git:5.0.2 (via git-client:4.4.0) depends on configuration-as-code:1647.ve39ca_b_829b_42, but there is an older version defined on the top level - configuration-as-code:1625.v27444588cc3dJenkins-Operator, there is no so much information about the error, just endless logs like this:
2023-06-12T15:01:29.150Z DEBUG controller-jenkins Jenkins master pod is present {"cr": "jenkins"}
2023-06-12T15:01:29.150Z INFO controller-jenkins Container 'jenkins-master' is terminated, status '{Name:jenkins-master State:{Waiting:nil Running:nil Terminated:&ContainerStateTerminated{ExitCode:1,Signal:0,Reason:Error,Message:,StartedAt:2023-06-12 15:01:22 +0000 UTC,FinishedAt:2023-06-12 15:01:28 +0000 UTC,ContainerID:docker://becbbc420dbbdb52ce8aa4116a6ccb98b7dcd1a29ff91d907f184ca3e58e3dc6,}} LastTerminationState:{Waiting:nil Running:nil Terminated:nil} Ready:false RestartCount:0 Image:jenkins/jenkins:2.387.3-lts ImageID:docker-pullable://jenkins/jenkins@sha256:2f47c4f27526dcf8c76b269ff1b98d78df47d7c5370ea133fe371f156abf0f59 ContainerID:docker://becbbc420dbbdb52ce8aa4116a6ccb98b7dcd1a29ff91d907f184ca3e58e3dc6 Started:0xc000c4a9c6}' {"cr": "jenkins"}
2023-06-12T15:01:29.163Z DEBUG controller-jenkins Reconciling Jenkins {"cr": "jenkins"}
2023-06-12T15:01:29.163Z DEBUG controller-jenkins Operator credentials secret is present {"cr": "jenkins"}
2023-06-12T15:01:29.192Z DEBUG controller-jenkins Scripts config map is present {"cr": "jenkins"}
2023-06-12T15:01:29.205Z DEBUG controller-jenkins Init configuration config map is present {"cr": "jenkins"}
2023-06-12T15:01:29.231Z DEBUG controller-jenkins Base configuration config map is present {"cr": "jenkins"}
2023-06-12T15:01:29.232Z DEBUG controller-jenkins GroovyScripts Secret and ConfigMap added watched labels {"cr": "jenkins"}
2023-06-12T15:01:29.232Z DEBUG controller-jenkins ConfigurationAsCode Secret and ConfigMap added watched labels {"cr": "jenkins"}
2023-06-12T15:01:29.232Z DEBUG controller-jenkins createServiceAccount with annotations map[] {"cr": "jenkins"}
2023-06-12T15:01:29.361Z DEBUG controller-jenkins Service account, role and role binding are present {"cr": "jenkins"}
2023-06-12T15:01:29.361Z DEBUG controller-jenkins Extra role bindings are present {"cr": "jenkins"}
2023-06-12T15:01:29.368Z DEBUG controller-jenkins Jenkins HTTP Service is present {"cr": "jenkins"}
2023-06-12T15:01:29.373Z DEBUG controller-jenkins Jenkins slave Service is present {"cr": "jenkins"}
2023-06-12T15:01:29.373Z DEBUG controller-jenkins Kubernetes resources are present {"cr": "jenkins"}
2023-06-12T15:01:29.373Z DEBUG controller-jenkins Jenkins master pod is present {"cr": "jenkins"}
2023-06-12T15:01:29.373Z DEBUG controller-jenkins Jenkins master pod is terminating {"cr": "jenkins"}
2023-06-12T15:01:29.373Z DEBUG controller-jenkins Reconciling Jenkins {"cr": "jenkins"}
2023-06-12T15:01:29.374Z DEBUG controller-jenkins Operator credentials secret is present {"cr": "jenkins"}
2023-06-12T15:01:29.390Z DEBUG controller-jenkins Scripts config map is present {"cr": "jenkins"}
2023-06-12T15:01:29.402Z DEBUG controller-jenkins Init configuration config map is present {"cr": "jenkins"}
2023-06-12T15:01:29.427Z DEBUG controller-jenkins Base configuration config map is present {"cr": "jenkins"}
2023-06-12T15:01:29.427Z DEBUG controller-jenkins GroovyScripts Secret and ConfigMap added watched labels {"cr": "jenkins"}
2023-06-12T15:01:29.427Z DEBUG controller-jenkins ConfigurationAsCode Secret and ConfigMap added watched labels {"cr": "jenkins"}
2023-06-12T15:01:29.427Z DEBUG controller-jenkins createServiceAccount with annotations map[] {"cr": "jenkins"}
2023-06-12T15:01:29.556Z DEBUG controller-jenkins Service account, role and role binding are present {"cr": "jenkins"}
2023-06-12T15:01:29.556Z DEBUG controller-jenkins Extra role bindings are present {"cr": "jenkins"}
2023-06-12T15:01:29.563Z DEBUG controller-jenkins Jenkins HTTP Service is present {"cr": "jenkins"}
2023-06-12T15:01:29.570Z DEBUG controller-jenkins Jenkins slave Service is present {"cr": "jenkins"}
2023-06-12T15:01:29.570Z DEBUG controller-jenkins Kubernetes resources are present {"cr": "jenkins"}
2023-06-12T15:01:29.570Z DEBUG controller-jenkins Jenkins master pod is present {"cr": "jenkins"}
2023-06-12T15:01:29.570Z DEBUG controller-jenkins Jenkins master pod is terminating {"cr": "jenkins"}
2023-06-12T15:01:34.373Z DEBUG controller-jenkins Reconciling Jenkins {"cr": "jenkins"}
2023-06-12T15:01:34.374Z DEBUG controller-jenkins Operator credentials secret is present {"cr": "jenkins"}
2023-06-12T15:01:34.393Z DEBUG controller-jenkins Scripts config map is present {"cr": "jenkins"}
2023-06-12T15:01:34.409Z DEBUG controller-jenkins Init configuration config map is present {"cr": "jenkins"}
2023-06-12T15:01:34.435Z DEBUG controller-jenkins Base configuration config map is present {"cr": "jenkins"}
2023-06-12T15:01:34.435Z DEBUG controller-jenkins GroovyScripts Secret and ConfigMap added watched labels {"cr": "jenkins"}
2023-06-12T15:01:34.435Z DEBUG controller-jenkins ConfigurationAsCode Secret and ConfigMap added watched labels {"cr": "jenkins"}
2023-06-12T15:01:34.435Z DEBUG controller-jenkins createServiceAccount with annotations map[] {"cr": "jenkins"}
2023-06-12T15:01:34.631Z DEBUG controller-jenkins Service account, role and role binding are present {"cr": "jenkins"}
2023-06-12T15:01:34.631Z DEBUG controller-jenkins Extra role bindings are present {"cr": "jenkins"}
2023-06-12T15:01:34.643Z DEBUG controller-jenkins Jenkins HTTP Service is present {"cr": "jenkins"}
2023-06-12T15:01:34.649Z DEBUG controller-jenkins Jenkins slave Service is present {"cr": "jenkins"}
2023-06-12T15:01:34.649Z DEBUG controller-jenkins Kubernetes resources are present {"cr": "jenkins"}
2023-06-12T15:01:34.649Z DEBUG controller-jenkins Jenkins master pod is present {"cr": "jenkins"}
2023-06-12T15:01:34.649Z DEBUG controller-jenkins Jenkins master pod is terminating {"cr": "jenkins"}
2023-06-12T15:01:39.649Z DEBUG controller-jenkins Reconciling Jenkins {"cr": "jenkins"}
2023-06-12T15:01:39.650Z DEBUG controller-jenkins Operator credentials secret is present {"cr": "jenkins"}
2023-06-12T15:01:39.675Z DEBUG controller-jenkins Scripts config map is present {"cr": "jenkins"}
2023-06-12T15:01:39.688Z DEBUG controller-jenkins Init configuration config map is present {"cr": "jenkins"}
2023-06-12T15:01:39.713Z DEBUG controller-jenkins Base configuration config map is present {"cr": "jenkins"}
2023-06-12T15:01:39.713Z DEBUG controller-jenkins GroovyScripts Secret and ConfigMap added watched labels {"cr": "jenkins"}
2023-06-12T15:01:39.713Z DEBUG controller-jenkins ConfigurationAsCode Secret and ConfigMap added watched labels {"cr": "jenkins"}
2023-06-12T15:01:39.713Z DEBUG controller-jenkins createServiceAccount with annotations map[] {"cr": "jenkins"}
2023-06-12T15:01:39.842Z DEBUG controller-jenkins Service account, role and role binding are present {"cr": "jenkins"}
2023-06-12T15:01:39.842Z DEBUG controller-jenkins Extra role bindings are present {"cr": "jenkins"}
2023-06-12T15:01:39.849Z DEBUG controller-jenkins Jenkins HTTP Service is present {"cr": "jenkins"}
2023-06-12T15:01:39.854Z DEBUG controller-jenkins Jenkins slave Service is present {"cr": "jenkins"}
2023-06-12T15:01:39.854Z DEBUG controller-jenkins Kubernetes resources are present {"cr": "jenkins"}
2023-06-12T15:01:39.854Z DEBUG controller-jenkins Jenkins master pod is present {"cr": "jenkins"}
2023-06-12T15:01:39.854Z DEBUG controller-jenkins Jenkins master pod is terminating {"cr": "jenkins"}
2023-06-12T15:01:44.854Z DEBUG controller-jenkins Reconciling Jenkins {"cr": "jenkins"}
2023-06-12T15:01:44.855Z DEBUG controller-jenkins Operator credentials secret is present {"cr": "jenkins"}
2023-06-12T15:01:44.873Z DEBUG controller-jenkins Scripts config map is present {"cr": "jenkins"}
2023-06-12T15:01:44.887Z DEBUG controller-jenkins Init configuration config map is present {"cr": "jenkins"}
2023-06-12T15:01:44.909Z DEBUG controller-jenkins Base configuration config map is present {"cr": "jenkins"}
2023-06-12T15:01:44.909Z DEBUG controller-jenkins GroovyScripts Secret and ConfigMap added watched labels {"cr": "jenkins"}
2023-06-12T15:01:44.909Z DEBUG controller-jenkins ConfigurationAsCode Secret and ConfigMap added watched labels {"cr": "jenkins"}
2023-06-12T15:01:44.909Z DEBUG controller-jenkins createServiceAccount with annotations map[] {"cr": "jenkins"}
2023-06-12T15:01:45.039Z DEBUG controller-jenkins Service account, role and role binding are present {"cr": "jenkins"}
2023-06-12T15:01:45.039Z DEBUG controller-jenkins Extra role bindings are present {"cr": "jenkins"}
2023-06-12T15:01:45.045Z DEBUG controller-jenkins Jenkins HTTP Service is present {"cr": "jenkins"}
2023-06-12T15:01:45.051Z DEBUG controller-jenkins Jenkins slave Service is present {"cr": "jenkins"}
2023-06-12T15:01:45.051Z DEBUG controller-jenkins Kubernetes resources are present {"cr": "jenkins"}
2023-06-12T15:01:45.051Z DEBUG controller-jenkins Jenkins master pod is present {"cr": "jenkins"}
2023-06-12T15:01:45.051Z DEBUG controller-jenkins Jenkins master pod is terminating {"cr": "jenkins"}
2023-06-12T15:01:50.051Z DEBUG controller-jenkins Reconciling Jenkins {"cr": "jenkins"}
2023-06-12T15:01:50.051Z DEBUG controller-jenkins Operator credentials secret is present {"cr": "jenkins"}
2023-06-12T15:01:50.074Z DEBUG controller-jenkins Scripts config map is present {"cr": "jenkins"}
2023-06-12T15:01:50.088Z DEBUG controller-jenkins Init configuration config map is present {"cr": "jenkins"}
2023-06-12T15:01:50.112Z DEBUG controller-jenkins Base configuration config map is present {"cr": "jenkins"}
2023-06-12T15:01:50.113Z DEBUG controller-jenkins GroovyScripts Secret and ConfigMap added watched labels {"cr": "jenkins"}
2023-06-12T15:01:50.113Z DEBUG controller-jenkins ConfigurationAsCode Secret and ConfigMap added watched labels {"cr": "jenkins"}
2023-06-12T15:01:50.113Z DEBUG controller-jenkins createServiceAccount with annotations map[] {"cr": "jenkins"}
2023-06-12T15:01:50.264Z DEBUG controller-jenkins Service account, role and role binding are present {"cr": "jenkins"}
2023-06-12T15:01:50.264Z DEBUG controller-jenkins Extra role bindings are present {"cr": "jenkins"}
2023-06-12T15:01:50.271Z DEBUG controller-jenkins Jenkins HTTP Service is present {"cr": "jenkins"}
2023-06-12T15:01:50.276Z DEBUG controller-jenkins Jenkins slave Service is present {"cr": "jenkins"}
2023-06-12T15:01:50.276Z DEBUG controller-jenkins Kubernetes resources are present {"cr": "jenkins"}
2023-06-12T15:01:50.276Z DEBUG controller-jenkins Jenkins master pod is present {"cr": "jenkins"}
2023-06-12T15:01:50.276Z DEBUG controller-jenkins Jenkins master pod is terminating {"cr": "jenkins"}
2023-06-12T15:01:55.276Z DEBUG controller-jenkins Reconciling Jenkins {"cr": "jenkins"}
2023-06-12T15:01:55.276Z DEBUG controller-jenkins Operator credentials secret is present {"cr": "jenkins"}
2023-06-12T15:01:55.294Z DEBUG controller-jenkins Scripts config map is present {"cr": "jenkins"}
2023-06-12T15:01:55.308Z DEBUG controller-jenkins Init configuration config map is present {"cr": "jenkins"}
2023-06-12T15:01:55.332Z DEBUG controller-jenkins Base configuration config map is present {"cr": "jenkins"}
2023-06-12T15:01:55.332Z DEBUG controller-jenkins GroovyScripts Secret and ConfigMap added watched labels {"cr": "jenkins"}
2023-06-12T15:01:55.332Z DEBUG controller-jenkins ConfigurationAsCode Secret and ConfigMap added watched labels {"cr": "jenkins"}
2023-06-12T15:01:55.332Z DEBUG controller-jenkins createServiceAccount with annotations map[] {"cr": "jenkins"}
2023-06-12T15:01:55.462Z DEBUG controller-jenkins Service account, role and role binding are present {"cr": "jenkins"}
2023-06-12T15:01:55.462Z DEBUG controller-jenkins Extra role bindings are present {"cr": "jenkins"}
2023-06-12T15:01:55.469Z DEBUG controller-jenkins Jenkins HTTP Service is present {"cr": "jenkins"}
2023-06-12T15:01:55.474Z DEBUG controller-jenkins Jenkins slave Service is present {"cr": "jenkins"}
2023-06-12T15:01:55.474Z DEBUG controller-jenkins Kubernetes resources are present {"cr": "jenkins"}
2023-06-12T15:01:55.474Z DEBUG controller-jenkins Jenkins master pod is present {"cr": "jenkins"}
2023-06-12T15:01:55.474Z DEBUG controller-jenkins Jenkins master pod is terminating {"cr": "jenkins"}
2023-06-12T15:02:00.474Z DEBUG controller-jenkins Reconciling Jenkins {"cr": "jenkins"}
2023-06-12T15:02:00.474Z DEBUG controller-jenkins Operator credentials secret is present {"cr": "jenkins"}
2023-06-12T15:02:00.492Z DEBUG controller-jenkins Scripts config map is present {"cr": "jenkins"}
2023-06-12T15:02:00.565Z DEBUG controller-jenkins Init configuration config map is present {"cr": "jenkins"}
2023-06-12T15:02:00.592Z DEBUG controller-jenkins Base configuration config map is present {"cr": "jenkins"}
2023-06-12T15:02:00.592Z DEBUG controller-jenkins GroovyScripts Secret and ConfigMap added watched labels {"cr": "jenkins"}
2023-06-12T15:02:00.592Z DEBUG controller-jenkins ConfigurationAsCode Secret and ConfigMap added watched labels {"cr": "jenkins"}
2023-06-12T15:02:00.592Z DEBUG controller-jenkins createServiceAccount with annotations map[] {"cr": "jenkins"}
2023-06-12T15:02:00.754Z DEBUG controller-jenkins Service account, role and role binding are present {"cr": "jenkins"}
2023-06-12T15:02:00.754Z DEBUG controller-jenkins Extra role bindings are present {"cr": "jenkins"}
2023-06-12T15:02:00.769Z DEBUG controller-jenkins Jenkins HTTP Service is present {"cr": "jenkins"}
2023-06-12T15:02:00.785Z DEBUG controller-jenkins Jenkins slave Service is present {"cr": "jenkins"}
2023-06-12T15:02:00.785Z DEBUG controller-jenkins Kubernetes resources are present {"cr": "jenkins"}
2023-06-12T15:02:00.785Z DEBUG controller-jenkins Jenkins master pod is present {"cr": "jenkins"}
2023-06-12T15:02:00.786Z DEBUG controller-jenkins Jenkins master pod is terminating {"cr": "jenkins"}
2023-06-12T15:02:02.440Z DEBUG controller-jenkins Reconciling Jenkins {"cr": "jenkins"}
2023-06-12T15:02:02.441Z DEBUG controller-jenkins Operator credentials secret is present {"cr": "jenkins"}
2023-06-12T15:02:02.476Z DEBUG controller-jenkins Scripts config map is present {"cr": "jenkins"}
2023-06-12T15:02:02.497Z DEBUG controller-jenkins Init configuration config map is present {"cr": "jenkins"}
2023-06-12T15:02:02.528Z DEBUG controller-jenkins Base configuration config map is present {"cr": "jenkins"}
2023-06-12T15:02:02.528Z DEBUG controller-jenkins GroovyScripts Secret and ConfigMap added watched labels {"cr": "jenkins"}
2023-06-12T15:02:02.528Z DEBUG controller-jenkins ConfigurationAsCode Secret and ConfigMap added watched labels {"cr": "jenkins"}
2023-06-12T15:02:02.528Z DEBUG controller-jenkins createServiceAccount with annotations map[] {"cr": "jenkins"}
2023-06-12T15:02:02.714Z DEBUG controller-jenkins Service account, role and role binding are present {"cr": "jenkins"}
2023-06-12T15:02:02.714Z DEBUG controller-jenkins Extra role bindings are present {"cr": "jenkins"}
2023-06-12T15:02:02.723Z DEBUG controller-jenkins Jenkins HTTP Service is present {"cr": "jenkins"}
2023-06-12T15:02:02.728Z DEBUG controller-jenkins Jenkins slave Service is present {"cr": "jenkins"}
2023-06-12T15:02:02.728Z DEBUG controller-jenkins Kubernetes resources are present {"cr": "jenkins"}
2023-06-12T15:02:02.728Z INFO controller-jenkins Creating a new Jenkins Master Pod jenkins/jenkins-jenkins {"cr": "jenkins"}
2023-06-12T15:02:02.788Z DEBUG controller-jenkins Reconciling Jenkins {"cr": "jenkins"}
2023-06-12T15:02:02.788Z DEBUG controller-jenkins Operator credentials secret is present {"cr": "jenkins"}
2023-06-12T15:02:02.813Z DEBUG controller-jenkins Scripts config map is present {"cr": "jenkins"}
2023-06-12T15:02:02.831Z DEBUG controller-jenkins Init configuration config map is present {"cr": "jenkins"}
2023-06-12T15:02:02.854Z DEBUG controller-jenkins Base configuration config map is present {"cr": "jenkins"}
2023-06-12T15:02:02.854Z DEBUG controller-jenkins GroovyScripts Secret and ConfigMap added watched labels {"cr": "jenkins"}
2023-06-12T15:02:02.854Z DEBUG controller-jenkins ConfigurationAsCode Secret and ConfigMap added watched labels {"cr": "jenkins"}
2023-06-12T15:02:02.854Z DEBUG controller-jenkins createServiceAccount with annotations map[] {"cr": "jenkins"}
2023-06-12T15:02:02.989Z DEBUG controller-jenkins Service account, role and role binding are present {"cr": "jenkins"}
2023-06-12T15:02:02.989Z DEBUG controller-jenkins Extra role bindings are present {"cr": "jenkins"}
2023-06-12T15:02:02.996Z DEBUG controller-jenkins Jenkins HTTP Service is present {"cr": "jenkins"}
2023-06-12T15:02:03.001Z DEBUG controller-jenkins Jenkins slave Service is present {"cr": "jenkins"}
2023-06-12T15:02:03.001Z DEBUG controller-jenkins Kubernetes resources are present {"cr": "jenkins"}
2023-06-12T15:02:03.001Z DEBUG controller-jenkins Jenkins master pod is present {"cr": "jenkins"}
2023-06-12T15:02:03.001Z DEBUG controller-jenkins Jenkins master pod not ready {"cr": "jenkins"}
2023-06-12T15:02:03.001Z DEBUG controller-jenkins Reconciling Jenkins {"cr": "jenkins"}
2023-06-12T15:02:03.001Z DEBUG controller-jenkins Operator credentials secret is present {"cr": "jenkins"}
2023-06-12T15:02:03.017Z DEBUG controller-jenkins Scripts config map is present {"cr": "jenkins"}
2023-06-12T15:02:03.030Z DEBUG controller-jenkins Init configuration config map is present {"cr": "jenkins"}
2023-06-12T15:02:03.054Z DEBUG controller-jenkins Base configuration config map is present {"cr": "jenkins"}
2023-06-12T15:02:03.054Z DEBUG controller-jenkins GroovyScripts Secret and ConfigMap added watched labels {"cr": "jenkins"}
2023-06-12T15:02:03.054Z DEBUG controller-jenkins ConfigurationAsCode Secret and ConfigMap added watched labels {"cr": "jenkins"}
2023-06-12T15:02:03.054Z DEBUG controller-jenkins createServiceAccount with annotations map[] {"cr": "jenkins"}
2023-06-12T15:02:03.179Z DEBUG controller-jenkins Service account, role and role binding are present {"cr": "jenkins"}
2023-06-12T15:02:03.180Z DEBUG controller-jenkins Extra role bindings are present {"cr": "jenkins"}
2023-06-12T15:02:03.186Z DEBUG controller-jenkins Jenkins HTTP Service is present {"cr": "jenkins"}
2023-06-12T15:02:03.191Z DEBUG controller-jenkins Jenkins slave Service is present {"cr": "jenkins"}
2023-06-12T15:02:03.191Z DEBUG controller-jenkins Kubernetes resources are present {"cr": "jenkins"}
2023-06-12T15:02:03.191Z DEBUG controller-jenkins Jenkins master pod is present {"cr": "jenkins"}
2023-06-12T15:02:03.191Z DEBUG controller-jenkins Jenkins master pod not ready {"cr": "jenkins"}
2023-06-12T15:02:08.001Z DEBUG controller-jenkins Reconciling Jenkins {"cr": "jenkins"}
2023-06-12T15:02:08.002Z DEBUG controller-jenkins Operator credentials secret is present {"cr": "jenkins"}
2023-06-12T15:02:08.020Z DEBUG controller-jenkins Scripts config map is present {"cr": "jenkins"}
2023-06-12T15:02:08.034Z DEBUG controller-jenkins Init configuration config map is present {"cr": "jenkins"}
2023-06-12T15:02:08.058Z DEBUG controller-jenkins Base configuration config map is present {"cr": "jenkins"}
2023-06-12T15:02:08.058Z DEBUG controller-jenkins GroovyScripts Secret and ConfigMap added watched labels {"cr": "jenkins"}
2023-06-12T15:02:08.058Z DEBUG controller-jenkins ConfigurationAsCode Secret and ConfigMap added watched labels {"cr": "jenkins"}
2023-06-12T15:02:08.058Z DEBUG controller-jenkins createServiceAccount with annotations map[] {"cr": "jenkins"}
2023-06-12T15:02:08.208Z DEBUG controller-jenkins Service account, role and role binding are present {"cr": "jenkins"}
2023-06-12T15:02:08.208Z DEBUG controller-jenkins Extra role bindings are present {"cr": "jenkins"}
2023-06-12T15:02:08.214Z DEBUG controller-jenkins Jenkins HTTP Service is present {"cr": "jenkins"}
2023-06-12T15:02:08.219Z DEBUG controller-jenkins Jenkins slave Service is present {"cr": "jenkins"}
2023-06-12T15:02:08.219Z DEBUG controller-jenkins Kubernetes resources are present {"cr": "jenkins"}
2023-06-12T15:02:08.219Z DEBUG controller-jenkins Jenkins master pod is present {"cr": "jenkins"}
2023-06-12T15:02:08.219Z DEBUG controller-jenkins Jenkins master pod not ready {"cr": "jenkins"}
2023-06-12T15:02:13.219Z DEBUG controller-jenkins Reconciling Jenkins {"cr": "jenkins"}
2023-06-12T15:02:13.219Z DEBUG controller-jenkins Operator credentials secret is present {"cr": "jenkins"}
2023-06-12T15:02:13.237Z DEBUG controller-jenkins Scripts config map is present {"cr": "jenkins"}
2023-06-12T15:02:13.250Z DEBUG controller-jenkins Init configuration config map is present {"cr": "jenkins"}
2023-06-12T15:02:13.275Z DEBUG controller-jenkins Base configuration config map is present {"cr": "jenkins"}
2023-06-12T15:02:13.275Z DEBUG controller-jenkins GroovyScripts Secret and ConfigMap added watched labels {"cr": "jenkins"}
2023-06-12T15:02:13.275Z DEBUG controller-jenkins ConfigurationAsCode Secret and ConfigMap added watched labels {"cr": "jenkins"}
2023-06-12T15:02:13.275Z DEBUG controller-jenkins createServiceAccount with annotations map[] {"cr": "jenkins"}
2023-06-12T15:02:13.425Z DEBUG controller-jenkins Service account, role and role binding are present {"cr": "jenkins"}
2023-06-12T15:02:13.425Z DEBUG controller-jenkins Extra role bindings are present {"cr": "jenkins"}
2023-06-12T15:02:13.430Z DEBUG controller-jenkins Jenkins HTTP Service is present {"cr": "jenkins"}
2023-06-12T15:02:13.435Z DEBUG controller-jenkins Jenkins slave Service is present {"cr": "jenkins"}
2023-06-12T15:02:13.435Z DEBUG controller-jenkins Kubernetes resources are present {"cr": "jenkins"}
2023-06-12T15:02:13.436Z DEBUG controller-jenkins Jenkins master pod is present {"cr": "jenkins"}
2023-06-12T15:02:13.436Z DEBUG controller-jenkins Jenkins master pod not ready {"cr": "jenkins"}
2023-06-12T15:02:18.436Z DEBUG controller-jenkins Reconciling Jenkins {"cr": "jenkins"}
2023-06-12T15:02:18.436Z DEBUG controller-jenkins Operator credentials secret is present {"cr": "jenkins"}
2023-06-12T15:02:18.462Z DEBUG controller-jenkins Scripts config map is present {"cr": "jenkins"}
2023-06-12T15:02:18.475Z DEBUG controller-jenkins Init configuration config map is present {"cr": "jenkins"}
2023-06-12T15:02:18.580Z DEBUG controller-jenkins Base configuration config map is present {"cr": "jenkins"}
2023-06-12T15:02:18.580Z DEBUG controller-jenkins GroovyScripts Secret and ConfigMap added watched labels {"cr": "jenkins"}
2023-06-12T15:02:18.580Z DEBUG controller-jenkins ConfigurationAsCode Secret and ConfigMap added watched labels {"cr": "jenkins"}
2023-06-12T15:02:18.580Z DEBUG controller-jenkins createServiceAccount with annotations map[] {"cr": "jenkins"}
2023-06-12T15:02:18.717Z DEBUG controller-jenkins Service account, role and role binding are present {"cr": "jenkins"}
2023-06-12T15:02:18.717Z DEBUG controller-jenkins Extra role bindings are present {"cr": "jenkins"}
2023-06-12T15:02:18.727Z DEBUG controller-jenkins Jenkins HTTP Service is present {"cr": "jenkins"}
2023-06-12T15:02:18.742Z DEBUG controller-jenkins Jenkins slave Service is present {"cr": "jenkins"}
2023-06-12T15:02:18.742Z DEBUG controller-jenkins Kubernetes resources are present {"cr": "jenkins"}
2023-06-12T15:02:18.742Z DEBUG controller-jenkins Jenkins master pod is present {"cr": "jenkins"}
2023-06-12T15:02:18.742Z DEBUG controller-jenkins Jenkins master pod not ready {"cr": "jenkins"}
2023-06-12T15:02:21.303Z DEBUG controller-jenkins Reconciling Jenkins {"cr": "jenkins"}
2023-06-12T15:02:21.303Z DEBUG controller-jenkins Operator credentials secret is present {"cr": "jenkins"}
2023-06-12T15:02:21.345Z DEBUG controller-jenkins Scripts config map is present {"cr": "jenkins"}
2023-06-12T15:02:21.359Z DEBUG controller-jenkins Init configuration config map is present {"cr": "jenkins"}
2023-06-12T15:02:21.382Z DEBUG controller-jenkins Base configuration config map is present {"cr": "jenkins"}
2023-06-12T15:02:21.382Z DEBUG controller-jenkins GroovyScripts Secret and ConfigMap added watched labels {"cr": "jenkins"}
2023-06-12T15:02:21.382Z DEBUG controller-jenkins ConfigurationAsCode Secret and ConfigMap added watched labels {"cr": "jenkins"}
2023-06-12T15:02:21.382Z DEBUG controller-jenkins createServiceAccount with annotations map[] {"cr": "jenkins"}
2023-06-12T15:02:21.524Z DEBUG controller-jenkins Service account, role and role binding are present {"cr": "jenkins"}
2023-06-12T15:02:21.524Z DEBUG controller-jenkins Extra role bindings are present {"cr": "jenkins"}
2023-06-12T15:02:21.533Z DEBUG controller-jenkins Jenkins HTTP Service is present {"cr": "jenkins"}
2023-06-12T15:02:21.539Z DEBUG controller-jenkins Jenkins slave Service is present {"cr": "jenkins"}
2023-06-12T15:02:21.539Z DEBUG controller-jenkins Kubernetes resources are present {"cr": "jenkins"}
2023-06-12T15:02:21.539Z DEBUG controller-jenkins Jenkins master pod is present {"cr": "jenkins"}
2023-06-12T15:02:21.539Z DEBUG controller-jenkins Container 'jenkins-master' not ready, readiness probe failed {"cr": "jenkins"}
2023-06-12T15:02:23.742Z DEBUG controller-jenkins Reconciling Jenkins {"cr": "jenkins"}
2023-06-12T15:02:23.742Z DEBUG controller-jenkins Operator credentials secret is present {"cr": "jenkins"}
2023-06-12T15:02:23.759Z DEBUG controller-jenkins Scripts config map is present {"cr": "jenkins"}
2023-06-12T15:02:23.773Z DEBUG controller-jenkins Init configuration config map is present {"cr": "jenkins"}
2023-06-12T15:02:23.796Z DEBUG controller-jenkins Base configuration config map is present {"cr": "jenkins"}
2023-06-12T15:02:23.796Z DEBUG controller-jenkins GroovyScripts Secret and ConfigMap added watched labels {"cr": "jenkins"}
2023-06-12T15:02:23.796Z DEBUG controller-jenkins ConfigurationAsCode Secret and ConfigMap added watched labels {"cr": "jenkins"}
2023-06-12T15:02:23.796Z DEBUG controller-jenkins createServiceAccount with annotations map[] {"cr": "jenkins"}
2023-06-12T15:02:23.921Z DEBUG controller-jenkins Service account, role and role binding are present {"cr": "jenkins"}
2023-06-12T15:02:23.922Z DEBUG controller-jenkins Extra role bindings are present {"cr": "jenkins"}
2023-06-12T15:02:23.929Z DEBUG controller-jenkins Jenkins HTTP Service is present {"cr": "jenkins"}
2023-06-12T15:02:23.934Z DEBUG controller-jenkins Jenkins slave Service is present {"cr": "jenkins"}
2023-06-12T15:02:23.934Z DEBUG controller-jenkins Kubernetes resources are present {"cr": "jenkins"}
2023-06-12T15:02:23.934Z DEBUG controller-jenkins Jenkins master pod is present {"cr": "jenkins"}
2023-06-12T15:02:23.934Z DEBUG controller-jenkins Container 'jenkins-master' not ready, readiness probe failed {"cr": "jenkins"}
2023-06-12T15:02:27.169Z DEBUG controller-jenkins Reconciling Jenkins {"cr": "jenkins"}
2023-06-12T15:02:27.169Z DEBUG controller-jenkins Operator credentials secret is present {"cr": "jenkins"}
2023-06-12T15:02:27.186Z DEBUG controller-jenkins Scripts config map is present {"cr": "jenkins"}
2023-06-12T15:02:27.200Z DEBUG controller-jenkins Init configuration config map is present {"cr": "jenkins"}
2023-06-12T15:02:27.223Z DEBUG controller-jenkins Base configuration config map is present {"cr": "jenkins"}
2023-06-12T15:02:27.223Z DEBUG controller-jenkins GroovyScripts Secret and ConfigMap added watched labels {"cr": "jenkins"}
2023-06-12T15:02:27.223Z DEBUG controller-jenkins ConfigurationAsCode Secret and ConfigMap added watched labels {"cr": "jenkins"}
2023-06-12T15:02:27.223Z DEBUG controller-jenkins createServiceAccount with annotations map[] {"cr": "jenkins"}
2023-06-12T15:02:27.346Z DEBUG controller-jenkins Service account, role and role binding are present {"cr": "jenkins"}
2023-06-12T15:02:27.346Z DEBUG controller-jenkins Extra role bindings are present {"cr": "jenkins"}
2023-06-12T15:02:27.352Z DEBUG controller-jenkins Jenkins HTTP Service is present {"cr": "jenkins"}
2023-06-12T15:02:27.358Z DEBUG controller-jenkins Jenkins slave Service is present {"cr": "jenkins"}
2023-06-12T15:02:27.358Z DEBUG controller-jenkins Kubernetes resources are present {"cr": "jenkins"}
2023-06-12T15:02:27.358Z DEBUG controller-jenkins Jenkins master pod is present {"cr": "jenkins"}
2023-06-12T15:02:27.358Z INFO controller-jenkins Container 'jenkins-master' is terminated, status '{Name:jenkins-master State:{Waiting:nil Running:nil Terminated:&ContainerStateTerminated{ExitCode:1,Signal:0,Reason:Error,Message:,StartedAt:2023-06-12 15:02:20 +0000 UTC,FinishedAt:2023-06-12 15:02:25 +0000 UTC,ContainerID:docker://0450e1eb30986fbfa09b2cae0ff390df8064d49019c65b26eb5a732ee4cfa03c,}} LastTerminationState:{Waiting:nil Running:nil Terminated:nil} Ready:false RestartCount:0 Image:jenkins/jenkins:2.387.3-lts ImageID:docker-pullable://jenkins/jenkins@sha256:2f47c4f27526dcf8c76b269ff1b98d78df47d7c5370ea133fe371f156abf0f59 ContainerID:docker://0450e1eb30986fbfa09b2cae0ff390df8064d49019c65b26eb5a732ee4cfa03c Started:0xc0007f4cc6}' {"cr": "jenkins"}
2023-06-12T15:02:27.373Z DEBUG controller-jenkins Reconciling Jenkins {"cr": "jenkins"}
2023-06-12T15:02:27.373Z DEBUG controller-jenkins Operator credentials secret is present {"cr": "jenkins"}
2023-06-12T15:02:27.397Z DEBUG controller-jenkins Scripts config map is present {"cr": "jenkins"}
2023-06-12T15:02:27.420Z DEBUG controller-jenkins Init configuration config map is present {"cr": "jenkins"}
2023-06-12T15:02:27.457Z DEBUG controller-jenkins Base configuration config map is present {"cr": "jenkins"}
2023-06-12T15:02:27.458Z DEBUG controller-jenkins GroovyScripts Secret and ConfigMap added watched labels {"cr": "jenkins"}
2023-06-12T15:02:27.458Z DEBUG controller-jenkins ConfigurationAsCode Secret and ConfigMap added watched labels {"cr": "jenkins"}
2023-06-12T15:02:27.458Z DEBUG controller-jenkins createServiceAccount with annotations map[] {"cr": "jenkins"}
2023-06-12T15:02:27.612Z DEBUG controller-jenkins Service account, role and role binding are present {"cr": "jenkins"}
2023-06-12T15:02:27.612Z DEBUG controller-jenkins Extra role bindings are present {"cr": "jenkins"}thanks again for looking at it and let me know if you need any aditional details/information.
ingineru
commented
1 year ago @Harguer The master pod logs states the incompatibilities and the versions you need in order to fix your issue.
Let's take the first one - Plugin workflow-aggregator:596.v8c21c963d92d (via workflow-cps:3691.v28b_14c465a_b_b_) depends on git:5.1.0, but there is an older version defined on the top level - git:5.0.2,:
Keep doing the same investigation for the remaining base/plugins until there is no error on the master pod. Or just update the full list to the latest available versions (this is what I did). Once you get the instance deployed with your full list of plugins, check the deployment in 1-2 days with one of the options:
For now I could work around this issue by unpinning/commenting the incompatible plugin. Then I was able to re-deploy Jenkins using the same code and I noticed that the latest version (I commented) was automatically installed. I am waiting a couple of days to see how Jenkins will behaves upon a new deletion.
Harguer
commented
1 year ago Hi @ingineru , thank you for looking into it. Yeah I know how to fix this, but this ticket is about to pin plugins or do something from jenkins-operator to work with a set of plugins, so that if my jenkins dies it should get up and running without human intervention.
If this is something won't be fixed, then I think there is no point to have jenkins-operator. As I explained before, sometimes jenkins pods die during weekends or during nights and jenkins is down until someone manually fix it (by doing the same thing you described, that TBH is horrible to do, while your boss and developers are awaiting for it).
This is my jenkins test and I won't fix it manually, rather I would like to have latestPlugins: false working and apply it to this jenkins and all my other prod jenkins, so that i can be free on weekends and night knowing my jenkins is always available after a reboot :).
ingineru
commented
1 year ago @Harguer I was not particularly looking into your case, I am just in a similar situation. For now I keep the plugins updated (add latest versions in code), I let Jenkins manage some dependencies (commented out the ones declared as incompatible) and I use a script to list incompatible plugins in case of a redeploy (deploy a test instance, get the logs, parse them, destroy the instance). This way of working is only worth the effort if a fix is on the way and it carries the risk that Jenkins might not come back up on pod deletion.
UPDATE: Some of the plugins that are found to be incompatible can be excluded from the code and Jenkins will handle them by installing the latest available version (I did it for instance-identity and okhttp-api). I deleted the test instance master after several days and the automatic redeployment was successful. This is worth trying as a workaround (cc: @Harguer)
vlad-ivanov-name
commented
1 year ago I was able to successfully deploy by overriding the list of base plugins with a list generated by jenkins-plugin-cli. The following command:
docker run -it --rm jenkins/jenkins:2.401.1-lts jenkins-plugin-cli --plugins "kubernetes:latest workflow-job:latest workflow-aggregator:latest git:latest job-dsl:latest configuration-as-code:latest kubernetes-credentials-provider:latest" --no-download --list --output yamlwill produce a fully pinned list of plugins with resolved dependencies on stdout. This list can be transformed to be compatible with the format expected in values.yaml. I got no exceptions upon startup.
I'm using the image referenced in the command above with operator image virtuslab/jenkins-operator:60b8ee5, and CRDs from 3fe842fff4bb910278a905af8f9171e77a9cd313
brokenpip3
commented
1 year ago hei @Harguer @ingineru @vlad-ivanov-name I think I fixed the always true latestPlugin bug, can you try this version? quay.io/jenkins-kubernetes-operator/operator:5ddcf10 or wait for the next release that will be released in the following days.
But also please be aware that using false will require a huge list of dependencies specified with a fixed version.
Also @vlad-ivanov-name please do not use that very old operator image, if you installed the operator with the kubectl apply method please take a look on this issue: #827
vlad-ivanov-name
commented
1 year ago But also please be aware that using false will require a huge list of dependencies specified with a fixed version.
This becomes less of an issue when using Terraform with Helm and Kustomization providers: the long list of plugins can be read from a file or some other data source.
very old operator image
Thanks! Would it be possible to link to quay.io from https://hub.docker.com/r/virtuslab/jenkins-operator or is it owned by a different person?
brokenpip3
commented
1 year ago Thanks! Would it be possible to link to quay.io from https://hub.docker.com/r/virtuslab/jenkins-operator or is it owned by a different person?
No, that dockerhub account is owned by the previous team
brokenpip3
commented
1 year ago fixed with this version: https://github.com/jenkinsci/kubernetes-operator/releases/tag/v0.8.0-beta2, let me know if it's still an issue
ingineru
commented
1 year ago hei @Harguer @ingineru @vlad-ivanov-name I think I fixed the always true latestPlugin bug, can you try this version?
quay.io/jenkins-kubernetes-operator/operator:5ddcf10or wait for the next release that will be released in the following days.But also please be aware that using
falsewill require a huge list of dependencies specified with a fixed version.Also @vlad-ivanov-name please do not use that very old operator image, if you installed the operator with the
kubectl applymethod please take a look on this issue: #827
I used the quay.io image and the issue seems to be fixed (still waiting for some plugin updates to see how it behaves then). Thank you for solving it! Also on that test instance, the operator does not seem to pick up the updates made to the Jenkins manifest, but it could be a problem with my testing environment.
brokenpip3
commented
1 year ago it depends of the types of changes that you are doing, not the 100% of the jenkins crd will trigger a restart, what you changed?
Harguer
commented
1 year ago Thank you @brokenpip3 , to me seems to be working fine. I tested on one of my test Jenkins. It was running for 40 days, then I pull all the plugins and manually pinned in the Jenkins CR, then I killed my Jenkins and it star crashing, then I updated the image of Jenkins operator with the latest one, also important to say that i had to update the Jenkins CRDs.
ingineru
commented
1 year ago it depends of the types of changes that you are doing, not the 100% of the jenkins crd will trigger a restart, what you changed?
It was a change in the list of pinned plugins that would have triggered the restart. The issue proved to be from my test environment In the end (the Jenkins LTS image was no longer whitelisted). Thanks again for your work!
pniederlag
commented
1 year ago changing/updating/incompatible plugins ist definitly a real problem with jenkins-operator or jenkins itself. Every time we have a (scheduled or not scheduled) restart for jenkins we ran into serious troubles as some plugins wont't install anymore.
brokenpip3
commented
1 year ago changing/updating/incompatible plugins ist definitly a real problem with jenkins-operator or jenkins itself. Every time we have a (scheduled or not scheduled) restart for jenkins we ran into serious troubles as some plugins wont't install anymore.
are you using the latestplugin equals to true or false?
Harguer
commented
1 year ago @pniederlag also make sure you are using the updated Jenkins operator image from quay.io and the latest CRDs.
pniederlag
commented
1 year ago First of all, big thx @Harguer @brokenpip3 for your feedback on this!
changing/updating/incompatible plugins ist definitly a real problem with jenkins-operator or jenkins itself. Every time we have a (scheduled or not scheduled) restart for jenkins we ran into serious troubles as some plugins wont't install anymore.
are you using the latestplugin equals to true or false?
I have toggled about hundred times already. currently using latestPlugins: false and (autogenerated/curated) list of all plugins in basePlugins:
/var/jenkins/scripts/init.sh contains --latest true (also spec.master.latestPlugins: false!)
As basePlugins are installed prior to plugins they might introduce some dependencies that later on have an effect of the list of plugins ?
using quay.io/jenkins-kubernetes-operator/operator:v0.8.0-beta2 (which hopefully carries the fix?)
The plugin you specified as code is incompatible with this jenkins version: plugin '{snakeyaml-api 1.33-95.va_b_a_e3e47b_fa_4 }' version, actual '2.2-111.vc6598e30cc65
I verified snakeyaml-api 1.33-95.va_b_a_e3e47b_fa_4 yesterday morning, and added it to plugins. Later on that evening the above message showed up, making me think something is broken. Actually snakeyaml-api 2.2-111.vc6598e30cc6 was just released a couple of hours ago. To my knowledge there is nothing that makes snakeyaml-api 1.33-95.va_b_a_e3e47b_fa_4 incompatible, it is just not the very most recent version as of now. ;)
Is there any deeper reasoning for having basePlugins and plugins other than trying to be clear on hard requirements(which are already part of the operator) and some plugins the user wants to use on top of that?
How about merging base-plugins and user-plugins and running one install on the resulting list?
Jenkins plugins handling in general is not quite clear as it seems hard to really nail a fixed set of working plugin-versions
WesselAtWork
commented
1 year ago snakeyaml-api got updated, and we ran into the same thing.
But for me the beta2 latestPlugins worked fine.
I set it both in basePlugins and plugins becuase I didn't know which one was correct.
Note: We were running an older version of Jenkins with many other pinned plugins. You should check your other plugins, maybe one of them requires the new version.
Harguer
commented
1 year ago Yeah I think you still need to pin all other plugins (dependencies) to make it work with basePlugins = false, otherwise those will break Jenkins initialization.
pniederlag
commented
1 year ago double checking once more... we actually really do have SnakeYAML2.2-111.vc6598e30cc65
personally I'd prefer if operator/jenkins would just bail and exit if configured plugins are not compatible with jenkins. I don't like to see other versions installed than the ones I do define, even if it is just to get jenkins up and running.
ingineru
commented
1 year ago I am confused by the latestPlugins parameter functionality. I pinned the base plugins and othe plugins and let the operator/Jenkins sort out their dependencies. After the deployment I expected all plugins are up-to-date, except maybe the ones I pinned. But there are several dependency plugins that can be updated. Is this expected?
pniederlag
commented
1 year ago finaly, with upgrade to v0.8.0 for chart and operator it seems stuff might be solved.... at least we don't see any warnings on incompatible versions
lastetPlugins: false does kick in our pinned list of (base)Plugins seems to effective
Great, thx!
Describe the bug I'm not sure if this is a bug or an improvement, but this is affecting a lot our prod jenkins, and also leadership is thinking to move from jenkins-operator to helm charts :(.
when my jenkins operator restarts, it start crashing because there are new plugins that are not compatible with my initial setup and/or with my current jenkins lts version pinned.
The problem is that in this section, jenkins is evaluating the plugins we pin in the jenkins CR (
basPluginsandplugins) against the latest available, this brings instability to production jenkins because jenkins won't start due plugins issues. https://github.com/jenkinsci/kubernetes-operator/blob/master/pkg/configuration/base/plugin.go#L15-L46I have tried to pin all the plugins in the jenkins CR and also I tried to pin all plugins in a Dockerfile, but it won't work because jenkins-operator.
I think if we put something ilike this
pinPlugins: <true/false>in the jenkins CR and code it, we could solve this issue.To Reproduce Install base plugins, some plugins and latest lts, let this running for a few days and kill the jenkins pod. the new jenkins pod will start crashing due plugin dependencies.
Additional information
Kubernetes version tested: 1.21, 1.22, 1.23
Jenkins Operator version: v0.7.1 and tag 60b8ee5
Add error logs about the problem here (operator logs and Kubernetes events).