Faild to upgrade 2.303.3 to 2.319.2

chengjun0124 commented 2 years ago

Hi team

I tried to upgrad jenkins from 2.303.3(registry.redhat.io/openshift4/ose-jenkins@sha256:788061714cd4d1fec426eee54d533018ebf37100ba7a1142e2a2833bd655c66e) to 2.319.2(registry.redhat.io/openshift4/ose-jenkins:v4.9.0-202202080146.p0.ga71a115.assembly.stream) with ENVs

OPENSHIFT_ENABLE_OAUTH='true'
OPENSHIFT_ENABLE_REDIRECT_PROMPT='false'
JENKINS_JAVA_OVERRIDES="-Dhudson.security.csrf.GlobalCrumbIssuerConfiguration.DISABLE_CSRF_PROTECTION=false -Dhudson.model.DirectoryBrowserSupport.CSP=\"default-src\\ 'self';\\ style-src\\ 'self'\\ 'unsafe-inline';\""
OVERRIDE_RELEASE_MIGRATION_OVERWRITE='true'

PVC is mounted to /var/lib/jenkins, and it's not empty. I need remain all my plugins. But it seems that jenkins cannot load my plugins correctly.

2022/02/16 05:44:43 [go-init] No pre-start command defined, skip 2022/02/16 05:44:43 [go-init] Main command launched : /usr/libexec/s2i/run The connection to the server 172.21.0.1:443 was refused - did you specify the right host or port? CONTAINER_MEMORY_IN_MB='8796093022207', using /usr/lib/jvm/java-11-openjdk-11.0.14.0.9-2.el8_4.x86_64/bin/java and /usr/lib/jvm/java-11-openjdk-11.0.14.0.9-2.el8_4.x86_64/bin/javac Unlinking plugin /var/lib/jenkins/plugins/bootstrap4-api.jpi since it has been removed Unlinking plugin /var/lib/jenkins/plugins/popper-api.jpi since it has been removed Linking /usr/lib/jenkins/jquery-detached.hpi RPM installed Jenkins plugins to /var/lib/jenkins ... Administrative monitors that contact the update center will remain active Picked up JAVA_TOOL_OPTIONS: -XX:+UnlockExperimentalVMOptions -Dsun.zip.disableMemoryMapping=true Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: Index 1 out of bounds for length 1 at com.redhat.openshift.PasswordChecker.main(PasswordChecker.java:14) Migrating slave image configuration to current version tag ... Using JENKINS_SERVICE_NAME=jenkins Generating jenkins.model.JenkinsLocationConfiguration.xml using (/var/lib/jenkins/jenkins.model.JenkinsLocationConfiguration.xml.tpl) ... Jenkins URL set to: http://null in file: /var/lib/jenkins/jenkins.model.JenkinsLocationConfiguration.xml

exec java -XX:+UseParallelGC -XX:MinHeapFreeRatio=5 -XX:MaxHeapFreeRatio=10 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -Dfile.encoding=UTF8 -Djavamelody.displayed-counters=log,error -Djava.util.logging.config.file=/var/lib/jenkins/logging.properties -Djdk.http.auth.tunneling.disabledSchemes= -Djdk.http.auth.proxying.disabledSchemes= -Duser.home=/var/lib/jenkins -Djavamelody.application-name=jenkins -Dhudson.security.csrf.GlobalCrumbIssuerConfiguration.DISABLE_CSRF_PROTECTION=true -Djenkins.install.runSetupWizard=false -Dhudson.security.csrf.GlobalCrumbIssuerConfiguration.DISABLE_CSRF_PROTECTION=false '-Dhudson.model.DirectoryBrowserSupport.CSP="default-src '\''self'\''; style-src '\''self'\'' '\''unsafe-inline'\'';"' -jar /usr/lib/jenkins/jenkins.war Picked up JAVA_TOOL_OPTIONS: -XX:+UnlockExperimentalVMOptions -Dsun.zip.disableMemoryMapping=true Running from: /usr/lib/jenkins/jenkins.war webroot: EnvVars.masterEnvVars.get("JENKINS_HOME") 2022-02-16 05:44:48 INFO org.eclipse.jetty.util.log.Log initialized Logging initialized @631ms to org.eclipse.jetty.util.log.JavaUtilLog 2022-02-16 05:44:48 INFO winstone.Logger logInternal Beginning extraction from war file 2022-02-16 05:44:49 WARNING org.eclipse.jetty.server.handler.ContextHandler setContextPath Empty contextPath 2022-02-16 05:44:49 INFO org.eclipse.jetty.server.Server doStart jetty-9.4.43.v20210629; built: 2021-06-30T11:07:22.254Z; git: 526006ecfa3af7f1a27ef3a288e2bef7ea9dd7e8; jvm 11.0.14+9-LTS 2022-02-16 05:44:50 INFO org.eclipse.jetty.webapp.StandardDescriptorProcessor visitServlet NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet 2022-02-16 05:44:50 INFO org.eclipse.jetty.server.session.DefaultSessionIdManager doStart DefaultSessionIdManager workerName=node0 2022-02-16 05:44:50 INFO org.eclipse.jetty.server.session.DefaultSessionIdManager doStart No SessionScavenger set, using defaults 2022-02-16 05:44:50 INFO org.eclipse.jetty.server.session.HouseKeeper startScavenging node0 Scavenging every 660000ms 2022-02-16 05:44:50 INFO hudson.WebAppMain contextInitialized Jenkins home directory: /var/lib/jenkins found at: EnvVars.masterEnvVars.get("JENKINS_HOME") 2022-02-16 05:44:51 INFO org.eclipse.jetty.server.handler.ContextHandler doStart Started w.@58a63629{Jenkins v2.319.2,/,file:///var/lib/jenkins/war/,AVAILABLE}{/var/lib/jenkins/war} 2022-02-16 05:44:51 INFO org.eclipse.jetty.server.AbstractConnector doStart Started ServerConnector@336f1079{HTTP/1.1, (http/1.1)}{0.0.0.0:8080} 2022-02-16 05:44:51 INFO org.eclipse.jetty.server.Server doStart Started @3621ms 2022-02-16 05:44:51 INFO winstone.Logger logInternal Winstone Servlet Engine running: controlPort=disabled 2022-02-16 05:44:51 INFO jenkins.InitReactorRunner$1 onAttained Started initialization 2022-02-16 05:44:51 INFO hudson.PluginManager loadDetachedPlugins Upgrading Jenkins. The last running version was 2.303.3. This Jenkins is version 2.319.2. 2022-02-16 05:44:51 INFO hudson.PluginManager loadDetachedPlugins Upgraded Jenkins from version 2.303.3 to version 2.319.2. Loaded detached plugins (and dependencies): [matrix-auth.hpi] 2022-02-16 05:44:53 WARNING hudson.ClassicPluginStrategy createClassJarFromWebInfClasses Created /var/lib/jenkins/plugins/groovy-label-assignment/WEB-INF/lib/classes.jar; update plugin to a version created with a newer harness 2022-02-16 05:44:53 WARNING hudson.ClassicPluginStrategy createClassJarFromWebInfClasses Created /var/lib/jenkins/plugins/oauth-credentials/WEB-INF/lib/classes.jar; update plugin to a version created with a newer harness 2022-02-16 05:44:56 WARNING hudson.ClassicPluginStrategy createClassJarFromWebInfClasses Created /var/lib/jenkins/plugins/job-dsl/WEB-INF/lib/classes.jar; update plugin to a version created with a newer harness 2022-02-16 05:45:01 WARNING hudson.ClassicPluginStrategy createClassJarFromWebInfClasses Created /var/lib/jenkins/plugins/ivy/WEB-INF/lib/classes.jar; update plugin to a version created with a newer harness 2022-02-16 05:45:02 INFO jenkins.InitReactorRunner$1 onAttained Listed all plugins 2022-02-16 05:45:02 SEVERE jenkins.InitReactorRunner$1 onTaskFailed Failed Loading plugin Configuration as Code Plugin v1.55.1 (configuration-as-code) java.io.IOException: Failed to load: Configuration as Code Plugin (1.55.1)

Update required: Snakeyaml API Plugin (1.27.0) to be updated to 1.29.1 or higher at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:1016) at hudson.PluginManager$2$1$1.run(PluginManager.java:535) at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296) at jenkins.model.Jenkins$5.runTask(Jenkins.java:1151) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829)

2022-02-16 05:45:02 SEVERE jenkins.InitReactorRunner$1 onTaskFailed Failed Loading plugin SSH Build Agents plugin v1.31.5 (ssh-slaves) java.io.IOException: Failed to load: SSH Build Agents plugin (1.31.5)

Update required: Trilead API Plugin (1.0.8) to be updated to 1.0.13 or higher at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:1016) at hudson.PluginManager$2$1$1.run(PluginManager.java:535) at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296) at jenkins.model.Jenkins$5.runTask(Jenkins.java:1151) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829)

2022-02-16 05:45:03 SEVERE jenkins.InitReactorRunner$1 onTaskFailed Failed Loading plugin Pipeline: Groovy v2.94 (workflow-cps) java.io.IOException: Failed to load: Pipeline: Groovy (2.94)

Update required: Pipeline: Supporting APIs (3.6) to be updated to 3.8 or higher
Update required: Script Security Plugin (1.75) to be updated to 1.77 or higher
Update required: Pipeline: SCM Step (2.11) to be updated to 2.13 or higher at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:1016) at hudson.PluginManager$2$1$1.run(PluginManager.java:535) at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296) at jenkins.model.Jenkins$5.runTask(Jenkins.java:1151) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829)

2022-02-16 05:45:03 SEVERE jenkins.InitReactorRunner$1 onTaskFailed Failed Loading plugin Jackson 2 API Plugin v2.13.1-246.va8a9f3eaf46a (jackson2-api) java.io.IOException: Failed to load: Jackson 2 API Plugin (2.13.1-246.va8a9f3eaf46a)

Update required: Snakeyaml API Plugin (1.27.0) to be updated to 1.29.1 or higher at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:1016) at hudson.PluginManager$2$1$1.run(PluginManager.java:535) at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296) at jenkins.model.Jenkins$5.runTask(Jenkins.java:1151) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829)

2022-02-16 05:45:03 SEVERE jenkins.InitReactorRunner$1 onTaskFailed Failed Loading plugin Common API for Blue Ocean v1.24.4 (blueocean-commons) java.io.IOException: Failed to load: Common API for Blue Ocean (1.24.4)

Failed to load: Jackson 2 API Plugin (2.13.1-246.va8a9f3eaf46a) at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:1016) at hudson.PluginManager$2$1$1.run(PluginManager.java:535) at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296) at jenkins.model.Jenkins$5.runTask(Jenkins.java:1151) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829)

2022-02-16 05:45:03 SEVERE jenkins.InitReactorRunner$1 onTaskFailed Failed Loading plugin REST API for Blue Ocean v1.24.4 (blueocean-rest) java.io.IOException: Failed to load: REST API for Blue Ocean (1.24.4)

Failed to load: Common API for Blue Ocean (1.24.4) at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:1016) at hudson.PluginManager$2$1$1.run(PluginManager.java:535) at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296) at jenkins.model.Jenkins$5.runTask(Jenkins.java:1151) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829)

2022-02-16 05:45:03 SEVERE jenkins.InitReactorRunner$1 onTaskFailed Failed Loading plugin Branch API Plugin v2.7.0 (branch-api) java.io.IOException: Failed to load: Branch API Plugin (2.7.0)

Update required: Folders Plugin (6.15) to be updated to 6.16 or higher
Update required: SCM API Plugin (2.6.4) to be updated to 2.6.5 or higher at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:1016) at hudson.PluginManager$2$1$1.run(PluginManager.java:535) at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296) at jenkins.model.Jenkins$5.runTask(Jenkins.java:1151) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829)

2022-02-16 05:45:03 SEVERE jenkins.InitReactorRunner$1 onTaskFailed Failed Loading plugin Pipeline: Multibranch v2.26 (workflow-multibranch) java.io.IOException: Failed to load: Pipeline: Multibranch (2.26)

Failed to load: Pipeline: Groovy (2.94) at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:1016) at hudson.PluginManager$2$1$1.run(PluginManager.java:535) at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296) at jenkins.model.Jenkins$5.runTask(Jenkins.java:1151) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829)

2022-02-16 05:45:03 SEVERE jenkins.InitReactorRunner$1 onTaskFailed Failed Loading plugin Pipeline SCM API for Blue Ocean v1.24.4 (blueocean-pipeline-scm-api) java.io.IOException: Failed to load: Pipeline SCM API for Blue Ocean (1.24.4)

Failed to load: REST API for Blue Ocean (1.24.4) at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:1016) at hudson.PluginManager$2$1$1.run(PluginManager.java:535) at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296) at jenkins.model.Jenkins$5.runTask(Jenkins.java:1151) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829)

2022-02-16 05:45:03 SEVERE jenkins.InitReactorRunner$1 onTaskFailed Failed Loading plugin Jenkins Mailer Plugin v408.vd726a_1130320 (mailer) java.io.IOException: Failed to load: Jenkins Mailer Plugin (408.vd726a_1130320)

Update required: Display URL API (2.3.4) to be updated to 2.3.5 or higher at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:1016) at hudson.PluginManager$2$1$1.run(PluginManager.java:535) at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296) at jenkins.model.Jenkins$5.runTask(Jenkins.java:1151) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829)

================================ many similar messages ================================ ... ... ... ================================ many similar messages ================================

2022-02-16 05:45:04 WARNING hudson.ExtensionFinder$Sezpoz scout Failed to scout jp.ikedam.jenkins.plugins.groovy_label_assignment.GroovyLabelAssignmentQueueDecisionHandler java.lang.NoClassDefFoundError: hudson/matrix/MatrixProject at java.base/java.lang.Class.forName0(Native Method) at java.base/java.lang.Class.forName(Class.java:398) at hudson.ExtensionFinder$Sezpoz.scout(ExtensionFinder.java:725) at hudson.ClassicPluginStrategy.findComponents(ClassicPluginStrategy.java:346) at hudson.ExtensionList.load(ExtensionList.java:382) at hudson.ExtensionList.ensureLoaded(ExtensionList.java:318) at hudson.ExtensionList.getComponents(ExtensionList.java:182) at jenkins.model.Jenkins$6.onInitMilestoneAttained(Jenkins.java:1181) at jenkins.InitReactorRunner$1.onAttained(InitReactorRunner.java:88) at org.jvnet.hudson.reactor.ReactorListener$Aggregator.lambda$onAttained$3(ReactorListener.java:102) at org.jvnet.hudson.reactor.ReactorListener$Aggregator.run(ReactorListener.java:109) at org.jvnet.hudson.reactor.ReactorListener$Aggregator.onAttained(ReactorListener.java:102) at org.jvnet.hudson.reactor.Reactor$1.run(Reactor.java:177) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: java.lang.ClassNotFoundException: hudson.matrix.MatrixProject at jenkins.util.AntClassLoader.findClassInComponents(AntClassLoader.java:1417) at jenkins.util.AntClassLoader.findClass(AntClassLoader.java:1372) at jenkins.util.AntClassLoader.loadClass(AntClassLoader.java:1127) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) ... 18 more

2022-02-16 05:45:05 WARNING hudson.ExtensionFinder$GuiceFinder$SezpozModule configure Failed to load jp.ikedam.jenkins.plugins.groovy_label_assignment.GroovyLabelAssignmentQueueDecisionHandler java.lang.NoClassDefFoundError: hudson/matrix/MatrixProject at java.base/java.lang.Class.getDeclaredConstructors0(Native Method) at java.base/java.lang.Class.privateGetDeclaredConstructors(Class.java:3137) at java.base/java.lang.Class.getDeclaredConstructors(Class.java:2357) at hudson.ExtensionFinder$GuiceFinder$SezpozModule.resolve(ExtensionFinder.java:496) at hudson.ExtensionFinder$GuiceFinder$SezpozModule.resolve(ExtensionFinder.java:483) at hudson.ExtensionFinder$GuiceFinder$SezpozModule.configure(ExtensionFinder.java:526) at com.google.inject.AbstractModule.configure(AbstractModule.java:62) at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340) at com.google.inject.spi.Elements.getElements(Elements.java:110) at com.google.inject.internal.InjectorShell$Builder.build(InjectorShell.java:138) at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:104) at com.google.inject.Guice.createInjector(Guice.java:96) at com.google.inject.Guice.createInjector(Guice.java:73) at hudson.ExtensionFinder$GuiceFinder.(ExtensionFinder.java:280) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) at java.base/java.lang.Class.newInstance(Class.java:584) at net.java.sezpoz.IndexItem.instance(IndexItem.java:181) at hudson.ExtensionFinder$Sezpoz._find(ExtensionFinder.java:701) at hudson.ExtensionFinder$Sezpoz.find(ExtensionFinder.java:687) at hudson.ClassicPluginStrategy.findComponents(ClassicPluginStrategy.java:352) at hudson.ExtensionList.load(ExtensionList.java:382) at hudson.ExtensionList.ensureLoaded(ExtensionList.java:318) at hudson.ExtensionList.getComponents(ExtensionList.java:182) at jenkins.model.Jenkins$6.onInitMilestoneAttained(Jenkins.java:1181) at jenkins.InitReactorRunner$1.onAttained(InitReactorRunner.java:88) at org.jvnet.hudson.reactor.ReactorListener$Aggregator.lambda$onAttained$3(ReactorListener.java:102) at org.jvnet.hudson.reactor.ReactorListener$Aggregator.run(ReactorListener.java:109) at org.jvnet.hudson.reactor.ReactorListener$Aggregator.onAttained(ReactorListener.java:102) at org.jvnet.hudson.reactor.Reactor$1.run(Reactor.java:177) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: java.lang.ClassNotFoundException: hudson.matrix.MatrixProject at jenkins.util.AntClassLoader.findClassInComponents(AntClassLoader.java:1417) at jenkins.util.AntClassLoader.findClass(AntClassLoader.java:1372) at jenkins.util.AntClassLoader.loadClass(AntClassLoader.java:1127) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) ... 37 more

WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by com.google.inject.internal.cglib.core.$ReflectUtils$2 (file:/var/lib/jenkins/war/WEB-INF/lib/guice-4.0.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain) WARNING: Please consider reporting this to the maintainers of com.google.inject.internal.cglib.core.$ReflectUtils$2 WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release 2022-02-16 05:45:07 INFO jenkins.InitReactorRunner$1 onAttained Prepared all plugins 2022-02-16 05:45:07 INFO jenkins.InitReactorRunner$1 onAttained Started all plugins 2022-02-16 05:45:07 INFO jenkins.InitReactorRunner$1 onAttained Augmented all extensions 2022-02-16 05:45:07 WARNING hudson.ExtensionFinder$GuiceFinder$FaultTolerantScope$1 error Failed to instantiate Key[type=org.jenkinsci.plugins.pipeline.utility.steps.csv.ReadCSVStep$DescriptorImpl, annotation=[none]]; skipping this component com.google.inject.ProvisionException: Unable to provision, see the following errors:

1) Error injecting constructor, java.lang.NoClassDefFoundError: org/jenkinsci/plugins/workflow/cps/GroovyShellDecorator at org.jenkinsci.plugins.pipeline.utility.steps.csv.ReadCSVStep$DescriptorImpl.(ReadCSVStep.java:76)

1 error at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:52) at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:145) at hudson.ExtensionFinder$GuiceFinder$FaultTolerantScope$1.get(ExtensionFinder.java:441) at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1016) at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1012) at hudson.ExtensionFinder$GuiceFinder._find(ExtensionFinder.java:401) at hudson.ExtensionFinder$GuiceFinder.find(ExtensionFinder.java:392) at hudson.ClassicPluginStrategy.findComponents(ClassicPluginStrategy.java:352) at hudson.ExtensionList.load(ExtensionList.java:382) at hudson.ExtensionList.ensureLoaded(ExtensionList.java:318) at hudson.ExtensionList.iterator(ExtensionList.java:170) at hudson.diagnosis.NullIdDescriptorMonitor.verify(NullIdDescriptorMonitor.java:71) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:104) at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:180) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296) at jenkins.model.Jenkins$5.runTask(Jenkins.java:1151) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: java.lang.NoClassDefFoundError: org/jenkinsci/plugins/workflow/cps/GroovyShellDecorator at java.base/java.lang.ClassLoader.defineClass1(Native Method) at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1017) at jenkins.util.AntClassLoader.defineClassFromData(AntClassLoader.java:1187) at jenkins.util.AntClassLoader.getClassFromStream(AntClassLoader.java:1355) at jenkins.util.AntClassLoader.findClassInComponents(AntClassLoader.java:1407) at jenkins.util.AntClassLoader.findClass(AntClassLoader.java:1372) at jenkins.util.AntClassLoader.loadClass(AntClassLoader.java:1127) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) at java.base/java.lang.Class.getDeclaringClass0(Native Method) at java.base/java.lang.Class.getEnclosingClass(Class.java:1517) at hudson.model.Descriptor.(Descriptor.java:286) at org.jenkinsci.plugins.workflow.steps.StepDescriptor.(StepDescriptor.java:62) at org.jenkinsci.plugins.pipeline.utility.steps.AbstractFileOrTextStepDescriptorImpl.(AbstractFileOrTextStepDescriptorImpl.java:15) at org.jenkinsci.plugins.pipeline.utility.steps.csv.ReadCSVStep$DescriptorImpl.(ReadCSVStep.java:76) at org.jenkinsci.plugins.pipeline.utility.steps.csv.ReadCSVStep$DescriptorImpl$$FastClassByGuice$$45bc348a.newInstance() at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40) at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:61) at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:105) at com.google.inject.internal.ConstructorInjector.access$000(ConstructorInjector.java:32) at com.google.inject.internal.ConstructorInjector$1.call(ConstructorInjector.java:89) at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:115) at hudson.ExtensionFinder$GuiceFinder$SezpozModule.onProvision(ExtensionFinder.java:563) at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:126) at com.google.inject.internal.ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:68) at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:87) at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:267) at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1103) at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) ... 27 more Caused by: java.lang.ClassNotFoundException: org.jenkinsci.plugins.workflow.cps.GroovyShellDecorator at jenkins.util.AntClassLoader.findClassInComponents(AntClassLoader.java:1417) at jenkins.util.AntClassLoader.findClass(AntClassLoader.java:1372) at jenkins.util.AntClassLoader.loadClass(AntClassLoader.java:1127) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) ... 56 more

2022-02-16 05:45:07 WARNING hudson.ExtensionFinder$GuiceFinder$FaultTolerantScope$1 error Failed to instantiate Key[type=org.jenkinsci.plugins.pipeline.utility.steps.maven.ReadMavenPomStep$DescriptorImpl, annotation=[none]]; skipping this component com.google.inject.ProvisionException: Unable to provision, see the following errors:

1) Error injecting constructor, java.lang.NoClassDefFoundError: org/jenkinsci/plugins/workflow/cps/GroovyShellDecorator at org.jenkinsci.plugins.pipeline.utility.steps.maven.ReadMavenPomStep$DescriptorImpl.(ReadMavenPomStep.java:103)

1 error at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:52) at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:145) at hudson.ExtensionFinder$GuiceFinder$FaultTolerantScope$1.get(ExtensionFinder.java:441) at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1016) at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1012) at hudson.ExtensionFinder$GuiceFinder._find(ExtensionFinder.java:401) at hudson.ExtensionFinder$GuiceFinder.find(ExtensionFinder.java:392) at hudson.ClassicPluginStrategy.findComponents(ClassicPluginStrategy.java:352) at hudson.ExtensionList.load(ExtensionList.java:382) at hudson.ExtensionList.ensureLoaded(ExtensionList.java:318) at hudson.ExtensionList.iterator(ExtensionList.java:170) at hudson.diagnosis.NullIdDescriptorMonitor.verify(NullIdDescriptorMonitor.java:71) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:104) at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:180) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296) at jenkins.model.Jenkins$5.runTask(Jenkins.java:1151) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: java.lang.NoClassDefFoundError: org/jenkinsci/plugins/workflow/cps/GroovyShellDecorator at java.base/java.lang.ClassLoader.defineClass1(Native Method) at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1017) at jenkins.util.AntClassLoader.defineClassFromData(AntClassLoader.java:1187) at jenkins.util.AntClassLoader.getClassFromStream(AntClassLoader.java:1355) at jenkins.util.AntClassLoader.findClassInComponents(AntClassLoader.java:1407) at jenkins.util.AntClassLoader.findClass(AntClassLoader.java:1372) at jenkins.util.AntClassLoader.loadClass(AntClassLoader.java:1127) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) at java.base/java.lang.Class.getDeclaringClass0(Native Method) at java.base/java.lang.Class.getEnclosingClass(Class.java:1517) at hudson.model.Descriptor.(Descriptor.java:286) at org.jenkinsci.plugins.workflow.steps.StepDescriptor.(StepDescriptor.java:62) at org.jenkinsci.plugins.pipeline.utility.steps.maven.ReadMavenPomStep$DescriptorImpl.(ReadMavenPomStep.java:103) at org.jenkinsci.plugins.pipeline.utility.steps.maven.ReadMavenPomStep$DescriptorImpl$$FastClassByGuice$$f3871400.newInstance() at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40) at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:61) at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:105) at com.google.inject.internal.ConstructorInjector.access$000(ConstructorInjector.java:32) at com.google.inject.internal.ConstructorInjector$1.call(ConstructorInjector.java:89) at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:115) at hudson.ExtensionFinder$GuiceFinder$SezpozModule.onProvision(ExtensionFinder.java:563) at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:126) at com.google.inject.internal.ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:68) at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:87) at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:267) at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1103) at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) ... 27 more Caused by: java.lang.ClassNotFoundException: org.jenkinsci.plugins.workflow.cps.GroovyShellDecorator at jenkins.util.AntClassLoader.findClassInComponents(AntClassLoader.java:1417) at jenkins.util.AntClassLoader.findClass(AntClassLoader.java:1372) at jenkins.util.AntClassLoader.loadClass(AntClassLoader.java:1127) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) ... 55 more

2022-02-16 05:45:09 WARNING org.jenkinsci.plugins.matrixauth.AbstractAuthorizationContainerConverter unmarshalContainer Skipping a non-existent permission java.lang.IllegalArgumentException: Failed to parse 'hudson.model.Run.Replay:IAM#email address-edit-view' --- no such permission at org.jenkinsci.plugins.matrixauth.AuthorizationContainer.add(AuthorizationContainer.java:83) at org.jenkinsci.plugins.matrixauth.AbstractAuthorizationContainerConverter.unmarshalContainer(AbstractAuthorizationContainerConverter.java:82) at org.jenkinsci.plugins.matrixauth.AbstractAuthorizationContainerConverter.unmarshal(AbstractAuthorizationContainerConverter.java:94) at hudson.util.XStream2$AssociatedConverterImpl.unmarshal(XStream2.java:479) at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72) at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:72) at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66) at hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:443) at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:345) at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:284) at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72) at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:72) at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66) at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50) at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:134) at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32) at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1391) at hudson.util.XStream2.unmarshal(XStream2.java:174) at hudson.util.XStream2.unmarshal(XStream2.java:145) at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1376) at hudson.XmlFile.unmarshal(XmlFile.java:194) at hudson.XmlFile.unmarshal(XmlFile.java:177) at jenkins.model.Jenkins.loadConfig(Jenkins.java:3272) at jenkins.model.Jenkins.access$1200(Jenkins.java:336) at jenkins.model.Jenkins$12.run(Jenkins.java:3374) at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296) at jenkins.model.Jenkins$5.runTask(Jenkins.java:1151) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829)

2022-02-16 05:45:09 SEVERE jenkins.InitReactorRunner$1 onTaskFailed Failed Loading global config java.io.IOException: Unable to read /var/lib/jenkins/config.xml at hudson.XmlFile.unmarshal(XmlFile.java:197) at hudson.XmlFile.unmarshal(XmlFile.java:177) at jenkins.model.Jenkins.loadConfig(Jenkins.java:3272) at jenkins.model.Jenkins.access$1200(Jenkins.java:336) at jenkins.model.Jenkins$12.run(Jenkins.java:3374) at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296) at jenkins.model.Jenkins$5.runTask(Jenkins.java:1151) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: jenkins.util.xstream.CriticalXStreamException: ---- Debugging information ---- cause-exception : com.thoughtworks.xstream.mapper.CannotResolveClassException cause-message : org.openshift.jenkins.plugins.openshiftlogin.OpenShiftOAuth2SecurityRealm class : hudson.model.Hudson required-type : hudson.model.Hudson converter-type : hudson.util.RobustReflectionConverter path : /hudson/securityRealm line number : 771 version : not available

at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:370)
at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:284)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:72)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:134)
at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)
at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1391)
at hudson.util.XStream2.unmarshal(XStream2.java:174)
at hudson.util.XStream2.unmarshal(XStream2.java:145)
at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1376)
at hudson.XmlFile.unmarshal(XmlFile.java:194)
... 13 more

Caused by: com.thoughtworks.xstream.mapper.CannotResolveClassException: org.openshift.jenkins.plugins.openshiftlogin.OpenShiftOAuth2SecurityRealm at com.thoughtworks.xstream.mapper.DefaultMapper.realClass(DefaultMapper.java:81) at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:125) at com.thoughtworks.xstream.mapper.DynamicProxyMapper.realClass(DynamicProxyMapper.java:55) at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:125) at com.thoughtworks.xstream.mapper.PackageAliasingMapper.realClass(PackageAliasingMapper.java:88) at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:125) at com.thoughtworks.xstream.mapper.ClassAliasingMapper.realClass(ClassAliasingMapper.java:79) at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:125) at com.thoughtworks.xstream.mapper.ArrayMapper.realClass(ArrayMapper.java:74) at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:125) at com.thoughtworks.xstream.mapper.SecurityMapper.realClass(SecurityMapper.java:71) at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:125) at hudson.util.XStream2$CompatibilityMapper.realClass(XStream2.java:391) at hudson.util.xstream.MapperDelegate.realClass(MapperDelegate.java:45) at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:125) at com.thoughtworks.xstream.mapper.CachingMapper.realClass(CachingMapper.java:47) at hudson.util.RobustReflectionConverter.determineType(RobustReflectionConverter.java:510) at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:341) ... 25 more

2022-02-16 05:45:09 SEVERE hudson.util.BootFailure publish Failed to initialize Jenkins hudson.util.HudsonFailedToLoad: org.jvnet.hudson.reactor.ReactorException: java.io.IOException: Unable to read /var/lib/jenkins/config.xml at hudson.WebAppMain$3.run(WebAppMain.java:315) Caused by: org.jvnet.hudson.reactor.ReactorException: java.io.IOException: Unable to read /var/lib/jenkins/config.xml at org.jvnet.hudson.reactor.Reactor.execute(Reactor.java:282) at jenkins.InitReactorRunner.run(InitReactorRunner.java:49) at jenkins.model.Jenkins.executeReactor(Jenkins.java:1184) at jenkins.model.Jenkins.(Jenkins.java:976) at hudson.model.Hudson.(Hudson.java:85) at hudson.model.Hudson.(Hudson.java:81) at hudson.WebAppMain$3.run(WebAppMain.java:298) Caused by: java.io.IOException: Unable to read /var/lib/jenkins/config.xml at hudson.XmlFile.unmarshal(XmlFile.java:197) at hudson.XmlFile.unmarshal(XmlFile.java:177) at jenkins.model.Jenkins.loadConfig(Jenkins.java:3272) at jenkins.model.Jenkins.access$1200(Jenkins.java:336) at jenkins.model.Jenkins$12.run(Jenkins.java:3374) at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296) at jenkins.model.Jenkins$5.runTask(Jenkins.java:1151) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: jenkins.util.xstream.CriticalXStreamException: ---- Debugging information ---- cause-exception : com.thoughtworks.xstream.mapper.CannotResolveClassException cause-message : org.openshift.jenkins.plugins.openshiftlogin.OpenShiftOAuth2SecurityRealm class : hudson.model.Hudson required-type : hudson.model.Hudson converter-type : hudson.util.RobustReflectionConverter path : /hudson/securityRealm line number : 771 version : not available

at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:370)
at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:284)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:72)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:134)
at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)
at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1391)
at hudson.util.XStream2.unmarshal(XStream2.java:174)
at hudson.util.XStream2.unmarshal(XStream2.java:145)
at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1376)
at hudson.XmlFile.unmarshal(XmlFile.java:194)
... 13 more

Caused by: com.thoughtworks.xstream.mapper.CannotResolveClassException: org.openshift.jenkins.plugins.openshiftlogin.OpenShiftOAuth2SecurityRealm at com.thoughtworks.xstream.mapper.DefaultMapper.realClass(DefaultMapper.java:81) at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:125) at com.thoughtworks.xstream.mapper.DynamicProxyMapper.realClass(DynamicProxyMapper.java:55) at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:125) at com.thoughtworks.xstream.mapper.PackageAliasingMapper.realClass(PackageAliasingMapper.java:88) at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:125) at com.thoughtworks.xstream.mapper.ClassAliasingMapper.realClass(ClassAliasingMapper.java:79) at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:125) at com.thoughtworks.xstream.mapper.ArrayMapper.realClass(ArrayMapper.java:74) at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:125) at com.thoughtworks.xstream.mapper.SecurityMapper.realClass(SecurityMapper.java:71) at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:125) at hudson.util.XStream2$CompatibilityMapper.realClass(XStream2.java:391) at hudson.util.xstream.MapperDelegate.realClass(MapperDelegate.java:45) at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:125) at com.thoughtworks.xstream.mapper.CachingMapper.realClass(CachingMapper.java:47) at hudson.util.RobustReflectionConverter.determineType(RobustReflectionConverter.java:510) at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:341) ... 25 more

2022-02-16 05:45:09 INFO jenkins.model.Jenkins cleanUp Stopping Jenkins 2022-02-16 05:45:09 INFO jenkins.model.Jenkins$16 onAttained Started termination 2022-02-16 05:45:09 INFO jenkins.model.Jenkins$16 onAttained Completed termination 2022-02-16 05:45:09 INFO jenkins.model.Jenkins _cleanUpDisconnectComputers Starting node disconnection 2022-02-16 05:45:09 INFO jenkins.model.Jenkins _cleanUpShutdownPluginManager Stopping plugin manager 2022-02-16 05:45:09 INFO jenkins.model.Jenkins _cleanUpPersistQueue Persisting build queue 2022-02-16 05:45:09 INFO jenkins.model.Jenkins cleanUp Jenkins stopped

akram commented 2 years ago

hi @chengjun0124 , in some cases, you will not be able to update plugins as-is because they may be incompatible with the new version of Jenkins. it seems that this is the case for 4 plugins mentioned in this error message.

version 2.319.2. Loaded detached plugins (and dependencies): [matrix-auth.hpi]
2022-02-16 05:44:53 WARNING hudson.ClassicPluginStrategy createClassJarFromWebInfClasses Created /var/lib/jenkins/plugins/**groovy-label-assignment**/WEB-INF/lib/classes.jar; update plugin to a version created with a newer harness
2022-02-16 05:44:53 WARNING hudson.ClassicPluginStrategy createClassJarFromWebInfClasses Created /var/lib/jenkins/plugins/**oauth-credentials**/WEB-INF/lib/classes.jar; update plugin to a version created with a newer harness
2022-02-16 05:44:56 WARNING hudson.ClassicPluginStrategy createClassJarFromWebInfClasses Created /var/lib/jenkins/plugins/**job-dsl**/WEB-INF/lib/classes.jar; update plugin to a version created with a newer harness
2022-02-16 05:45:01 WARNING hudson.ClassicPluginStrategy createClassJarFromWebInfClasses Created /var/lib/jenkins/plugins/**ivy**/WEB-INF/lib/classes.jar; update plugin to a version created with a newer harness

Make a backup of your PVC, and I would suggest you try to remove these plugins to see if the upgrade can happens.

We were having a discussion about this scenario yesterday with @gabemontero

ghost commented 2 years ago

Hi @akram and @chengjun0124,

I have been facing the same issue. Manually deleting the plugins did not help as far as I have understood it.

I believe that s2i run script is creating symbol links to the plugins before jenkins startup. (https://github.com/openshift/jenkins/blob/master/2/contrib/s2i/run#L432). The links point to the plugin versions that are part of the image.

Is it possible that ose-jenkins image is holding plugin versions that are not compatible with the latest jenkins-core?

Updating the mentioned plugins before updating the jenkins-core could help (due to the logic of the run script).

I worked around the issue by manually downloading and patching the latest plugin versions into the plugin directory.

akram commented 2 years ago

@jitendar-singh can you PTAL ?

chengjun0124 commented 2 years ago

@ma-fall I used the same way as yours, and upgraded jenkins successfully. Thank you @ma-fall @akram .

chengjun0124 commented 2 years ago

It takes more time to download and patching the specific version of plugin, it is not convenience if we need do these steps every time we upgrade ose-jenkins.

And another problem is that although I upgraded jenkins successfully via downloading and patching some necessary plugins which are listed in the jenkins log with error message, many other plugins are covered by ose-jenkins. That means after I upgraded jenkins, many versions of plugins are different from before, however I want to keep all plugins same after upgrading.

gabemontero commented 2 years ago

Hi @akram and @chengjun0124,

I have been facing the same issue. Manually deleting the plugins did not help as far as I have understood it.

I believe that s2i run script is creating symbol links to the plugins before jenkins startup. (https://github.com/openshift/jenkins/blob/master/2/contrib/s2i/run#L432). The links point to the plugin versions that are part of the image.

Is it possible that ose-jenkins image is holding plugin versions that are not compatible with the latest jenkins-core?

Updating the mentioned plugins before updating the jenkins-core could help (due to the logic of the run script).

I worked around the issue by manually downloading and patching the latest plugin versions into the plugin directory.

The symbolic linking you found @ma-fall is part of the OVERRIDE_PV_PLUGINS_WITH_IMAGE_PLUGINS configuration parameter discussed in https://docs.openshift.com/container-platform/4.9/openshift_images/using_images/images-other-jenkins.html#images-other-jenkins-env-var_images-other-jenkins

The idea that we have carried since the 3.x days is that if you manually upgrade a plugin we include with the openshift jenkins image, you were assuming responsibility to upgrade that plugin moving forward. In other words, you were interested in that plugin regardless of the fact we have included it in the image, and desired to control its version for some reason. But once you have made that choice, we no longer manage that plugin for you during OCP upgrade, unless you "reset" the system by setting OVERRIDE_PV_PLUGINS_WITH_IMAGE_PLUGINS to true, in which case our startup script forces all the plugins we ship with the image to be at the versions we include in the image.

That is why in with the symbollic linking you found, you see ln -s and not ln -sf, where ln -sf would force the link and overwrite the version you previously went to.

Next, the removing of the plugins that @akram suggested was an attempt to mimic setting OVERRIDE_PV_PLUGINS_WITH_IMAGE_PLUGINS to true, though the official, supported way of doing this is in fact to set that env var on the jenkins DeploymentConfig. I have confirmed the behavior of setting that env var to true with an OCP level as recent as 4.9.15. In particular

provisioned jenkins using the jenkins-persistent template
upgraded one of the plugins we include in the payload
added the OVERRIDE_PV_PLUGINS_WITH_IMAGE_PLUGINS env var with a setting of 'true' to the jenkins DeploymentConfig created by the template
with the ConfigChange trigger in that DeploymentConfig, a rollout occurred, and new Pod was started with that env var set
the plugin was reverted back to the version that came with the jenkins image that ships with OCP 4.9.15

Side bar: given recent user interactions, and the overall state of upstream jenkins, we are thinking changing the image so that the behavior here defaults to OVERRIDE_PV_PLUGINS_WITH_IMAGE_PLUGINS being true instead of false, which have been the traditional setting. Most likely that is at the crux of what you all are running into. These days, it seems like users are upgrading these plugins for the same reason we are: Jenkins Security Advisories. So let's just force the plugins to be the list we have tested.

Now, I'd like to clarify on what we "officially support" and where we will "help where we can" @ma-fall and @chengjun0124 wrt to the openshift jenkins image, the core jenkins version, and the plugin versions 1) we support upgrade of jenkins core via upgrades of OCP (since the openshift jenkins image is part of the OCP payload)

that means that if you try to upgrade the Jenkins version from the Jenkins console, we'll help where we can, but you will most likely have to make manual adjustments on your own to the plugins.

But if you upgrade jenkins core via an OCP upgrade, say for example 4.8.x to 4.8.x+3, we want to do whatever we can to make this a automatic experience.

2) similar, with plugins, if you upgrade plugins we include in the openshift jenkins image manually from the console, unless you have set OVERRIDE_PV_PLUGINS_WITH_IMAGE_PLUGINS to true you will have to manage that plugin's version moving forward. If you upgrade plugins via say a 4.8.x to 4.8.x+3 OCP upgrade, things should be OK.

3) If you add plugins that are not in our image, and if those plugins at the versions you added them interfere with the plugins or jenkins core we do include in our openshift jenkins image, again, you will have to manually managed those plugins.

So, @chengjun0124 @ma-fall - can you clarify exactly how you initiated the jenkins core upgrade? From the jenkins console? Or if via an OCP upgrade, can you tell us the OCP version you were at before, and the one you upgraded to?

If via OCP upgrade, we'll minimally have @jitendar-singh peform the same upgrade witihout manually changing any plugins, and go from there. If that upgrade fails, then most likely upstream jenkins has broken us. There has been occasions before where they change a plugin's dependencies WITHOUT changing the version of the plulgin. And we'll need to update or OCP jenkins image to adjust.

If the upgrade works for @jitendar-singh then we are in the "need to use OVERRIDE_PV_PLUGINS_WITH_IMAGE_PLUGINS" path. I believe @akram has gotten to the root of @chengjun0124 's issue. matrix-auth is one of the core jenkins plugins we currently specify because of Jenkins securitiy advisories. But if there were other plugins you had to manually upgrade @chengjun0124 please list them specifically, and we can cross reference them with the plugins included in the openshift jenkins image.

@ma-fall - aside from letting is know if you upgraded to jenkins 2.319.2 via an OCP upgrade, can you clarify which plugins you had to manually update?

As an FYI, we we updated our upcoming OCP release (4.10) to 2.319.2, the plugin list used for the image was listed at https://github.com/openshift/jenkins/pull/1368#issuecomment-1022499744

ghost commented 2 years ago

Hi @gabemontero

thanks for all the information and insights.

What I have done: I have updated all Jenkins plugins to the latest available version. Then have performed the Jenkins update by changing the used image

from: Jenkins 2.303.3 (4.9 Image)
to: Jenkins 2.319.2 (4.9 Image)

I hope this is what you assume as OCP upgrade. Afterwards I have been facing the described issue. The Jenkins log was telling me to update the following plugins:

Update required: Display URL API (2.3.4) to be updated to 2.3.5 or higher
Update required: Snakeyaml API Plugin (1.27.0) to be updated to 1.29.1 or higher
Update required: Pipeline: SCM Step (2.11) to be updated to 2.13 or higher
Update required: SSH Credentials Plugin (1.18.1) to be updated to 1.19 or higher
Update required: Trilead API Plugin (1.0.8) to be updated to 1.0.13 or higher
Update required: Jenkins Apache HttpComponents Client 4.x API Plugin (4.5.10-2.0) to be updated to 4.5.13-1.0 or higher

I am not sure if this is those are all plugins since I have tried out many things.

As a second test I have restored the pre backup state of my PV and erased the mentioned plugins from the PV. During the startup the symbolic links to the plugins in the image were created again and I got the same error.

Regarding OVERRIDE_PV_PLUGINS_WITH_IMAGE_PLUGINS :

I try to keep the Jenkins plugins up to date (due to security advisories) and I have the habit of always updating them before a core update. So having OVERRIDE_PV_PLUGINS_WITH_IMAGE_PLUGINS set to false as default works out for me.

Regarding OVERRIDE_RELEASE_MIGRATION_OVERWRITE: I am a little bit confused. But I am also quite new to the whole topic so this could be a misunderstanding on my side. If I check the list of the environment variables https://github.com/openshift/jenkins/blob/master/README.md#environment-variables in Github then I find the env var "OVERRIDE_RELEASE_MIGRATION_OVERWRITE". I also found that feature in the 4.9 version of the run script: https://github.com/openshift/jenkins/blob/master/2/contrib/s2i/run#L390

But this env var is not mentioned in the RedHat docu that you are referencing.

For me this looks like as if the plugins from the image are always added to the PV during a version upgrade. If this is true then even updating all plugins before a version / core upgrade will not resolve the issue.

I did not further debug this though since I just manually updated the plugins after the upgrade. So I might be missing something here.

HTH!

gabemontero commented 2 years ago

Hi @gabemontero

thanks for all the information and insights.

What I have done: I have updated all Jenkins plugins to the latest available version. Then have performed the Jenkins update by changing the used image
* from: Jenkins [2.303.3 (4.9 Image) ](https://catalog.redhat.com/software/containers/openshift4/ose-jenkins/5cdd918ad70cc57c44b2d279?tag=v4.9.0-202111221622.p0.g20ddf94.assembly.stream&push_date=1638183190000)

* to: [Jenkins 2.319.2 (4.9 Image) ](https://catalog.redhat.com/software/containers/openshift4/ose-jenkins/5cdd918ad70cc57c44b2d279?tag=v4.9.0-202202080146.p0.ga71a115.assembly.stream&push_date=1644839983000)

OK thanks for the confirmation there. That at least helps.

I hope this is what you assume as OCP upgrade. Afterwards I have been facing the described issue. The Jenkins log was telling me to update the following plugins:
* Update required: Display URL API (2.3.4) to be updated to 2.3.5 or higher

* Update required: Snakeyaml API Plugin (1.27.0) to be updated to 1.29.1 or higher

* Update required: Pipeline: SCM Step (2.11) to be updated to 2.13 or higher

* Update required: SSH Credentials Plugin (1.18.1) to be updated to 1.19 or higher

* Update required: Trilead API Plugin (1.0.8) to be updated to 1.0.13 or higher

* Update required: Jenkins Apache HttpComponents Client 4.x API Plugin (4.5.10-2.0) to be updated to 4.5.13-1.0 or higher
I am not sure if this is those are all plugins since I have tried out many things.

As a second test I have restored the pre backup state of my PV and erased the mentioned plugins from the PV. During the startup the symbolic links to the plugins in the image were created again and I got the same error.

Regarding OVERRIDE_PV_PLUGINS_WITH_IMAGE_PLUGINS :

I try to keep the Jenkins plugins up to date (due to security advisories) and I have the habit of always updating them before a core update. So having OVERRIDE_PV_PLUGINS_WITH_IMAGE_PLUGINS set to false as default works out for me.

OK thanks for the confirmation here as well. Yes, I think yours is the typical pattern historically. But with the myriad of plugins it simply gets onerous for users like you, to the point that you have to do a bunch of manual upgrades to get the various plugins to a happy state.

We will have to see moving forward if what happened with this move to 2.319.2 if that experience becomes the exception or the norm, and if changing the default behavior here makes things "less problematic".

Regarding OVERRIDE_RELEASE_MIGRATION_OVERWRITE: I am a little bit confused. But I am also quite new to the whole topic so this could be a misunderstanding on my side. If I check the list of the environment variables https://github.com/openshift/jenkins/blob/master/README.md#environment-variables in Github then I find the env var "OVERRIDE_RELEASE_MIGRATION_OVERWRITE". I also found that feature in the 4.9 version of the run script: https://github.com/openshift/jenkins/blob/master/2/contrib/s2i/run#L390

But this env var is not mentioned in the RedHat docu that you are referencing.

Sorry, you are incorrect here @ma-fall, it is.

I'll copy and paste from that page ... but in case it was not clear, "parameters" and "env vars" are the same thing in the end. The templates for jenkins map parameters to env vars.


OVERRIDE_PV_PLUGINS_WITH_IMAGE_PLUGINS 

When  running this image with an OpenShift Container Platform PV for the  Jenkins configuration directory, the transfer of plug-ins from the image  to the PV is performed only the first time the image starts because the  PV is assigned when the PVC is created. If you create a custom image  that extends this image and updates plug-ins in the custom image after  the initial startup, the plug-ins are not copied over unless you set  this environment variable to true. 

Default: false

For me this looks like as if the plugins from the image are always added to the PV during a version upgrade. If this is true then even updating all plugins before a version / core upgrade will not resolve the issue.

I did not further debug this though since I just manually updated the plugins after the upgrade. So I might be missing something here.

HTH!

ghost commented 2 years ago

Sorry, you are incorrect here @ma-fall, it is.

I'll copy and paste from that page ... but in case it was not clear, "parameters" and "env vars" are the same thing in the end. The templates for jenkins map parameters to env vars.
OVERRIDE_PV_PLUGINS_WITH_IMAGE_PLUGINS 

When  running this image with an OpenShift Container Platform PV for the  Jenkins configuration directory, the transfer of plug-ins from the image  to the PV is performed only the first time the image starts because the  PV is assigned when the PVC is created. If you create a custom image  that extends this image and updates plug-ins in the custom image after  the initial startup, the plug-ins are not copied over unless you set  this environment variable to true. 
Default: false

@gabemontero thanks for the clarification but unfortunately I don't fully get it.

I wanted to highlight that I found the env var OVERRIDE_RELEASE_MIGRATION_OVERWRITE which is also having impact on the plugin installation during startup. It was not mentioned in this discussion before and I can not find it in the RedHat documentation. In your reply you are mentioning the env var OVERRIDE_PV_PLUGINS_WITH_IMAGE_PLUGINS which seems to be a different variable.

I am sorry if I miss anything obvious here.

gabemontero commented 2 years ago

OVERRIDE_RELEASE_MIGRATION_OVERWRITE

Yeah OVERRIDE_RELEASE_MIGRATION_OVERWRITE is only applicable for older 3.x versions. That is not obvious though.

For 4.x only OVERRIDE_PV_PLUGINS_WITH_IMAGE_PLUGINS is applicable

chengjun0124 commented 2 years ago

Hi @gabemontero

The upgrade method I used was just change my deploymentConfig yaml. old yaml: Image: "registry.redhat.io/openshift4/ose-jenkins:v4.9.0-202111221622.p0.g20ddf94.assembly.stream" new yaml: image: "registry.redhat.io/openshift4/ose-jenkins:v4.9.0-202202080146.p0.ga71a115.assembly.stream"

I didn't set OVERRIDE_PV_PLUGINS_WITH_IMAGE_PLUGINS, I only set OVERRIDE_RELEASE_MIGRATION_OVERWRITE=true. And my openshift cluster version is 4.7, I am not sure whether I can use the image with tag 4.9.0-x

Then I got almost same logs with @ma-fall

Update required: Display URL API (2.3.4) to be updated to 2.3.5 or higher Update required: Snakeyaml API Plugin (1.27.0) to be updated to 1.29.1 or higher Update required: Pipeline: SCM Step (2.11) to be updated to 2.13 or higher Update required: SSH Credentials Plugin (1.18.1) to be updated to 1.19 or higher Update required: Trilead API Plugin (1.0.8) to be updated to 1.0.13 or higher Update required: Jenkins Apache HttpComponents Client 4.x API Plugin (4.5.10-2.0) to be updated to 4.5.13-1.0 or higher

gabemontero commented 2 years ago

Hi @gabemontero

The upgrade method I used was just change my deploymentConfig yaml. old yaml: Image: "registry.redhat.io/openshift4/ose-jenkins:v4.9.0-202111221622.p0.g20ddf94.assembly.stream" new yaml: image: "registry.redhat.io/openshift4/ose-jenkins:v4.9.0-202202080146.p0.ga71a115.assembly.stream"

I didn't set OVERRIDE_PV_PLUGINS_WITH_IMAGE_PLUGINS, I only set OVERRIDE_RELEASE_MIGRATION_OVERWRITE=true. And my openshift cluster version is 4.7, I am not sure whether I can use the image with tag 4.9.0-x

Then I got almost same logs with @ma-fall

Update required: Display URL API (2.3.4) to be updated to 2.3.5 or higher Update required: Snakeyaml API Plugin (1.27.0) to be updated to 1.29.1 or higher Update required: Pipeline: SCM Step (2.11) to be updated to 2.13 or higher Update required: SSH Credentials Plugin (1.18.1) to be updated to 1.19 or higher Update required: Trilead API Plugin (1.0.8) to be updated to 1.0.13 or higher Update required: Jenkins Apache HttpComponents Client 4.x API Plugin (4.5.10-2.0) to be updated to 4.5.13-1.0 or higher

Thanks for clarifying @chengjun0124 that your core Jenkins upgrade was via upgrading OCP.

Do you know if at some point since the 4.7 install was done, did anybody manually upgrade plugins from the Jenkins Console? The plugins you listed there as being the wrong version are plugins that are included in the image, and would only not be upgraded if someone previously manually upgraded those plugins from the jenkins console, and broke the symbollic links OCP uses during upgrades.

thanks again gabe

gabemontero commented 2 years ago

Hi @gabemontero The upgrade method I used was just change my deploymentConfig yaml. old yaml: Image: "registry.redhat.io/openshift4/ose-jenkins:v4.9.0-202111221622.p0.g20ddf94.assembly.stream" new yaml: image: "registry.redhat.io/openshift4/ose-jenkins:v4.9.0-202202080146.p0.ga71a115.assembly.stream" I didn't set OVERRIDE_PV_PLUGINS_WITH_IMAGE_PLUGINS, I only set OVERRIDE_RELEASE_MIGRATION_OVERWRITE=true. And my openshift cluster version is 4.7, I am not sure whether I can use the image with tag 4.9.0-x Then I got almost same logs with @ma-fall

Update required: Display URL API (2.3.4) to be updated to 2.3.5 or higher Update required: Snakeyaml API Plugin (1.27.0) to be updated to 1.29.1 or higher Update required: Pipeline: SCM Step (2.11) to be updated to 2.13 or higher Update required: SSH Credentials Plugin (1.18.1) to be updated to 1.19 or higher Update required: Trilead API Plugin (1.0.8) to be updated to 1.0.13 or higher Update required: Jenkins Apache HttpComponents Client 4.x API Plugin (4.5.10-2.0) to be updated to 4.5.13-1.0 or higher

Thanks for clarifying @chengjun0124 that your core Jenkins upgrade was via upgrading OCP.

Do you know if at some point since the 4.7 install was done, did anybody manually upgrade plugins from the Jenkins Console? The plugins you listed there as being the wrong version are plugins that are included in the image, and would only not be upgraded if someone previously manually upgraded those plugins from the jenkins console, and broke the symbollic links OCP uses during upgrades.

Also, setting OVERRIDE_PV_PLUGINS_WITH_IMAGE_PLUGINS to true would re-establish those symbollic links and allow for future OCP upgrades to put the various dependent plugins at the correct versions.

thanks again gabe

chengjun0124 commented 2 years ago

Yes, I had installed and upgraded the image with tag 4.9.0x successfully in openshift 4.7, so I assume I can use tag 4.9.0x in openshift 4.7.

I upgraded some plugins via jenkins UI(Updates page and Advanced page). Does this mean 'Jenkins Console'? Updates page:

Advanced page:

I upgraded these plugins: Bitbucket Branch Source Mailer Docker Commons Matrix Project Metrics Badge Credentials Binding Warnings Next Generation Configuration as Code

I didn't set OVERRIDE_PV_PLUGINS_WITH_IMAGE_PLUGINS=true, I don't want to override my plugins.

gabemontero commented 2 years ago

I upgraded some plugins via jenkins UI(Updates page and Advanced page). Does this mean 'Jenkins Console'?

Yes that is what I mean by "Jenkins Console" @chengjun0124

I didn't set OVERRIDE_PV_PLUGINS_WITH_IMAGE_PLUGINS=true, I don't want to override my plugins.

that's fine @chengjun0124, but then it is your responsibility to manually upgrade the plugins that were noted in your log as being at the wrong version after your upgrade. Things are working "as designed".

chengjun0124 commented 2 years ago

@gabemontero thank you.

chengjun0124 commented 2 years ago

Can I have a convenient way to upgrade jenkins core without manual upgrading plugins?

gabemontero commented 2 years ago

Can I have a convenient way to upgrade jenkins core without manual upgrading plugins?

OVERRIDE_PV_PLUGINS_WITH_IMAGE_PLUGINS=true is our answer for that @chengjun0124

chengjun0124 commented 2 years ago

@gabemontero Yes, OVERRIDE_PV_PLUGINS_WITH_IMAGE_PLUGINS=true works. But we cannot remain plugins' version as same as before upgrading unless we manually upgrade them after upgrading jenkins core. Manual upgrade takes additional efforts.

gabemontero commented 2 years ago

@gabemontero Yes, OVERRIDE_PV_PLUGINS_WITH_IMAGE_PLUGINS=true works.

OK great @chengjun0124 thanks for confirming

But we cannot remain plugins' version as same as before upgrading unless we manually upgrade them after upgrading jenkins core. Manual upgrade takes additional efforts.

Apologies, I'm having some difficulty parsing this statement ^^, though I think I understand.

I think what your are saying, if I revisit you comment https://github.com/openshift/jenkins/issues/1388#issuecomment-1054952342, is that you had to manually upgrade

Bitbucket Branch Source Mailer Docker Commons Matrix Project Metrics Badge Credentials Binding Warnings Next Generation Configuration as Code

and only on some of those were handled when you used OVERRIDE_PV_PLUGINS_WITH_IMAGE_PLUGINS=true

Is that correct?

If so, OK, yes, some of those plugins are NOT plugins we include in our image. You undoubtedly added them from the Jenkins UI. Hence OVERRIDE_PV_PLUGINS_WITH_IMAGE_PLUGINS=true will not handle them.

In general, we are NOT going to go down the path of managing plugins users have installed from the Jenkins UI that are not included in our base image and have gone through our testing.

At most, if you use the support we provide around extending the OpenShift Jenkins image and building your own Jenkins image, rebuilding of that image would get automation of updating dependencies. However, tt would be a multiple step process beyond an OCP upgrade:

build an image extension via BuildConfigs, where the output is stored in a new ImageStream ... see https://github.com/openshift/jenkins#adding-plugins-or-updating-existing-plugins for the options we provide for doing that
you could use ImageChangeTriggers on that BuildConfig to start a Build when the existing OpenShift Jenkins ImageStream changes (which would occur on an OCP upgrade),
where the build will run the install plugins script to find the later version of you non-payload plugins and install them, push to the new ImageStream,
and then an ImageChangeTrigger in your new DeploymentConfig would see that change and redeploy your Jenkins.

But yes, that is setting up a bunch of stuff vs. some "on/off" switch in our base OpenShift Jenkins image. But again, we are not going to add features to upgrade plugins we do not include.

Also, we have become very adverse to adding more plugins to our base image, and that increases the chances that we will have to deal with even more Jenkins Security Advisories, which is very expensive for us. So I don't see us adding the plugins you have which are NOT currently in our image.

chengjun0124 commented 2 years ago

OK, I understood. Thank you very much.

openshift / jenkins

Faild to upgrade 2.303.3 to 2.319.2 #1388