[JENKINS-67550] ClassNotFoundException when making any change in a Pipeline project

[timja]

Problem: Making any change in the UI for pipeline project, will lead to an exception:

"...A problem occurred while processing the request..."

The jenkins.log file complains with a ClassNotFoundException. Please see below, or in detail in the attached logs.

Jenkins is up-and-running and serving all our build jobs, even the pipeline jobs. But we can't make any change via the Web/UI. Making manual changes in the filesystem and then reloading Jenkins works fine. Somehow we managed to break the UI, unfortunately we don't remember at what point.

We have another Jenkins instance running, which serves pretty much the same jobs, and this runs without problem.

 Note: Jenkins is running behind an NGINX reverse proxy.

2022-01-10 11:32:23.492+0000 [id=9] WARNING o.e.j.s.h.ContextHandler$Context#log: Error while serving https://XXXX.net/job/gum-test2/configSubmit2022-01-10 11:32:23.492+0000 [id=9] WARNING o.e.j.s.h.ContextHandler$Context#log: Error while serving https://jenkins.lane2.net/job/gum-test2/configSubmitjava.lang.ClassNotFoundException: cached miss for ["org.jenkinsci.plugins.workflow.cps.CpsFlowDefinition","org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition"] at hudson.PluginManager$UberClassLoader.findClass(PluginManager.java:2118)
 

 

I also upgraded to a more recent version and I have confirmed that that class should be available in the filesystem:

 

cd /var/lib/jenkins/plugins$
jar -tvf workflow-cps/WEB-INF/lib/workflow-cps.jar  | grep org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.class
 10850 Fri Dec 17 08:14:36 UTC 2021 org/jenkinsci/plugins/workflow/cps/CpsScmFlowDefinition.class

-> So the jar file and class exists.

From the logs, it looks like a class loading problem. Any hint howto debug that step is really appreciated.

 

---

Originally reported by martgut, imported from: ClassNotFoundException when making any change in a Pipeline project

• status: Open
• priority: Major
• resolution: Unresolved
• imported: 2022/01/10

[timja]

markewaite:

I'm unable to duplicate the problem as described. I used your plugins list from jenkins.txt with Jenkins 2.319.1 and created 3 different pipelines, one scripted, one declarative, and one with Pipeline definition from SCM. Each of those pipeline definitions could be edited and retained the edits.

Many of the plugins in your list are out of date. It looks like some have new releases that have been available for a month. You might try updating those plugins to the most recent releases to see if that resolves the issue.

The Jenkins 2.319.1 upgrade guide notes that Azure plugins are affected by the removal of the Woodstox implementation. The recommendation is to upgrade the affected Azure plugins to their most recent releases.

One of your Azure plugins (Azure Commons) is deprecated. It should be removed.

[timja]

JIRAUSER138377:

Thanks for the suggestions! Unfortunately upgrading the plug-ins did not really resolve the issue. Would it be possible to enable logging, in order to understand why the plugin / jar file was not loaded? From the exception it looks like the UberClassLoader does not see the plug-in? 
Anyway it maybe faster to re-install our Jenkins infrastructure, something I wanted to avoid for our production environment. 

 

[timja]

markewaite:

I don't have any other suggestions to offer. You might review the source code of the class loader to see if higher logging levels might help.

[timja]

ffloimair:

Hi! The plugins are now updated to latest version (expect Matrix Authorization and Azure AD as these have a new format, documentation for migration does not exist or at least cannot be easily found and would lock us out if used, so had to revert to previous version for now), the issue remains. I even tried running under Java 11 instead of 8 as before, but that did not help either.
As my colleague said, this is pretty much the same machine as another Jenkins instance running the same versions where we do not have the problem.
The issue started appearing approximately half a year ago and was not related to any plugin or Jenkins update. That is why we are pretty much out of ideas what could cause this.

[timja]

basil:

Similar to JENKINS-67218. In that case, the frontend was sending a $class value of a list rather than a single string to the backend. Looks like the same thing is happening here. The class loader obviously can't load the list. The root cause of JENKINS-67218 ended up being a tables-to-divs issue with a plugin, which was causing the frontend to send that wrong value. That also fits with ffloimair's timeline that this started happening approximately half a year ago. Tables-to-divs issues are a bit tricky to debug, but there is a process outlined at:

https://www.jenkins.io/doc/developer/views/table-to-div-migration/

[timja]

basil:

I opened jenkinsci/stapler#322 to make the error message better at least.