Closed timja closed 4 years ago
timja
commented
5 years ago We're also on jenkins 2.190-1 and hitting the same error. The Manage Jenkins page hits the stacktrace every time without fail, so we can't do any administration.
There's a new version so we can try and upgrade to that, but at the moment, this is very frustrating.
timja
commented
4 years ago Same issue, tried updating to 2.201 and updating all plugins and had no effect.
Doesn't always happen in the same "Jelly" file:
Manage Jenkins page it occurs inside the
On a Freestyle job main page it happens in the side panel after the "Configuration" task.
actions.jelly:39:70:
Tried backing up the ".jenkins" directory and doing a fresh install of 2.201 and that functioned but was of course missing all of my jobs, configuration, and plugins. (Either I was mistake or this is no longer the case, see later comments)
timja
commented
4 years ago bv2019, tharden0825, darrien, could you please provide the list of plugin that you have currently installed + the type of environement you are using? (Docker, VM, Windows, Linux, all potentially relevant information)
Thank you very much!
Due to the comment from Timothy, it seems like that there is a plugin doing something wrong with its ManagementLink.
timja
commented
4 years ago 2.189-2.192 had an issue with an admin monitor (JENKINS-59094), so those versions are known to be bad. Only if the original issue still happens on 2.193+ would this be interesting at all.
java.lang.StackOverflowError
Is a different error than reported here, so the comment is useless without a detailed error message and information on configuration.
timja
commented
4 years ago This issue is actually occuring on two servers for me (a build server and one functioning as a performance monitor). Attached are text files listing the contents of the "plugins" directory for each and an excel sheet that lists the contents that the two have in common. On the build server I've already decided to cut my losses and start from a fresh install and set it up the way I had it but the monitor is relatively untouched and can still be used for troubleshooting/info-gathering.
Both servers have a domain that was used for Jenkins' configured URL (instead of "localhost")
Both servers are running on:
Apache Tomcat 8.5.11 running as a service logged in with a service account.
Tomcat's Java Options: (some options are left over from troubleshooting an issue in the distant past and can probably be removed)
After issues started I added the following line to the context.xml to no effect:
timja
commented
4 years ago Correction, I don't know if something changed but upon fresh 2.201 installation with the suggested plugins, the issue showed up again.
So I re-re-installed 2.201 with no plugins and the issue was there yet again.
Seeing as this doesn't happen to everyone I'm assuming that, for my incarnation of this issue at least, this is being caused by something that happened on my machine without my knowledge such as an update to the OS or security software.
I'm going to talk to Infrastructure where I work and see if any work has been done on these servers recently. In the mean time, here is the stack trace from the bare-bones version and the content for the same time from the tomcat8-stderr log.
stacktrace from the manage page and tomcat8-stderr output for Tomcat startup, the setup for Jenkins (no plugins), and navigating to the manage jenkins page attached.
timja
commented
4 years ago tharden0825 Thanks for the update, that's very thorough!
Interestingly, this appears to be caused by one of the regular ManagementLink items rather than the (IME) more common culprit AdministrativeMonitor.
What I am surprised by is that the stack trace involves Groovy, when there are no info.groovy files for ManagementLink}}s in core. In fact, there's only a single {{info.jelly, and it's pretty trivial: https://github.com/jenkinsci/jenkins/blob/master/core/src/main/resources/jenkins/management/PluginsLink/info.jelly
Could you confirm again that this problem even occurs when you install Jenkins with the official jenkins.war file deployed into Tomcat, and no plugins installed at all? This just seems so strange.
BTW, from 2.200 on there's a better alternative to overriding hudson.model.DirectoryBrowserSupport.CSP, an option called "Resource root URL". Check that out when this is over 
timja
commented
4 years ago As a further diagnostic tool we can mess with the internals of your test instance a bit.
/script is the URL of the script console (docs: https://wiki.jenkins.io/display/JENKINS/Jenkins+Script+Console )
This will tell you all known links showing on the "Manage Jenkins" screen:
ManagementLink.all()
The following will temporarily (until restart) remove a link from that screen (and basically remove its functionality – depending on the link, some things will just be unavailable, e.g. the Global Security Configuration is implemented like that):
ManagementLink.all().remove(0)
(0 is the first entry in the list, and it's ordered the same way as they show up on the screen, i.e. System, Security, Tools, …).
I wonder whether you could use this & a few restarts as needed to determine the culprit(s).
That said, this looks like a very weird issue, so this might not lead anywhere. Perhaps a problem around Tomcat? If that bundles Groovy, for example, and its Groovy takes precedence over Jenkins', that would be a potential problem.
timja
commented
4 years ago Here's what I see when I install Jenkins 2.202 without plugins (upgraded again) see attached screenshots.
Afterward, I reverted to the backed up version with plugins so that the server could be used while I troubleshoot.
I'll try the script console next.
timja
commented
4 years ago Well... it's a step closer.
After trying many many things I decided to start trying to get rid of each error or warning individually to at least eliminate possible causes.
While troubleshooting the "reverse proxy is broken" issue that everyone insists won't impact Jenkins at all, I found this comment at the bottom of a help article. (see below)
I added the highlighted argument to the java options in tomcat and the manager page finally loaded without any errors. However, the Configure System and Configure Global Security links still take me to broken pages.
With the Security page referencing the charset decoder I'm wondering if it's another similar issue.
https://wiki.jenkins.io/display/JENKINS/Jenkins+says+my+reverse+proxy+setup+is+broken
timja
commented
4 years ago I reduced the java options in tomcat down to the following and everything works again.
I wouldn't close this issue until hearing from the people who opened it because their issue may be different than mine but I think it had something to do with limiting Tomcat's caching?
-Dcatalina.home=C:\Program Files\Apache Software Foundation\Tomcat 8.5
-Dcatalina.base=C:\Program Files\Apache Software Foundation\Tomcat 8.5
-Djava.io.tmpdir=C:\Program Files\Apache Software Foundation\Tomcat 8.5\temp
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
-Djava.util.logging.config.file=C:\Program Files\Apache Software Foundation\Tomcat 8.5\conf\logging.properties
-Djava.library.path="C:\Program Files\Apache Software Foundation\Tomcat 8.5\lib"
-Xmx3072m
-Xms3072m
-Dorg.apache.tomcat.util.buf.UDecoder.ALLOW_ENCODED_SLASH=true
timja
commented
4 years ago After fiddling with the monitor server java options it seems the culprit was this line: -Xss128k
(for me, at least)
timja
commented
4 years ago 64-bit JVMs have a default stack size of 1M, so that's quite a bit smaller. Unsurprising you're hitting SOEs earlier.
timja
commented
4 years ago Custom configuration with a much smaller stack size seems to be the culprit.
Keep getting this exception after upgrading Jenkins to Jenkins 2.189-1
Originally reported by bv2019, imported from: javax.servlet.ServletException: org.apache.commons.jelly.JellyTagException: jar:file:/var/cache/jenkins/war/WEB-INF/lib/jenkins-core-2.189.jar!/jenkins/model/Jenkins/manage.jelly:46:53: