search

gluster / project-infrastructure

Issues related to GlusterFs infrastructure components.
0 stars 0 forks source link

Jenkins at build.gluster.org is not working #179

Closed xhernandez closed 1 year ago

xhernandez commented 1 year ago

Connecting to https://jenkins.gluster.org, returns a page with the Jenkins header but no content. It seems that something is broken in Jenkins.

mscherer commented 1 year ago

Seems a automated update of jenkins happened. I restarted it, no luck. I am investigating.

mscherer commented 1 year ago

So it fail with this error as seen with curl:

HTTP ERROR 500 javax.servlet.ServletException: org.apache.commons.jelly.JellyTagException: jar:file:/var/lib/jenkins/%25C/jenkins/war/WEB-INF/lib/jenkins-core-2.387.1.jar!/jenkins/views/Jenkins* TLSv1.2 (IN), TLS header, Supplemental data (23):
Header/headerContent.jelly:71:88: <st:include> org/apache/commons/httpclient/URIException
URI:/
STATUS:500
MESSAGE:javax.servlet.ServletException: org.apache.commons.jelly.JellyTagException: jar:file:/var/lib/jenkins/%25C/jenkins/war/WEB-INF/lib/jenkins-core-2.387.1.jar!/jenkins/views/JenkinsHeader/headerContent.jelly:71:88: <st:include> org/apache/commons/httpclient/URIException
SERVLET:Stapler
CAUSED BY:javax.servlet.ServletException: org.apache.commons.jelly.JellyTagException: jar:file:/var/lib/jenkins/%25C/jenkins/war/WEB-INF/lib/jenkins-core-2.387.1.jar!/jenkins/views/JenkinsHeader/headerContent.jelly:71:88: <st:include> org/apache/commons/httpclient/URIException
CAUSED BY:org.apache.commons.jelly.JellyTagException: jar:file:/var/lib/jenkins/%25C/jenkins/war/WEB-INF/lib/jenkins-core-2.387.1.jar!/jenkins/views/JenkinsHeader/headerContent.jelly:71:88: <st:include> org/apache/commons/httpclient/URIException
CAUSED BY:java.lang.NoClassDefFoundError: org/apache/commons/httpclient/URIException
javax.servlet.ServletException: org.apache.commons.jelly.JellyTagException: jar:file:/var/lib/jenkins/%25C/jenkins/war/WEB-INF/lib/jenkins-core-2.387.1.jar!/jenkins/views/JenkinsHeader/headerContent.jelly:71:88: <st:include> org/apache/commons/httpclient/URIException
    at org.kohsuke.stapler.Facet$1.dispatch(Facet.java:249)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:770)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:900)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:698)
    at hudson.init.impl.InstallUncaughtExceptionHandler.handleException(InstallUncaughtExceptionHandler.java:59)
    at hudson.init.impl.InstallUncaughtExceptionHandler.lambda$init$0(InstallUncaughtExceptionHandler.java:33)
    at org.kohsuke.stapler.compression.CompressionFilter.repo* TLSv1.2 (IN), TLS header, Supplemental data (23):
rtException(CompressionFilter.java:72)
    at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:56)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
    at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:86)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
    at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
    at jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:38)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:549)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1570)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1383)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1543)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1305)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
    at org.eclipse.jetty.server.Server.handle(Server.java:563)
    at org.eclipse.jetty.server.HttpChannel.lambda$handle$0(HttpChannel.java:505)
    at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:762)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:497)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:282)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
    at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
    at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:416)
    at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:385)
    at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:272)
    at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.lambda$new$0(AdaptiveExecutionStrategy.java:140)
    at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:934)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1078)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.commons.jelly.JellyTagException: jar:file:/var/lib/jenkins/%25C/jenkins/war/WEB-INF/lib/jenkins-core-2.387.1.jar!/jenkins/views/JenkinsHeader/headerContent.jelly:71:88: <st:include> org/apache/commons/httpclient/URIException
    at org.apache.commons.jelly.impl.TagScript.handleException(TagScript.java:738)
    at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:285)
    at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
    at org.apache.commons.jelly.tags.core.OtherwiseTag.doTag(OtherwiseTag.java:41)
    at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:265)
    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
    at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
    at org.apache.commons.jelly.tags.core.ChooseTag.doTag(ChooseTag.java:38)
    at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:265)
    at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)
    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
    at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:101)
    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
    at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:101)
    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
    at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
    at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:99)
    at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:172)
    at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:265)
    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
    at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
    at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:122)
    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
    at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)
    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
    at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:101)
* TLSv1.2 (IN), TLS header, Supplemental data (23):
    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
    at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:101)
    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
    at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:101)
    at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
    at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:265)
    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
    at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
    at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:122)
    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
    at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
    at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:122)
    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
    at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
    at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:99)
    at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:66)
    at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:55)
    at org.kohsuke.stapler.jelly.ScriptInvoker.execute(ScriptInvoker.java:56)
    at org.kohsuke.stapler.jelly.ScriptInvoker.execute(ScriptInvoker.java:43)
    at org.kohsuke.stapler.Facet$1.dispatch(Facet.java:240)
    ... 49 more
Caused by: java.lang.NoClassDefFoundError: org/apache/commons/httpclient/URIException
    at java.base/java.lang.Class.getDeclaredMethods0(Native Method)
    at java.base/java.lang.Class.privateGetDeclaredMethods(Class.java:3166)
    at java.base/java.lang.Class.getDeclaredMethods(Class.java:2309)
    at org.kohsuke.stapler.ClassDescriptor.findMethods(ClassDescriptor.java:143)
    at org.kohsuke.stapler.ClassDescriptor.<init>(ClassDescriptor.java:80)
    at org.kohsuke.stapler.lang.KlassNavigator$1.getFunctions(KlassNavigator.java:217)
    at org.kohsuke.stapler.lang.KlassNavigator$1.getFunctions(KlassNavigator.java:142)
* TLSv1.2 (IN), TLS header, Supplemental data (23):
    at org.kohsuke.stapler.lang.Klass.getFunctions(Klass.java:98)
    at org.kohsuke.stapler.KlassDescriptor.<init>(KlassDescriptor.java:27)
    at org.kohsuke.stapler.MetaClass.buildDispatchers(MetaClass.java:111)
    at org.kohsuke.stapler.MetaClass.<init>(MetaClass.java:99)
    at org.kohsuke.stapler.WebApp$1.computeValue(WebApp.java:230)
    at org.kohsuke.stapler.WebApp$1.computeValue(WebApp.java:227)
    at java.base/java.lang.ClassValue.getFromHashMap(ClassValue.java:228)
    at java.base/java.lang.ClassValue.getFromBackup(ClassValue.java:210)
    at java.base/java.lang.ClassValue.get(ClassValue.java:116)
    at org.kohsuke.stapler.WebApp.getMetaClass(WebApp.java:246)
    at org.kohsuke.stapler.WebApp.getMetaClass(WebApp.java:240)
    at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:134)
    at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:265)
    ... 93 more
mscherer commented 1 year ago

I am downgrading the jenkins RPM for the moment.

mscherer commented 1 year ago

Ok so that's back. Now, I need to see what happened, and how to block the upgrade for now.

mscherer commented 1 year ago

So I think one library was removed from jenkins:

https://github.com/jenkinsci/jenkins/pull/7312

And some plugins need to be updated. I will update them, try the update, and if not, I will revert the jenkins update until I find the plugin causing trouble.

mscherer commented 1 year ago

So, while on it, I removed all plugins who were not used, but still installed (like the gerrit one...). I am updating everything, and it will restart again.

Then I will check why it take ages to find a builder.

mscherer commented 1 year ago

Ok so I updated jenkins and plugins and now it seems to work fine. I am finishing the last ones.

However, if anything is broken, do not hesitate to ping us.

mscherer commented 1 year ago

And closing. Jenkins is back, I updated everything possible, and there is no warning.

xhernandez commented 1 year ago

Thanks for looking into it, but it seems something is not working fine yet:

https://build.gluster.org/job/gh_devrpm-el7/3958/console

13:02:41 GitHub pull request #4032 of commit 540fe33f1cdfcea87593bd0245bad9919ed3dbcc, no merge conflicts.
13:02:41 Running as SYSTEM
13:02:47 Setting status of 540fe33f1cdfcea87593bd0245bad9919ed3dbcc to PENDING with url https://build.gluster.org/job/gh_devrpm-el7/3958/ and message: 'Running: devrpm-el7'
13:02:47 Using context: Testing: devrpm-el7
13:02:47 Building remotely on [builder-c7-4.int.aws.gluster.org](https://build.gluster.org/computer/builder-c7-4.int.aws.gluster.org) (asan centos7) in workspace /home/jenkins/root/workspace/gh_devrpm-el7
13:02:48 The recommended git tool is: NONE
13:02:55 No credentials specified
13:02:55 Wiping out workspace first.
13:02:56 Also:   java.nio.file.FileSystemException: /home/jenkins/root/workspace/gh_devrpm-el7/RPMS/el7/x86_64/hw_info.log: Operation not permitted
13:02:56        at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
13:02:56        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
13:02:56        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
13:02:56        at java.base/sun.nio.fs.UnixFileAttributeViews$Posix.setMode(UnixFileAttributeViews.java:254)
13:02:56        at java.base/sun.nio.fs.UnixFileAttributeViews$Posix.setPermissions(UnixFileAttributeViews.java:276)
[...]
13:03:07 jenkins.util.io.CompositeIOException: Unable to delete '/home/jenkins/root/workspace/gh_devrpm-el7'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts. (Discarded 25 additional exceptions)
13:03:07    at jenkins.util.io.PathRemover.forceRemoveDirectoryContents(PathRemover.java:87)
13:03:07    at hudson.Util.deleteContentsRecursive(Util.java:286)
13:03:07    at hudson.FilePath$DeleteContents.invoke(FilePath.java:1459)
13:03:07    at hudson.FilePath$DeleteContents.invoke(FilePath.java:1454)
13:03:07    at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3578)
13:03:07    at hudson.remoting.UserRequest.perform(UserRequest.java:211)
[...]
rakshithakamath94 commented 1 year ago

Oh that's just a cleanup issue!

rakshithakamath94 commented 1 year ago

You can trigger again and check!

mscherer commented 1 year ago

That's weird:

# ls -l /home/jenkins/root/workspace/gh_devrpm-el7/RPMS/el7/x86_64/hw_info.log 
-rwxr-xr-x. 1 jenkins jenkins 1418 Mar 14 12:34 /home/jenkins/root/workspace/gh_devrpm-el7/RPMS/el7/x86_64/hw_info.log
mscherer commented 1 year ago

Ok, I cleaned it manually, we will see how it goes.

xhernandez commented 1 year ago

It seems to be working now. Thanks !!!