search

timja / jenkins-gh-issues-poc-06-18

0 stars 0 forks source link

[JENKINS-68443] HTTP POST of hpi files to pluginManager causes ArrayOutOfBoundException #1999

Open timja opened 2 years ago

timja commented 2 years ago

In the past years we were able to successfully automate plugin installation by HTTP POSTing `hpi` files to jenkins instances (via `curl`).

Starting with version 2.332 this now returns an HTTP 500, caused by an `IndexOutOfBoundsException` (full stack trace bellow). We first recognized this issue in 2.332. We're positive it did not occurr in 2.289.3. This issue still persists in the current version 2.346.

Ironically, uploading the plugin via Jenkins' web interface still works. I even analyzed the HTTP packages on the wire but could not see what difference there is between success via Web UI and failure using curl. I also tried HTTPie, which also fails.

For reference: Where and why do we do this, see also github.com/cloudogu/gitops-playground#86

The issue can be reproduced like so:

 

docker run --name jenkins -p 8080:8080 -v jenkins_home:/var/jenkins_home jenkins/jenkins:2.346-jdk11

docker exec jenkins cat /var/jenkins_home/secrets/initialAdminPassword
# Go to localhost:8080
# Init admin user with admin:admin; Not necessary to install any plugins

# Download some plugin for testing
wget -P/tmp http://updates.jenkins-ci.org/download/plugins/ace-editor/1.1/ace-editor.hpi

curl -s \
  -H Jenkins-Crumb:$(curl -s --cookie-jar /tmp/cookies --retry 3 --retry-delay 1 -u admin:admin --write-out '%{json}' http://localhost:8080/crumbIssuer/api/json |  jq -rsc '(.[0] | .crumb)') \
  --cookie /tmp/cookies -u admin:admin --fail -L -o /dev/null --write-out '%{http_code}' '-F file=@/tmp/ace-editor.hpi' \
  http://localhost:8080/pluginManager/uploadPlugin

docker logs jenkins

Full stack trace:

 

2022-05-09 11:55:53.535+0000 [id=16]    WARNING h.i.i.InstallUncaughtExceptionHandler#handleException: Caught unhandled exception with ID b3a78aea-ec6a-4e95-93a0-4c2d729b7862
java.lang.IndexOutOfBoundsException: Index 1 out of bounds for length 1
        at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)
        at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)
        at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248)
        at java.base/java.util.Objects.checkIndex(Objects.java:372)
        at java.base/java.util.ArrayList.get(ArrayList.java:459)
        at hudson.PluginManager.doUploadPlugin(PluginManager.java:1807)
        at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:710)
        at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:397)
        at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:409)
        at org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:78)
        at org.kohsuke.stapler.PreInvokeInterceptedFunction.invoke(PreInvokeInterceptedFunction.java:26)
        at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:207)
        at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:140)
        at org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:558)
        at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:59)
        at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
Caused: javax.servlet.ServletException
        at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:816)
        at org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
        at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:172)
        at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:59)
        at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
        at org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
        at org.kohsuke.stapler.Stapler.invoke(Stapler.java:694)
        at org.kohsuke.stapler.Stapler.service(Stapler.java:240)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
        at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1631)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:157)
        at jenkins.security.ResourceDomainFilter.doFilter(ResourceDomainFilter.java:81)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
        at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:129)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
        at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:160)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
        at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:154)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:94)
        at jenkins.security.AcegiSecurityExceptionFilter.doFilter(AcegiSecurityExceptionFilter.java:52)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
        at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:54)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
        at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:122)
        at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:116)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
        at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:109)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
        at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:102)
        at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:93)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
        at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:219)
        at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:213)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
        at jenkins.security.BasicHeaderProcessor.success(BasicHeaderProcessor.java:139)
        at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:86)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
        at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:110)
        at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:80)
        at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:63)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
        at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:111)
        at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:172)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
        at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:53)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
        at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:86)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
        at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
        at jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:38)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:571)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.Server.handle(Server.java:516)
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
        at java.base/java.lang.Thread.run(Thread.java:829)

 

 

 

Originally reported by jschnatterer, imported from: HTTP POST of hpi files to pluginManager causes ArrayOutOfBoundException
  • status: Open
  • priority: Minor
  • resolution: Unresolved
  • imported: 2022/01/10
timja commented 2 years ago

basil:

Hello jschnatterer, since you have been so kind as to provide steps to reproduce the problem from scratch, would you also be willing to run git bisect on Jenkins core and identify the commit that introduced the problem? This would help us find the cause and ultimately either revert the offending change or develop a solution.

timja commented 2 years ago

timja:

It will be this:

https://github.com/jenkinsci/jenkins/pull/5862

This is not a supported API, it's using web browser only methods.

For supported options please see https://www.jenkins.io/doc/book/managing/plugins/#installing-a-plugin

Or

https://github.com/jenkinsci/plugin-installation-manager-tool

Or you can adapt your API call to include the web ui parameter

timja commented 2 years ago

JIRAUSER143974:

Thanks for your responses basil and timja! I think I understand the problem now. Here is my thesis:

The browser sends "Content-Type: application/vnd.hp-hpid" within the form-data (not as HTTP header). Curl sends "application/octet-stream". This leads to the uploading being interpreted as url (not file), which leads to the IndexOutOfBoundsException, almost as anticipated in this discussion within PR 5862.

Does this sound plausible? AFAIK there is no way to change the content-type within the form data using curl. Do you have any ideas for me how to work around this or how to make this happen on the jenkins side?

I actually viewed all of the docs and tools posted by timja before using this unsupported API.  I am running in an environment external to Jenkins with no JVM. This rules out jenkins cli, plugin-installation-manager-tool and install-plugins.sh  😐️

timja commented 2 years ago

timja:

I think it's just that you need to add a parameter to the post whether it's a file or url

timja commented 2 years ago

JIRAUSER143974:

Ah, I see! Just an arbitrary empty form parameter to avoid the Exception?  I just added -F 'a=' and it returned HTTP 200  🎉

So this looks like a valid workaround. Thanks very much!

It could probably be avoided,though, using a length check, as you stated in your comment on the PR in November 21.

 

timja commented 2 years ago

timja:

could be, but this is really not supported.

timja commented 2 years ago

basil:

jschnatterer This is an open source project, so you are welcome to file a pull request to change the behavior to whatever you think the behavior should be, as long as the existing UI remains working. The maintainers have no plans on working on this ticket since your use case is not officially supported.

timja commented 2 years ago

JIRAUSER143974:

As long as someone is willing to review the PR, I might actually create one, next time I'm on this topic. Looks like a perfect first issue.

Thanks for your support so far!

timja commented 2 years ago

basil:

jschnatterer Both Tim and I are core maintainers, and we generally aim to provide timely reviews for all reasonable PRs.