timja
commented
6 years ago All classes from modules should be serializable, will try to reproduce
Closed timja closed 6 years ago
timja
commented
6 years ago All classes from modules should be serializable, will try to reproduce
timja
commented
6 years ago notanother I tried to reproduce it manually and in unit tests, no success so far.
Any chance that your instance defines a custom class filter?
timja
commented
6 years ago Most likely the problem lies in ClassFilterImpl.isPluginManifest as called by isLocationWhitelisted. The reporter is running on Tomcat rather than the built-in Winstone like most users, so that is very likely the cause. Now 2.104 fixed JENKINS-49147 but perhaps you are seeing some similar issue caused by another weird Tomcat behavior, perhaps depending on the specific version being run. You can turn on logging on ClassFilterImpl to pinpoint the problem easily.
timja
commented
6 years ago Changing status to reflect the fact that the filed PR merely confirms that there is nothing broken in Jenkins core under normal conditions; it does not pretend to fix the issue as reported.
timja
commented
6 years ago I did not see anything from turning on logging for ClassFilterImpl and triggering the exception.
After reading through JENKINS-49147 I upgraded our Tomcat server from 8.0.12 to 8.0.50(latest 8.0), this resolves the issue. I also tried upgrading to 8.5.28(latest 8.5) and confirmed that also resolves the issue.
Out of curiosity, is running Jenkins in Tomcat a "supported" platform or should I look to moving to the built-in servlet container?
timja
commented
6 years ago It is definitely "supported", but AFAIK we test Jenkins only with embedded Jetty web container. Maybe rarabaolaza and vilacides know about specific TomCat tests.
In the case of this ticket I will think how to properly update https://wiki.jenkins.io/display/JENKINS/Plugins+affected+by+fix+for+JEP-200 to reflect TomCat compat issues as well
timja
commented
6 years ago No tomcat tests that I am aware of 
timja
commented
6 years ago Code changed in jenkins
User: Oleg Nenashev
Path:
core/src/main/java/jenkins/security/ClassFilterImpl.java
test/src/test/java/jenkins/security/ClassFilterImplTest.java
http://jenkins-ci.org/commit/jenkins/800668ba4305964afe59d8744fcfc24013ff6ee6
Log:
JENKINS-49543 - Add direct unit test for module class whitelisting
timja
commented
6 years ago Code changed in jenkins
User: Oleg Nenashev
Path:
core/src/main/java/jenkins/security/ClassFilterImpl.java
test/src/test/java/jenkins/security/ClassFilterImplTest.java
http://jenkins-ci.org/commit/jenkins/04bd7e60ca8954d7665febb1f6f7663598ac67d9
Log:
Merge pull request #3290 from oleg-nenashev/tests/JENKINS-49543
JENKINS-49543 - Add direct unit test for module class whitelisting
Compare: https://github.com/jenkinsci/jenkins/compare/59bfe03ebe70...04bd7e60ca89
timja
commented
6 years ago notanotherI have added the documentation to https://wiki.jenkins.io/display/JENKINS/Plugins+affected+by+fix+for+JEP-200#PluginsaffectedbyfixforJEP-200-Otheraffectedcomponents/configurations , PTAL
I will check whether it is possible to extend the patch quickly, but maybe we could agree that the update is the feasible mitigation
timja
commented
6 years ago I agree that updating Tomcat is the correct mitigation route. This is reinforced by the fact that Tomcat 8.0.x line is officially entering EOL soon. (announcement)
Thanks for your help in resolving this.
timja
commented
6 years ago Obviously updating to the most recent supported version of Tomcat would be a good idea, if you need to run Jenkins on Tomcat at all (most people use the built-in Jetty server), but that does not change the fact that there appears to be a regression in Jenkins core related to certain Tomcat versions and we would like to correct that regression. Once we either know how to reproduce from scratch, or have access to sufficient field diagnostics, the fix is likely simple.
I did not see anything from turning on logging for ClassFilterImpl and triggering the exception.
Then you did not properly configuring logging. There should be a logger for jenkins.security.ClassFilterImpl registering messages at FINE or above. At some point, probably early in startup (most likely long before you try to reconfigure a user and see the error), there should be a message from the isLocationWhitelisted method about ssh-cli-auth-1.4.jar. When Jenkins is operating normally, this should be saying
… seems to be a Jenkins plugin, OK
followed by a message from isBlacklisted saying
… permitting … due to its location in …
In your case, I suspect there is some other message being logged from isLocationWhitelisted, most likely
… is not recognized; rejecting
where the message is showing a URL which is not in the expected format file://some/path/to/ssh-cli-auth-1.4.jar. At least, that was the root cause of JENKINS-49147, so I am guessing this one is similar.
BTW it is advisable to install the support-core plugin as that will ensure that all output from custom loggers is captured to log files on disk and included in a ZIP file you can share (in part or in whole). Jenkins core only saves (by default) the last 256 messages from any given logger, so you might miss the critical messages from ClassFilterImpl in scrollback.
timja
commented
6 years ago jglick do you want to work on that? If no, I suggest closing it as Won't Fix for now
timja
commented
6 years ago Then you did not properly configuring logging.
I did, but I did not know that what I was looking for was during start up.
it is advisable to install the support-core plugin
Thanks that is a handy plugin. I have it installed now.
I have downgraded our Tomcat version to get you the information requested. I attached the logs from the jenkins.security.ClassFilterImpl recorder. If you need a different log, please let me know soon as I will need to put our server back on Tomcat 8.5.28.
Here is the line that seems relevant from the log:
2018-02-19 18:09:58.169+0000 [id=25] FINE jenkins.security.ClassFilterImpl: jar:file:/srv/tomcat/tomcat_app/webapps/jenkins/WEB-INF/lib/ssh-cli-auth-1.4.jar!/ is not recognized; rejecting
As far as running Jenkins on the embedded Jetty server, is there any information about migrating from Tomcat to the embedded server? We ran our server using Tomcat as that was what was convenient when we setup our server ~3 years ago, but obviously are interested in running our Jenkins instance in the most supported way possible.
Thank you for your assistance on this issue.
timja
commented
6 years ago Indeed that URL pattern is not currently recognized; rather than the expected file:/srv/tomcat/tomcat_app/webapps/jenkins/WEB-INF/lib/ssh-cli-auth-1.4.jar Tomcat is producing jar:file:/srv/tomcat/tomcat_app/webapps/jenkins/WEB-INF/lib/ssh-cli-auth-1.4.jar!/. Fix should be trivial but let me see if I can reproduce it.
timja
commented
6 years ago Reproduced using the same setup as in JENKINS-49147 but extended by configuring a security realm, logging in as some user, and attempting to reconfigure that user.
timja
commented
6 years ago Code changed in jenkins
User: Jesse Glick
Path:
core/src/main/java/jenkins/security/ClassFilterImpl.java
http://jenkins-ci.org/commit/jenkins/376c6a0add41e0c2049b64edfdd464bb8717ed1b
Log:
JENKINS-49543 Old versions of Tomcat also failed to serialize classes from Jenkins modules.
timja
commented
6 years ago Code changed in jenkins
User: Oleg Nenashev
Path:
core/src/main/java/jenkins/security/ClassFilterImpl.java
http://jenkins-ci.org/commit/jenkins/262a7a1345e6847a7f075eba0bde3a3d31bda6fa
Log:
Merge pull request #3313 from jglick/Tomcat-redux-JENKINS-49543
JENKINS-49543 Old versions of Tomcat also failed to serialize classes from Jenkins modules
Compare: https://github.com/jenkinsci/jenkins/compare/c33f14620425...262a7a1345e6
timja
commented
6 years ago It has been merged towards 2.110.
olivergondza We do not know how many users run on the old Tomcat versions. Since the fix is narrow-scoped, would it make sense to add it to 2.107.1-rc? Or should we add it to known issues in the upgrade guide and postpone it till .2?
timja
commented
6 years ago Though it certainly does not meet the usual “soak period” criteria, I would advocate backporting this to 2.107.1 since the fix seems pretty safe and demonstrably fixes a serious regression (compared to the previous LTS) for users in this environment. But waiting for 2.107.2 is probably acceptable as well if the issue is noted in the upgrade guide—the workaround after all is to just upgrade Tomcat (or stop using it altogether).
timja
commented
6 years ago Code changed in jenkins
User: Oleg Nenashev
Path:
core/src/main/java/jenkins/security/ClassFilterImpl.java
test/src/test/java/jenkins/security/ClassFilterImplTest.java
http://jenkins-ci.org/commit/jenkins/2ce5036cb06a7dab0d4868e9539c8d42e7a5678c
Log:
JENKINS-49543 - Add direct unit test for module class whitelisting
(cherry picked from commit 800668ba4305964afe59d8744fcfc24013ff6ee6)
timja
commented
6 years ago Code changed in jenkins
User: Jesse Glick
Path:
core/src/main/java/jenkins/security/ClassFilterImpl.java
http://jenkins-ci.org/commit/jenkins/dd3ddf3ceb6428dc0b3a15148d65e8baece0a42c
Log:
JENKINS-49543 Old versions of Tomcat also failed to serialize classes from Jenkins modules.
(cherry picked from commit 376c6a0add41e0c2049b64edfdd464bb8717ed1b)
Compare: https://github.com/jenkinsci/jenkins/compare/db0bddeb2cb5...dd3ddf3ceb64
timja
commented
6 years ago Agreed this can be quite severe and the fix seems fairly straightforward. Though as the fix is unreleased for now, it will be reverted during RC period in case it will cause problems. It will be part of the RC I will push tomorrow unless tests suggests otherwise.
timja
commented
6 years ago The fix has been integrated towards 2.110
Workaround: Update to Apache Tomcat 8.0.50 or above
When saving on the configuration page for a user (http://cool.jenkins.url/user/user.name/configure) I get the following stack trace.
Adding "-Dhudson.remoting.ClassFilter=org.jenkinsci.main.modules.cli.auth.ssh.UserPropertyImpl" fixes the issue.
This seems to also be causing issues for workflow-cps-global-lib-plugin's local git repository.
Stack Trace:
java.lang.UnsupportedOperationException: Refusing to marshal org.jenkinsci.main.modules.cli.auth.ssh.UserPropertyImpl for security reasons; see https://jenkins.io/redirect/class-filter/ at hudson.util.XStream2$BlacklistedTypesConverter.marshal(XStream2.java:543) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:88) at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeItem(AbstractCollectionConverter.java:64) at com.thoughtworks.xstream.converters.collections.CollectionConverter.marshal(CollectionConverter.java:74) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84) at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:265) at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:252) Caused: java.lang.RuntimeException: Failed to serialize hudson.model.User#properties for class hudson.model.User at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:256) at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:224) at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138) at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:209) at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:150) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43) at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:82) at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37) at com.thoughtworks.xstream.XStream.marshal(XStream.java:1026) at com.thoughtworks.xstream.XStream.marshal(XStream.java:1015) at com.thoughtworks.xstream.XStream.toXML(XStream.java:988) at hudson.XmlFile.write(XmlFile.java:193) Caused: java.io.IOException at hudson.XmlFile.write(XmlFile.java:200) at hudson.model.User.save(User.java:827) at hudson.model.User.doConfigSubmit(User.java:901) at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627) at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:343) at org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:77) at org.kohsuke.stapler.PreInvokeInterceptedFunction.invoke(PreInvokeInterceptedFunction.java:26) at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:184) at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:117) at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:129) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845) at org.kohsuke.stapler.MetaClass$5.doDispatch(MetaClass.java:248) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649) at org.kohsuke.stapler.Stapler.service(Stapler.java:238) at javax.servlet.http.HttpServlet.service(HttpServlet.java:725) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154) at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:225) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:61) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at com.smartcodeltd.jenkinsci.plugin.assetbundler.filters.LessCSS.doFilter(LessCSS.java:47) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:237) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:214) at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:88) at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:114) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:59) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:64) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:616) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:142) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:534) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1081) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:658) at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:222) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1566) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1523) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745)Plugins
Originally reported by notanother, imported from: Refusing to marshal org.jenkinsci.main.modules.cli.auth.ssh.UserPropertyImpl on Old Apache TomCat 8.x versions