search

openshift / jenkins-plugin

Apache License 2.0
81 stars 50 forks source link

Jenkins plugin using default serviceaccount instead of "jenkins" #103

Closed jupierce closed 7 years ago

jupierce commented 7 years ago

Still trying to figure out what changed here, but several origin extended test failures have induced me to track this problem. When running within OpenShift, the Jenkins plugin is attempting to access APIServer resources using the "default" service account instead of the "jenkins" service account.

@gabemontero @oatmealraisin @csrwng

Errors:

  1. Starting "Tag OpenShift Image" with the source [image stream:tag] "multitag:orig" from the project "extended-test-jenkins-plugin-eg2xp-6vdhc" and destination stream(s) "multitag" with tag(s) "prod" from the project "extended-test-jenkins-plugin-eg2xp-6vdhc".
ERROR: Build step failed with exception
com.openshift.restclient.authorization.ResourceForbiddenException: User "system:serviceaccount:extended-test-jenkins-plugin-eg2xp-6vdhc-jenkins:default" cannot get imagestreams in project "extended-test-jenkins-plugin-eg2xp-6vdhc" User "system:serviceaccount:extended-test-jenkins-plugin-eg2xp-6vdhc-jenkins:default" cannot get imagestreams in project "extended-test-jenkins-plugin-eg2xp-6vdhc"
at com.openshift.internal.restclient.okhttp.ResponseCodeInterceptor.createOpenShiftException(ResponseCodeInterceptor.java:106)
at com.openshift.internal.restclient.okhttp.ResponseCodeInterceptor.intercept(ResponseCodeInterceptor.java:65)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:170)
at okhttp3.RealCall.execute(RealCall.java:60)
at com.openshift.internal.restclient.DefaultClient.execute(DefaultClient.java:217)
at com.openshift.internal.restclient.DefaultClient.execute(DefaultClient.java:194)
at com.openshift.internal.restclient.DefaultClient.execute(DefaultClient.java:183)
at com.openshift.internal.restclient.DefaultClient.get(DefaultClient.java:291)
at com.openshift.jenkins.plugins.pipeline.model.IOpenShiftImageTagger.coreLogic(IOpenShiftImageTagger.java:139)
at com.openshift.jenkins.plugins.pipeline.model.IOpenShiftPlugin.doItCore(IOpenShiftPlugin.java:299)
at com.openshift.jenkins.plugins.pipeline.model.IOpenShiftPlugin.doIt(IOpenShiftPlugin.java:312)
at com.openshift.jenkins.plugins.pipeline.OpenShiftBaseStep.perform(OpenShiftBaseStep.java:81)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:782)
at hudson.model.Build$BuildExecution.build(Build.java:205)
at hudson.model.Build$BuildExecution.doRun(Build.java:162)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:534)
at hudson.model.Run.execute(Run.java:1738)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:410)
Build step 'Tag OpenShift Image' marked build as failure
Finished: FAILURE

  2. https://ci.openshift.redhat.com/jenkins/job/origin_extended_image_tests/731/consoleFull

    [image_ecosystem][jenkins] schedule jobs on pod slaves
/data/src/github.com/openshift/origin/test/extended/image_ecosystem/kubernetes_plugin.go:169
use of jenkins with kubernetes plugin
/data/src/github.com/openshift/origin/test/extended/image_ecosystem/kubernetes_plugin.go:168
by creating slave from existing builder and adding it to Jenkins master [It]
/data/src/github.com/openshift/origin/test/extended/image_ecosystem/kubernetes_plugin.go:167

Expected
    <string>: User "system:serviceaccount:extended-test-jenkins-kube-a67wm-tv3ra:default" cannot list imagestreams in project "extended-test-jenkins-kube-a67wm-tv3ra"
    User "system:serviceaccount:extended-test-jenkins-kube-a67wm-tv3ra:default" cannot list imagestreams in project "extended-test-jenkins-kube-a67wm-tv3ra"
    Generating kubernetes-plugin configuration (/opt/openshift/configuration/config.xml.tpl) ...
    Generating kubernetes-plugin credentials (/var/lib/jenkins/credentials.xml.tpl) ...
    Copying Jenkins configuration to /var/lib/jenkins ...
    Copying 73 Jenkins plugins to /var/lib/jenkins ...
    Creating initial Jenkins 'admin' user ...
    Running from: /usr/lib/jenkins/jenkins.war
    webroot: EnvVars.masterEnvVars.get("JENKINS_HOME")
    Oct 31, 2016 5:29:38 AM org.eclipse.jetty.util.log.JavaUtilLog info
    INFO: Logging initialized @626ms
    Oct 31, 2016 5:29:38 AM winstone.Logger logInternal
    INFO: Beginning extraction from war file
    Oct 31, 2016 5:29:39 AM org.eclipse.jetty.util.log.JavaUtilLog warn
    WARNING: Empty contextPath
    Oct 31, 2016 5:29:39 AM org.eclipse.jetty.util.log.JavaUtilLog info
    INFO: jetty-9.2.z-SNAPSHOT
    Oct 31, 2016 5:29:41 AM org.eclipse.jetty.util.log.JavaUtilLog info
    INFO: NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet
    Jenkins home directory: /var/lib/jenkins found at: EnvVars.masterEnvVars.get("JENKINS_HOME")
    Oct 31, 2016 5:29:41 AM org.eclipse.jetty.util.log.JavaUtilLog info
    INFO: Started w.@51cd7ffc{/,file:/var/lib/jenkins/war/,AVAILABLE}{/var/lib/jenkins/war}
    Oct 31, 2016 5:29:41 AM org.eclipse.jetty.util.log.JavaUtilLog info
    INFO: Started ServerConnector@6c6c5427{HTTP/1.1}{0.0.0.0:8080}
    Oct 31, 2016 5:29:41 AM org.eclipse.jetty.util.log.JavaUtilLog info
    INFO: Started @4137ms
    Oct 31, 2016 5:29:41 AM winstone.Logger logInternal
    INFO: Winstone Servlet Engine v2.0 running: controlPort=disabled
    Oct 31, 2016 5:29:42 AM jenkins.InitReactorRunner$1 onAttained
    INFO: Started initialization
    Oct 31, 2016 5:29:48 AM jenkins.InitReactorRunner$1 onAttained
    INFO: Listed all plugins
    Oct 31, 2016 5:29:54 AM jenkins.InitReactorRunner$1 onAttained
    INFO: Prepared all plugins
    Oct 31, 2016 5:29:54 AM jenkins.InitReactorRunner$1 onAttained
    INFO: Started all plugins
    Oct 31, 2016 5:29:56 AM io.fabric8.jenkins.openshiftsync.GlobalPluginConfiguration configChange
    SEVERE: Failed to configure OpenShift Jenkins Sync Plugin: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://kubernetes.default.svc/oapi/v1/namespaces/extended-test-jenkins-kube-a67wm-tv3ra/buildconfigs. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked..
    Oct 31, 2016 5:29:56 AM jenkins.InitReactorRunner$1 onAttained
    INFO: Augmented all extensions
    Oct 31, 2016 5:29:57 AM jenkins.InitReactorRunner$1 onAttained
    INFO: Loaded all jobs
    Oct 31, 2016 5:29:57 AM hudson.model.AsyncPeriodicWork$1 run
    INFO: Started Download metadata
    Oct 31, 2016 5:29:58 AM org.jenkinsci.main.modules.sshd.SSHD start
    INFO: Started SSHD at port 40096
    Oct 31, 2016 5:29:58 AM jenkins.InitReactorRunner$1 onAttained
    INFO: Completed initialization
    Oct 31, 2016 5:29:58 AM org.springframework.context.support.AbstractApplicationContext prepareRefresh
    INFO: Refreshing org.springframework.web.context.support.StaticWebApplicationContext@44a6b6f7: display name [Root WebApplicationContext]; startup date [Mon Oct 31 05:29:58 UTC 2016]; root of context hierarchy
    Oct 31, 2016 5:29:58 AM org.springframework.context.support.AbstractApplicationContext obtainFreshBeanFactory
    INFO: Bean factory for application context [org.springframework.web.context.support.StaticWebApplicationContext@44a6b6f7]: org.springframework.beans.factory.support.DefaultListableBeanFactory@3675bac6
    Oct 31, 2016 5:29:58 AM org.springframework.beans.factory.support.DefaultListableBeanFactory preInstantiateSingletons
    INFO: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@3675bac6: defining beans [authenticationManager]; root of factory hierarchy
    Oct 31, 2016 5:29:59 AM org.springframework.context.support.AbstractApplicationContext prepareRefresh
    INFO: Refreshing org.springframework.web.context.support.StaticWebApplicationContext@247edd9c: display name [Root WebApplicationContext]; startup date [Mon Oct 31 05:29:59 UTC 2016]; root of context hierarchy
    Oct 31, 2016 5:29:59 AM org.springframework.context.support.AbstractApplicationContext obtainFreshBeanFactory
    INFO: Bean factory for application context [org.springframework.web.context.support.StaticWebApplicationContext@247edd9c]: org.springframework.beans.factory.support.DefaultListableBeanFactory@628b2e73
    Oct 31, 2016 5:29:59 AM org.springframework.beans.factory.support.DefaultListableBeanFactory preInstantiateSingletons
    INFO: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@628b2e73: defining beans [filter,legacy]; root of factory hierarchy
    Oct 31, 2016 5:30:00 AM org.openshift.jenkins.plugins.openshiftlogin.OpenShiftItemListener onLoaded
    INFO: OpenShift OAuth: enable oauth set to null
    Oct 31, 2016 5:30:00 AM hudson.WebAppMain$3 run
    INFO: Jenkins is fully up and running
    Oct 31, 2016 5:30:00 AM hudson.model.UpdateSite updateData
    INFO: Obtained the latest update center data file for UpdateSource default
    Oct 31, 2016 5:30:00 AM hudson.model.DownloadService$Downloadable load
    INFO: Obtained the updated data file for hudson.tasks.Maven.MavenInstaller
    Oct 31, 2016 5:30:01 AM hudson.model.DownloadService$Downloadable load
    INFO: Obtained the updated data file for hudson.tools.JDKInstaller
    Oct 31, 2016 5:30:01 AM hudson.model.AsyncPeriodicWork$1 run
    INFO: Finished Download metadata. 3,606 ms
to contain substring
    <string>: Adding image ruby-22-centos7-jenkins-slave:latest as Kubernetes slave

/data/src/github.com/openshift/origin/test/extended/image_ecosystem/kubernetes_plugin.go:153
bparees commented 7 years ago

that would imply the pod is running using the default service account, no?

jupierce commented 7 years ago

@bparees That's my understanding.

Looking at kubernetes_plugin.go, the template it instantiates doesn't even setup a "jenkins" service account. The test setup then incorrectly gives additional permissions to the "jenkins" account instead of "default". It seems to follow then that the test has not been running prior to Oct 29th or something has changed about the default SA permissions.

As for issue (1), the template it (should have) used specifies the "jenkins" service account unless it took this code path which should generally only execute on a development system: https://github.com/openshift/origin/blob/master/test/extended/image_ecosystem/plugin.go#L421

Several threads to follow -- continuing to investigate.

bparees commented 7 years ago

check with @liggitt or @deads2k about what might have changed in the default service account permissions.

liggitt commented 7 years ago

nothing has changed in the default service account permissions that I'm aware of

jupierce commented 7 years ago
  1. The Jenkins test results (https://github.com/openshift/origin/blob/master/test/extended/image_ecosystem/plugin.go#L421) do indicate the kubernetes_plugin.go test had been regularly skipped.
  2. @gabemontero 's failure looks to be explained by the conditional branch https://github.com/openshift/origin/blob/master/test/extended/image_ecosystem/plugin.go#L419 . In my development environment and in the extended test environment, the canonical ephemeral template is instantiated -- which specifies the service account. In Gabe's environment, I think it is safe to assume the PR testing template is instantiated -- which does not specify the service account.

In other words, these observations are independent. Closing the issue since the causes must be addressed in origin. Thanks @bparees @liggitt .

jupierce commented 7 years ago

See for followup: