search

fabric8io / jenkins-docker

docker file for a jenkins docker image
84 stars 96 forks source link

Forbidden!Configured service account doesn't have access. Service account may have been revoked.. #114

Open magick93 opened 7 years ago

magick93 commented 7 years ago

When trying to run a Jenkins job we are now getting:

SEVERE: Error in provisioning; slave=KubernetesSlave name: kubernetes-06843467478e4da2aa3d1dfe96079934-196809805b424, template=org.csanchez.jenkins.plugins.kubernetes.PodTemplate@5eadd457
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://kubernetes.default/api/v1/namespaces/default/pods. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked..
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:315)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:266)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:237)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:230)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:208)
    at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:643)
    at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:300)
    at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback.call(KubernetesCloud.java:557)
    at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback.call(KubernetesCloud.java:532)
    at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

Its very likely that this is a result of a change we have made. But we have documented most, if not all changes, and reviewed the Jenkins scc, and cannot see anything obvious.

Jenkins SCC

$ oc describe scc jenkins
Name:                       jenkins
Priority:                   99
Access:                     
  Users:                    system:serviceaccount:default:admin,system:serviceaccount:default:jenkins,admin
  Groups:                   system:cluster-admins,system:nodes,system:serviceaccounts:default,root,jenkins
Settings:                   
  Allow Privileged:             true
  Default Add Capabilities:         <none>
  Required Drop Capabilities:           <none>
  Allowed Capabilities:             <none>
  Allowed Volume Types:             *
  Allow Host Network:               true
  Allow Host Ports:             true
  Allow Host PID:               false
  Allow Host IPC:               false
  Read Only Root Filesystem:            false
  Run As User Strategy: RunAsAny        
    UID:                    <none>
    UID Range Min:              <none>
    UID Range Max:              <none>
  SELinux Context Strategy: RunAsAny        
    User:                   <none>
    Role:                   <none>
    Type:                   <none>
    Level:                  <none>
  FSGroup Strategy: RunAsAny            
    Ranges:                 <none>
  Supplemental Groups Strategy: RunAsAny    
    Ranges:                 <none>
magick93 commented 7 years ago

Attempt

oadm policy add-role-to-user view system:serviceaccount:default:jenkins oadm policy add-role-to-user edit system:serviceaccount:default:jenkins

Result

Same error

iocanel commented 7 years ago

It seems that the error occurs during jenkins attempt to create a slave pod.

what's the output of oc oadm policy who-can create pod ?

On Mon, Feb 13, 2017 at 5:48 PM, magick93 notifications@github.com wrote:

Attempt

oadm policy add-role-to-user view system:serviceaccount:default:jenkins Result

Same error

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/fabric8io/jenkins-docker/issues/114#issuecomment-279430855, or mute the thread https://github.com/notifications/unsubscribe-auth/AAYiWLYcszjYosDjhiLoCBpGupcLzD09ks5rcHs_gaJpZM4L_Uzh .

-- Ioannis Canellos

Blog: http://iocanel.blogspot.com http://iocanel.blogspot.com/ Twitter: iocanel

rawlingsj commented 7 years ago

It's also worth checking if it's this issue https://issues.jenkins-ci.org/browse/JENKINS-41388

TD;DR check the jenkins configuration page and make sure the kubernetes plugin namespace field has the value that matches the namespace jenkins is running in.

magick93 commented 7 years ago 
# oadm policy who-can create pod
Namespace: default
Verb:      create
Resource:  pods

Users:  admin
        developer
        system:admin
        system:serviceaccount:default:configmapcontroller
        system:serviceaccount:default:deployer
        system:serviceaccount:default:exposecontroller
        system:serviceaccount:default:fabric8
        system:serviceaccount:default:jenkins
        system:serviceaccount:jenkins:default
        system:serviceaccount:jenkins:jenkins
        system:serviceaccount:openshift-infra:build-controller
        system:serviceaccount:openshift-infra:daemonset-controller
        system:serviceaccount:openshift-infra:deploymentconfig-controller
        system:serviceaccount:openshift-infra:job-controller
        system:serviceaccount:openshift-infra:pet-set-controller
        system:serviceaccount:openshift-infra:pv-binder-controller
        system:serviceaccount:openshift-infra:pv-recycler-controller
        system:serviceaccount:openshift-infra:replicaset-controller
        system:serviceaccount:openshift-infra:replication-controller
        system:serviceaccount:test-docker-push:configmapcontroller
        system:serviceaccount:test-docker-push:exposecontroller
        system:serviceaccount:test-docker-push:fabric8
        system:serviceaccount:test-docker-push:jenkins
        system:serviceaccount:test1:configmapcontroller
        system:serviceaccount:test1:exposecontroller
        system:serviceaccount:test1:fabric8
        system:serviceaccount:test1:jenkins
        system:serviceaccount:test:configmapcontroller
        system:serviceaccount:test:exposecontroller
        system:serviceaccount:test:fabric8
        system:serviceaccount:test:jenkins

Groups: system:cluster-admins
        system:masters
        system:nodes
magick93 commented 7 years ago

check the jenkins configuration page and make sure the kubernetes plugin namespace field has the value that matches the namespace jenkins is running in.

selection_670 Unless I am mistaken, it is correct. Jenkins is the default namespace

rawlingsj commented 7 years ago

Ok well that rules my theory out then - back to @iocanel line of thought

magick93 commented 7 years ago

More info

The error is:

Feb 13, 2017 3:59:19 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
SEVERE: Failed to terminate pod for slave kubernetes-b2102a9dbce645bab1bd6838a748e2a1-198b7ea6eb33b
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: DELETE at: https://kubernetes.default/api/v1/namespaces/default/pods/kubernetes-b2102a9dbce645bab1bd6838a748e2a1-198b7ea6eb33b. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked..
# oadm policy who-can delete  pod
Namespace: default
Verb:      delete
Resource:  pods

Users:  admin
        developer
        system:admin
        system:serviceaccount:default:configmapcontroller
        system:serviceaccount:default:exposecontroller
        system:serviceaccount:default:fabric8
        system:serviceaccount:default:jenkins
        system:serviceaccount:jenkins:default
        system:serviceaccount:jenkins:jenkins
        system:serviceaccount:openshift-infra:build-controller
        system:serviceaccount:openshift-infra:daemonset-controller
        system:serviceaccount:openshift-infra:deploymentconfig-controller
        system:serviceaccount:openshift-infra:gc-controller
        system:serviceaccount:openshift-infra:job-controller
        system:serviceaccount:openshift-infra:namespace-controller
        system:serviceaccount:openshift-infra:pet-set-controller
        system:serviceaccount:openshift-infra:pv-binder-controller
        system:serviceaccount:openshift-infra:pv-recycler-controller
        system:serviceaccount:openshift-infra:replicaset-controller
        system:serviceaccount:openshift-infra:replication-controller
        system:serviceaccount:test-docker-push:configmapcontroller
        system:serviceaccount:test-docker-push:exposecontroller
        system:serviceaccount:test-docker-push:fabric8
        system:serviceaccount:test-docker-push:jenkins
        system:serviceaccount:test1:configmapcontroller
        system:serviceaccount:test1:exposecontroller
        system:serviceaccount:test1:fabric8
        system:serviceaccount:test1:jenkins
        system:serviceaccount:test:configmapcontroller
        system:serviceaccount:test:exposecontroller
        system:serviceaccount:test:fabric8
        system:serviceaccount:test:jenkins

Groups: system:cluster-admins
        system:masters
        system:nodes

The lines system:serviceaccount:jenkins:default and system:serviceaccount:jenkins:jenkins look incorrect. Agree?

How do I remove this?

iocanel commented 7 years ago

It's incorrect, but it doesn't seem to cause any issues.

On Mon, Feb 13, 2017 at 6:12 PM, magick93 notifications@github.com wrote:

More info

The error is:

Feb 13, 2017 3:59:19 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate SEVERE: Failed to terminate pod for slave kubernetes-b2102a9dbce645bab1bd6838a748e2a1-198b7ea6eb33b io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: DELETE at: https://kubernetes.default/api/v1/namespaces/default/pods/kubernetes-b2102a9dbce645bab1bd6838a748e2a1-198b7ea6eb33b. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked..

oadm policy who-can delete pod

Namespace: default Verb: delete Resource: pods

Users: admin developer system:admin system:serviceaccount:default:configmapcontroller system:serviceaccount:default:exposecontroller system:serviceaccount:default:fabric8 system:serviceaccount:default:jenkins system:serviceaccount:jenkins:default system:serviceaccount:jenkins:jenkins system:serviceaccount:openshift-infra:build-controller system:serviceaccount:openshift-infra:daemonset-controller system:serviceaccount:openshift-infra:deploymentconfig-controller system:serviceaccount:openshift-infra:gc-controller system:serviceaccount:openshift-infra:job-controller system:serviceaccount:openshift-infra:namespace-controller system:serviceaccount:openshift-infra:pet-set-controller system:serviceaccount:openshift-infra:pv-binder-controller system:serviceaccount:openshift-infra:pv-recycler-controller system:serviceaccount:openshift-infra:replicaset-controller system:serviceaccount:openshift-infra:replication-controller system:serviceaccount:test-docker-push:configmapcontroller system:serviceaccount:test-docker-push:exposecontroller system:serviceaccount:test-docker-push:fabric8 system:serviceaccount:test-docker-push:jenkins system:serviceaccount:test1:configmapcontroller system:serviceaccount:test1:exposecontroller system:serviceaccount:test1:fabric8 system:serviceaccount:test1:jenkins system:serviceaccount:test:configmapcontroller system:serviceaccount:test:exposecontroller system:serviceaccount:test:fabric8 system:serviceaccount:test:jenkins

Groups: system:cluster-admins system:masters system:nodes

The line system:serviceaccount:jenkins:default look incorrect. Agree?

How do I remove this?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/fabric8io/jenkins-docker/issues/114#issuecomment-279438389, or mute the thread https://github.com/notifications/unsubscribe-auth/AAYiWMwHXkcdNfx-q5GJpeh6dp-PMLyrks5rcIDxgaJpZM4L_Uzh .

-- Ioannis Canellos

Blog: http://iocanel.blogspot.com http://iocanel.blogspot.com/ Twitter: iocanel

rawlingsj commented 7 years ago

You could try and bounce the jenkins master pod and see if a new secret is mounted for the service account?

magick93 commented 7 years ago

Yes, tried that. No joy.

magick93 commented 7 years ago

Just tried restarting the node, and a minor change, now the error is:

SEVERE: Error in provisioning; slave=KubernetesSlave name: kubernetes-66952ffdad934876868020fd596178df-53d6d4b9b7, template=org.csanchez.jenkins.plugins.kubernetes.PodTemplate@1badfb8b
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://kubernetes.default/api/v1/namespaces/default/pods. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked..
einyx commented 7 years ago

any luck with this?

moortimis commented 7 years ago

Hey, we're seeing the same issue. oadm policy who-can create pod returns the expected results, but still getting the following errors:


io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://100.73.0.1/api/v1/namespaces/ournamespace/pods. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked.```
moortimis commented 7 years ago

Actually found the root cause of our issue:

Solution:

GangChenTFS commented 5 years ago

@moortimis , how to find the limit-name and quota-name? could you please share the command thanks.