search

fabric8-services / fabric8-jenkins-idler

OpenShift.io service to idle resp.unidle Jenkins instances
Apache License 2.0
4 stars 15 forks source link

Idler not idling namespace after 1h because user namespace is missing #242

Closed jfchevrette closed 4 years ago

jfchevrette commented 6 years ago

Cluster: starter-us-east-2 Namespace: mspisiak-osio-jenkins

The user does not have a user namespace (mspisiak-osio) which may explain why the idler would be failing to check the builds.

I see the following stack track repeated every few minutes in the jenkins pod showing the jenkins can'T reach the non-existing namespace

May 25, 2018 8:27:06 PM io.fabric8.jenkins.openshiftsync.BuildConfigWatcher$1 doRun
SEVERE: Failed to load BuildConfigs: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://kubernetes.default/apis/build.openshift.io/v1/namespaces/mspisiak-osio/buildconfigs. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. buildconfigs.build.openshift.io is forbidden: User "system:serviceaccount:mspisiak-osio-jenkins:jenkins" cannot list buildconfigs.build.openshift.io in the namespace "mspisiak-osio": User "system:serviceaccount:mspisiak-osio-jenkins:jenkins" cannot list buildconfigs.build.openshift.io in project "mspisiak-osio".
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://kubernetes.default/apis/build.openshift.io/v1/namespaces/mspisiak-osio/buildconfigs. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. buildconfigs.build.openshift.io is forbidden: User "system:serviceaccount:mspisiak-osio-jenkins:jenkins" cannot list buildconfigs.build.openshift.io in the namespace "mspisiak-osio": User "system:serviceaccount:mspisiak-osio-jenkins:jenkins" cannot list buildconfigs.build.openshift.io in project "mspisiak-osio".
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:470)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:407)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:379)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:343)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:327)
    at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:605)
    at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:70)
    at io.fabric8.jenkins.openshiftsync.BuildConfigWatcher$1.doRun(BuildConfigWatcher.java:115)
    at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51)
    at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

May 25, 2018 8:27:06 PM io.fabric8.jenkins.openshiftsync.BuildWatcher$1 doRun
SEVERE: Failed to load initial Builds: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://kubernetes.default/apis/build.openshift.io/v1/namespaces/mspisiak-osio/builds. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. builds.build.openshift.io is forbidden: User "system:serviceaccount:mspisiak-osio-jenkins:jenkins" cannot list builds.build.openshift.io in the namespace "mspisiak-osio": User "system:serviceaccount:mspisiak-osio-jenkins:jenkins" cannot list builds.build.openshift.io in project "mspisiak-osio".
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://kubernetes.default/apis/build.openshift.io/v1/namespaces/mspisiak-osio/builds. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. builds.build.openshift.io is forbidden: User "system:serviceaccount:mspisiak-osio-jenkins:jenkins" cannot list builds.build.openshift.io in the namespace "mspisiak-osio": User "system:serviceaccount:mspisiak-osio-jenkins:jenkins" cannot list builds.build.openshift.io in project "mspisiak-osio".
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:470)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:407)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:379)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:343)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:327)
    at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:605)
    at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:70)
    at io.fabric8.jenkins.openshiftsync.BuildWatcher$1.doRun(BuildWatcher.java:92)
    at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51)
    at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
jfchevrette commented 6 years ago

The missing user namespace is a condition we have observed a few times and I don't believe we have a fix for that yet.

As a workaround, could we make it so that if the user namespace is missing and idler can't check the builds/buildconfigs, the idler will go ahead and idle jenkins, assuming the other conditions are met?

klopfdreh commented 5 years ago

Hi @jfchevrette,

any updates here? I am also facing this issue, but during the startup of the POD.

kind regards

Tobias