User "system:serviceaccount:default:mockup" cannot list resource "services" in API group "" in the namespace "default"

[rcbandit111]

I'm trying to deploy a sample project with implementation 'org.springframework.cloud:spring-cloud-kubernetes-client-discovery:2.1.9'. Looks like I need to set some permissions. I get this error stack:

[2024-03-31 20:40:54.759  INFO 1 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
2024-03-31 20:40:54.847  INFO 1 --- [           main] org.apache.catalina.core.StandardEngine  : Starting Servlet engine: [Apache Tomcat/9.0.75]
2024-03-31 20:40:56.149  INFO 1 --- [           main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
2024-03-31 20:40:56.149  INFO 1 --- [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 34900 ms
2024-03-31 20:41:20.248  INFO 1 --- [oller-V1Service] i.k.client.informer.cache.Controller     : informer#Controller: ready to run resync & reflector runnable
2024-03-31 20:41:20.249  INFO 1 --- [oller-V1Service] i.k.client.informer.cache.Controller     : informer#Controller: resync skipped due to 0 full resync period
2024-03-31 20:41:20.249  INFO 1 --- [ler-V1Endpoints] i.k.client.informer.cache.Controller     : informer#Controller: ready to run resync & reflector runnable
2024-03-31 20:41:20.251  INFO 1 --- [ler-V1Endpoints] i.k.client.informer.cache.Controller     : informer#Controller: resync skipped due to 0 full resync period
2024-03-31 20:41:20.254  INFO 1 --- [s.V1Endpoints-1] i.k.c.informer.cache.ReflectorRunnable   : class io.kubernetes.client.openapi.models.V1Endpoints#Start listing and watching...
2024-03-31 20:41:20.254  INFO 1 --- [els.V1Service-1] i.k.c.informer.cache.ReflectorRunnable   : class io.kubernetes.client.openapi.models.V1Service#Start listing and watching...
2024-03-31 20:41:21.052  INFO 1 --- [pool-9-thread-1] .k.c.d.KubernetesInformerDiscoveryClient : Waiting for the cache of informers to be fully loaded..
2024-03-31 20:41:21.956  INFO 1 --- [pool-9-thread-1] .k.c.d.KubernetesInformerDiscoveryClient : Waiting for the cache of informers to be fully loaded..
2024-03-31 20:41:23.046  INFO 1 --- [pool-9-thread-1] .k.c.d.KubernetesInformerDiscoveryClient : Waiting for the cache of informers to be fully loaded..
2024-03-31 20:41:23.956  INFO 1 --- [pool-9-thread-1] .k.c.d.KubernetesInformerDiscoveryClient : Waiting for the cache of informers to be fully loaded..
2024-03-31 20:41:24.956  INFO 1 --- [pool-9-thread-1] .k.c.d.KubernetesInformerDiscoveryClient : Waiting for the cache of informers to be fully loaded..
2024-03-31 20:41:25.956  INFO 1 --- [pool-9-thread-1] .k.c.d.KubernetesInformerDiscoveryClient : Waiting for the cache of informers to be fully loaded..
2024-03-31 20:41:26.956  INFO 1 --- [pool-9-thread-1] .k.c.d.KubernetesInformerDiscoveryClient : Waiting for the cache of informers to be fully loaded..
2024-03-31 20:41:28.052  INFO 1 --- [pool-9-thread-1] .k.c.d.KubernetesInformerDiscoveryClient : Waiting for the cache of informers to be fully loaded..
2024-03-31 20:41:28.946 ERROR 1 --- [s.V1Endpoints-1] i.k.c.informer.cache.ReflectorRunnable   : class io.kubernetes.client.openapi.models.V1Endpoints#Reflector loop failed unexpectedly

io.kubernetes.client.openapi.ApiException: class V1Status {
    apiVersion: v1
    code: 403
    details: class V1StatusDetails {
        causes: null
        group: null
        kind: endpoints
        name: null
        retryAfterSeconds: null
        uid: null
    }
    kind: Status
    message: endpoints is forbidden: User "system:serviceaccount:default:mockup" cannot list resource "endpoints" in API group "" in the namespace "default"
    metadata: class V1ListMeta {
        _continue: null
        remainingItemCount: null
        resourceVersion: null
        selfLink: null
    }
    reason: Forbidden
    status: Failure
}
        at io.kubernetes.client.util.generic.KubernetesApiResponse.lambda$throwsApiException$0(KubernetesApiResponse.java:64) ~[client-java-13.0.2.jar!/:na]
        at io.kubernetes.client.util.generic.KubernetesApiResponse.onFailure(KubernetesApiResponse.java:78) ~[client-java-13.0.2.jar!/:na]
        at io.kubernetes.client.util.generic.KubernetesApiResponse.throwsApiException(KubernetesApiResponse.java:62) ~[client-java-13.0.2.jar!/:na]
        at io.kubernetes.client.informer.SharedInformerFactory$2.list(SharedInformerFactory.java:254) ~[client-java-13.0.2.jar!/:na]
        at io.kubernetes.client.informer.cache.ReflectorRunnable.run(ReflectorRunnable.java:88) ~[client-java-13.0.2.jar!/:na]
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[na:na]
        at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358) ~[na:na]
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[na:na]
        at java.base/java.lang.Thread.run(Thread.java:1583) ~[na:na]

2024-03-31 20:41:28.956  INFO 1 --- [pool-9-thread-1] .k.c.d.KubernetesInformerDiscoveryClient : Waiting for the cache of informers to be fully loaded..
2024-03-31 20:41:29.046 ERROR 1 --- [els.V1Service-1] i.k.c.informer.cache.ReflectorRunnable   : class io.kubernetes.client.openapi.models.V1Service#Reflector loop failed unexpectedly

io.kubernetes.client.openapi.ApiException: class V1Status {
    apiVersion: v1
    code: 403
    details: class V1StatusDetails {
        causes: null
        group: null
        kind: services
        name: null
        retryAfterSeconds: null
        uid: null
    }
    kind: Status
    message: services is forbidden: User "system:serviceaccount:default:mockup" cannot list resource "services" in API group "" in the namespace "default"
    metadata: class V1ListMeta {
        _continue: null
        remainingItemCount: null
        resourceVersion: null
        selfLink: null
    }
    reason: Forbidden
    status: Failure
}
        at io.kubernetes.client.util.generic.KubernetesApiResponse.lambda$throwsApiException$0(KubernetesApiResponse.java:64) ~[client-java-13.0.2.jar!/:na]
        at io.kubernetes.client.util.generic.KubernetesApiResponse.onFailure(KubernetesApiResponse.java:78) ~[client-java-13.0.2.jar!/:na]
        at io.kubernetes.client.util.generic.KubernetesApiResponse.throwsApiException(KubernetesApiResponse.java:62) ~[client-java-13.0.2.jar!/:na]
        at io.kubernetes.client.informer.SharedInformerFactory$2.list(SharedInformerFactory.java:254) ~[client-java-13.0.2.jar!/:na]
        at io.kubernetes.client.informer.cache.ReflectorRunnable.run(ReflectorRunnable.java:88) ~[client-java-13.0.2.jar!/:na]
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[na:na]
        at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358) ~[na:na]
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[na:na]
        at java.base/java.lang.Thread.run(Thread.java:1583) ~[na:na]

2024-03-31 20:41:29.948  INFO 1 --- [s.V1Endpoints-1] i.k.c.informer.cache.ReflectorRunnable   : class io.kubernetes.client.openapi.models.V1Endpoints#Start listing and watching...
2024-03-31 20:41:29.954 ERROR 1 --- [s.V1Endpoints-1] i.k.c.informer.cache.ReflectorRunnable   : class io.kubernetes.client.openapi.models.V1Endpoints#Reflector loop failed unexpectedly

io.kubernetes.client.openapi.ApiException: class V1Status {
    apiVersion: v1
    code: 403
    details: class V1StatusDetails {
        causes: null
        group: null
        kind: endpoints
        name: null
        retryAfterSeconds: null
        uid: null
    }
    kind: Status
    message: endpoints is forbidden: User "system:serviceaccount:default:mockup" cannot list resource "endpoints" in API group "" in the namespace "default"
    metadata: class V1ListMeta {
        _continue: null
        remainingItemCount: null
        resourceVersion: null
        selfLink: null
    }
    reason: Forbidden
    status: Failure
}
        at io.kubernetes.client.util.generic.KubernetesApiResponse.lambda$throwsApiException$0(KubernetesApiResponse.java:64) ~[client-java-13.0.2.jar!/:na]
        at io.kubernetes.client.util.generic.KubernetesApiResponse.onFailure(KubernetesApiResponse.java:78) ~[client-java-13.0.2.jar!/:na]
        at io.kubernetes.client.util.generic.KubernetesApiResponse.throwsApiException(KubernetesApiResponse.java:62) ~[client-java-13.0.2.jar!/:na]
        at io.kubernetes.client.informer.SharedInformerFactory$2.list(SharedInformerFactory.java:254) ~[client-java-13.0.2.jar!/:na]
        at io.kubernetes.client.informer.cache.ReflectorRunnable.run(ReflectorRunnable.java:88) ~[client-java-13.0.2.jar!/:na]
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[na:na]
        at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358) ~[na:na]
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[na:na]
        at java.base/java.lang.Thread.run(Thread.java:1583) ~[na:na]

2024-03-31 20:41:30.046  INFO 1 --- [pool-9-thread-1] .k.c.d.KubernetesInformerDiscoveryClient : Waiting for the cache of informers to be fully loaded..
2024-03-31 20:41:30.046  INFO 1 --- [els.V1Service-1] i.k.c.informer.cache.ReflectorRunnable   : class io.kubernetes.client.openapi.models.V1Service#Start listing and watching...
2024-03-31 20:41:30.152 ERROR 1 --- [els.V1Service-1] i.k.c.informer.cache.ReflectorRunnable   : class io.kubernetes.client.openapi.models.V1Service#Reflector loop failed unexpectedly

io.kubernetes.client.openapi.ApiException: class V1Status {
    apiVersion: v1
    code: 403
    details: class V1StatusDetails {
        causes: null
        group: null
        kind: services
        name: null
        retryAfterSeconds: null
        uid: null
    }
    kind: Status
    message: services is forbidden: User "system:serviceaccount:default:mockup" cannot list resource "services" in API group "" in the namespace "default"
    metadata: class V1ListMeta {
        _continue: null
        remainingItemCount: null
        resourceVersion: null
        selfLink: null
    }
    reason: Forbidden
    status: Failure
}
        at io.kubernetes.client.util.generic.KubernetesApiResponse.lambda$throwsApiException$0(KubernetesApiResponse.java:64) ~[client-java-13.0.2.jar!/:na]
        at io.kubernetes.client.util.generic.KubernetesApiResponse.onFailure(KubernetesApiResponse.java:78) ~[client-java-13.0.2.jar!/:na]
        at io.kubernetes.client.util.generic.KubernetesApiResponse.throwsApiException(KubernetesApiResponse.java:62) ~[client-java-13.0.2.jar!/:na]
        at io.kubernetes.client.informer.SharedInformerFactory$2.list(SharedInformerFactory.java:254) ~[client-java-13.0.2.jar!/:na]
        at io.kubernetes.client.informer.cache.ReflectorRunnable.run(ReflectorRunnable.java:88) ~[client-java-13.0.2.jar!/:na]
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[na:na]
        at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358) ~[na:na]
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[na:na]
        at java.base/java.lang.Thread.run(Thread.java:1583) ~[na:na]

root@node1:~#](url)

Do you know how I have to set permissions?

[rcbandit111]

@wind57 Can you advice please?

[wind57]

The usual warning to begin with: org.springframework.cloud:spring-cloud-kubernetes-client-discovery:2.1.9 is a version we do not support anymore, use it at your own risk :)

Yes, that error you are seeing denotes missing or incorrect RBAC rules, we have some samples in the integration test, see here. You will need to read on those a little and how those are set-up in your cluster.

Our integration tests are kind of "samples" where you can take a look how to set that up.

[rcbandit111]

@wind57 I tried to do this using HELM chart.

https://github.com/rcbandit111/mockup/blob/master/helm_chart/mockup-chart/templates/role.yml

But unfortunately it's not working. If you have experience with Helm charts can you advice how this can be implemented properly?

[wind57]

this is not related to helm per-se, helm charts are a way to package many manifests and deploy them. While my explanation is simplified, the main idea is that your problem has nothing to do with helm charts. You need to read what RBAC is (and may be what kubectl can-i is, that will help you).

I have already pointed you to the manifests that we use in integration tests above, those do work, otherwise the integration tests would have failed.

And "it's not working" is not something we can debug or understand. If you have a problem with those manifests we use in integration tests, then create a minimal reproducible example, so that we can reproduce the problem. Whatever issue you have with helm, is not under our jurisdiction, because we do not provide any helm charts to begin with.

[wind57]

@ryanjbaxter we can probably add the feedback tag in here also

[rcbandit111]

I found the issue into the helm chart. I can deploy and stat the project using org.springframework.cloud:spring-cloud-kubernetes-client-discovery:2.1.9.

[wind57]

Good :)

So you can close this issue then?