vmware-archive / octant

Highly extensible platform for developers to better understand the complexity of Kubernetes clusters.
https://octant.dev
Apache License 2.0
6.28k stars 486 forks source link

Unable to get lister for /v1, Resource=pods #2570

Open nipuna-perera opened 3 years ago

nipuna-perera commented 3 years ago

What steps did you take and what happened: Updated the 0.21 from 0.19 and I am unable to see basic resources such as pods or deployments anymore. I do have access to these resources because I can use kubectl to access them.

ran octant cli and checked the browser, no resources are showing after I select namespace.

2021-06-18T15:35:50.243-0500    ERROR   api/content_manager.go:158  generate content    {"client-id": "f18ce448-d073-11eb-b482-58ef68e6c31b", "err": "generate content: unable to get Lister for /v1, Resource=pods, watcher was unable to start", "content-path": "overview/namespace/default/workloads/pods"}

The console log outputs similar messages for other resources as well and I see the following then I try to open the namespace overview. I can list namespaces however.

r
print summary tab: failed to create item view: unable to get Lister for /v1, Resource=limitranges, watcher was unable to start

What did you expect to happen: Deployments/pods and any other resources I have access to should be visible in Octant.

Anything else you would like to add: Here's some more context from the debug log

2021-06-18T15:46:21.697-0500    DEBUG   api/poller.go:84    poller is running action    {"component": "websocket-client", "client-id": "314dd248-d076-11eb-8804-58ef68e6c31b", "poller-name": "content", "poller-instance": "f2dac8ac-af6d-456e-ba1a-0e663c8577e5"}
2021-06-18T15:46:21.697-0500    DEBUG   api/poller.go:88    poller ran action   {"component": "websocket-client", "client-id": "314dd248-d076-11eb-8804-58ef68e6c31b", "poller-name": "helperManager", "poller-instance": "0a35d92e-6506-4dd3-b37d-c6e5b5ca31e0", "elapsed": "3.889µs"}
2021-06-18T15:46:21.697-0500    DEBUG   api/poller.go:88    poller ran action   {"component": "websocket-client", "client-id": "314dd248-d076-11eb-8804-58ef68e6c31b", "poller-name": "content", "poller-instance": "f2dac8ac-af6d-456e-ba1a-0e663c8577e5", "elapsed": "829ns"}
2021-06-18T15:46:22.124-0500    DEBUG   api/poller.go:84    poller is running action    {"component": "websocket-client", "client-id": "312b2d42-d076-11eb-8804-58ef68e6c31b", "poller-name": "content", "poller-instance": "d77159a2-cd89-4dce-9bc1-94c9e27a4f18"}
2021-06-18T15:46:22.125-0500    DEBUG   api/content_manager.go:185  generating content  {"client-id": "312b2d42-d076-11eb-8804-58ef68e6c31b", "contentPath": "overview/namespace/emg-pi-preprod/workloads/pods", "elapsed": "149.042µs"}
2021-06-18T15:46:22.125-0500    ERROR   api/content_manager.go:158  generate content    {"client-id": "312b2d42-d076-11eb-8804-58ef68e6c31b", "err": "generate content: unable to get Lister for /v1, Resource=pods, watcher was unable to start", "content-path": "overview/namespace/emg-pi-preprod/workloads/pods"}
github.com/vmware-tanzu/octant/internal/api.(*ContentManager).runUpdate.func1
    /private/tmp/octant-20210617-71377-4p5v5a/src/github.com/vmware-tanzu/octant/internal/api/content_manager.go:158
github.com/vmware-tanzu/octant/internal/api.(*InterruptiblePoller).Run.func1
    /private/tmp/octant-20210617-71377-4p5v5a/src/github.com/vmware-tanzu/octant/internal/api/poller.go:86
github.com/vmware-tanzu/octant/internal/api.(*InterruptiblePoller).Run
    /private/tmp/octant-20210617-71377-4p5v5a/src/github.com/vmware-tanzu/octant/internal/api/poller.go:95
github.com/vmware-tanzu/octant/internal/api.(*ContentManager).Start
    /private/tmp/octant-20210617-71377-4p5v5a/src/github.com/vmware-tanzu/octant/internal/api/content_manager.go:132
2021-06-18T15:46:22.125-0500    DEBUG   api/poller.go:88    poller ran action   {"component": "websocket-client", "client-id": "312b2d42-d076-11eb-8804-58ef68e6c31b", "poller-name": "content", "poller-instance": "d77159a2-cd89-4dce-9bc1-94c9e27a4f18", "elapsed": "342.174µs"}
2021-06-18T15:46:22.260-0500    DEBUG   api/poller.go:84    poller is running action    {"component": "websocket-client", "client-id": "312b2d42-d076-11eb-8804-58ef68e6c31b", "poller-name": "navigation", "poller-instance": "de8fff9b-7e6a-4d95-91c5-ceec27f7d8b0"}
2021-06-18T15:46:22.279-0500    DEBUG   api/poller.go:84    poller is running action    {"component": "websocket-client", "client-id": "314dd248-d076-11eb-8804-58ef68e6c31b", "poller-name": "navigation", "poller-instance": "9f1033bc-b988-4dad-8f52-70c304aec5a2"}
2021-06-18T15:46:22.281-0500    DEBUG   api/poller.go:88    poller ran action   {"component": "websocket-client", "client-id": "312b2d42-d076-11eb-8804-58ef68e6c31b", "poller-name": "navigation", "poller-instance": "de8fff9b-7e6a-4d95-91c5-ceec27f7d8b0", "elapsed": "20.498833ms"}
2021-06-18T15:46:22.296-0500    DEBUG   api/poller.go:84    poller is running action    {"component": "websocket-client", "client-id": "326297a4-d076-11eb-8804-58ef68e6c31b", "poller-name": "navigation", "poller-instance": "a12edad9-ea5c-4149-969a-36431333b5c4"}
2021-06-18T15:46:22.298-0500    DEBUG   api/poller.go:88    poller ran action   {"component": "websocket-client", "client-id": "314dd248-d076-11eb-8804-58ef68e6c31b", "poller-name": "navigation", "poller-instance": "9f1033bc-b988-4dad-8f52-70c304aec5a2", "elapsed": "19.726929ms"}
2021-06-18T15:46:22.316-0500    DEBUG   api/poller.go:88    poller ran action   {"component": "websocket-client", "client-id": "326297a4-d076-11eb-8804-58ef68e6c31b", "poller-name": "navigation", "poller-instance": "a12edad9-ea5c-4149-969a-36431333b5c4", "elapsed": "19.460078ms"}
2021-06-18T15:46:22.320-0500    DEBUG   api/poller.go:84    poller is running action    {"component": "websocket-client", "client-id": "326297a4-d076-11eb-8804-58ef68e6c31b", "poller-name": "namespaces", "poller-instance": "cfb5611a-030a-4ef6-b617-a8dea4e4e0a6"}
2021-06-18T15:46:22.365-0500    DEBUG   api/poller.go:88    poller ran action   {"component": "websocket-client", "client-id": "326297a4-d076-11eb-8804-58ef68e6c31b", "poller-name": "namespaces", "poller-instance": "cfb5611a-030a-4ef6-b617-a8dea4e4e0a6", "elapsed": "45.323432ms"}
2021-06-18T15:46:22.368-0500    DEBUG   api/poller.go:84    poller is running action    {"component": "websocket-client", "client-id": "312b2d42-d076-11eb-8804-58ef68e6c31b", "poller-name": "namespaces", "poller-instance": "230391db-366b-4600-a3a5-b46c88a0dd56"}

Environment:

wwitzel3 commented 3 years ago

We recently changed the object store for Octant to require the minimum of being able to Watch a resource.

That said, we could re-introduce the ability to query the cluster directly with List/Get and not have a cache in the middle at all, this would result in a significant slow down, but would allow you to list resources again.

nipuna-perera commented 3 years ago

Yes, maybe you could make that an option? In the current state, with the restrictions my company has put on resources, I am unable to see anything. It was working great for my purposes before, even though it was a bit slow.

wwitzel3 commented 3 years ago

Ok, for this we will re-introduce the ability to start Octant using a DynamicClient directly, which will not be backed by the cache, this will allow Octant to display resources is more restricted environment that don't support Watch at the cost of speed.

For who ever takes this issue to work on it we will want a couple things:

In the above, I think it is important to let users explicitly know they are in direct client mode via some icon indicator somewhere and provide a brief tip/explanation about what it means and the speed implications.

nipuna-perera commented 3 years ago

@wwitzel3 thank you! Can the poll speed be customizable in direct client mode? If someone wanted to experiment with a 3 second poll time they should be able to do so.

ojagodzinski commented 3 years ago

We recently changed the object store for Octant to require the minimum of being able to Watch a resource.

Is there any example of working read-only role (except secrets) for Octant? I got clusterrole with watch capability and still does not work.

jpreese commented 3 years ago

Wanted to ask the same. Our users only have access to a single namespace, but Octant doesn't show any resources at all. (0.21.0)

wwitzel3 commented 3 years ago

We recently changed the object store for Octant to require the minimum of being able to Watch a resource.

Is there any example of working read-only role (except secrets) for Octant? I got clusterrole with watch capability and still does not work.

This should work if you have a clusterrole with Watch, what error are you seeing in the console output?

wwitzel3 commented 3 years ago

Wanted to ask the same. Our users only have access to a single namespace, but Octant doesn't show any resources at all. (0.21.0)

Are you able to start Octant with the --namespace flag and provide the namespace the user has access to?

jpreese commented 3 years ago

Wanted to ask the same. Our users only have access to a single namespace, but Octant doesn't show any resources at all. (0.21.0)

Are you able to start Octant with the --namespace flag and provide the namespace the user has access to?

Yes. I have tried --namespace and --disable-cluster-overview

wwitzel3 commented 3 years ago

@jpreese and @nipuna-perera I'm curious, if you run the following kubectl commands:

Check that you can watch for the namespace: kubectl auth can-i watch pods --namespace <your-namespace>

Now watch, then create a throw-away pod, or scale up a deployment, do you see the pods populate in output?

kubectl get pods --namespace <your-namespace> --watch-only

jpreese commented 3 years ago

Yep! All the above outputs as expected. We have admin access over the namespace so I don't think it's a permission issue there -- just lack permissions to do much outside of the namespace.

ojagodzinski commented 2 years ago

Same here, when service account have only access to get,list,watch on everything excepts secrets on every namespace listing anything works only on default namespace. It was working fine one version 0.16.3.

example rbac: https://github.com/kubernetes/kubernetes/issues/70387#issuecomment-434288599

KUBECONFIG=/path/kubeconfig kubectl auth can-i watch Deployment --namespace some-name
yes
KUBECONFIG=/path/kubeconfig kubectl auth can-i list Deployment --namespace some-name
yes
KUBECONFIG=/path/kubeconfig kubectl auth can-i get Deployment --namespace some-name
yes

but:

2021-11-22T12:15:54.607+0100    ERROR   describer/describer.go:95   LoadObjects {"err": "List: CacheKey[Namespace='some-name', APIVersion='apps/v1', Kind='Deployment'] (error: unable to get Lister for /, Resource=, watcher was unable to start)"}
github.com/vmware-tanzu/octant/internal/describer.LoadObjects
    /__w/octant/octant/internal/describer/describer.go:95
github.com/vmware-tanzu/octant/internal/describer.(*ObjectLoaderFactory).LoadObjects
    /__w/octant/octant/internal/describer/describer.go:43
github.com/vmware-tanzu/octant/internal/describer.(*List).Describe
    /__w/octant/octant/internal/describer/list.go:68
github.com/vmware-tanzu/octant/internal/generator.(*Generator).Generate
    /__w/octant/octant/internal/generator/generator.go:121
github.com/vmware-tanzu/octant/internal/modules/overview.(*Overview).Content
    /__w/octant/octant/internal/modules/overview/overview.go:288
github.com/vmware-tanzu/octant/internal/api.(*ContentManager).generateContent
    /__w/octant/octant/internal/api/content_manager.go:210
github.com/vmware-tanzu/octant/internal/api.(*ContentManager).runUpdate.func1
    /__w/octant/octant/internal/api/content_manager.go:145
github.com/vmware-tanzu/octant/internal/api.(*InterruptiblePoller).Run.func1
    /__w/octant/octant/internal/api/poller.go:86
github.com/vmware-tanzu/octant/internal/api.(*InterruptiblePoller).Run
    /__w/octant/octant/internal/api/poller.go:95
github.com/vmware-tanzu/octant/internal/api.(*ContentManager).Start
    /__w/octant/octant/internal/api/content_manager.go:133