RBAC traffic manager installation not functioning as expected

[bzlom]

Describe the bug Trying to restrict telepresence intercept to specific k8s namespaces by following this guide https://www.telepresence.io/docs/latest/install/helm/. If I install telepresence without any RBAC settings (managerRbac.namespaces) everything works as expected: I'm able to telepresence connect and then telepresence intercept with no issues. When I add RBAC setting with -f values.yaml I'm unable to intercept any services.

Debug logs attached: telepresence_logs.zip

To Reproduce Steps to reproduce the behavior:

1. the helm chart installs successfully with the RBAC:

   helm install traffic-manager --namespace ambassador datawire/telepresence -f values.yaml

   where values.yaml looks like:

   managerRbac:
   create: true
   namespaced: true
   namespaces:
   - oos
   - op-intelligence
   - ambassador

2. I'm then able to run telepresence connect with no issues
3. When I run intercept I get timed out:

   ./telepresence intercept token-provider-v1 --namespace oos --port 8080:8080
   telepresence: error: rpc error: code = DeadlineExceeded desc = request timed out while waiting for agent token-provider-v1.oos to arrive
   Error: rpc error: code = DeadlineExceeded desc = request timed out while waiting for agent token-provider-v1.oos to arrive

Expected behavior Be able to run telepresence intercept with RBAC limits to specific k8s namespaces

Versions (please complete the following information):

• Output of telepresence version: Client: v2.7.0 (api v3) Root Daemon: v2.7.0 (api v3) User Daemon: v2.7.0 (api v3)
• Operating system of workstation running telepresence commands: Ubuntu 20.04.4
• Kubernetes environment and Version: AWS EKS

  kubectl version
  Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.10", GitCommit:"eae22ba6238096f5dec1ceb62766e97783f0ba2f", GitTreeState:"clean", BuildDate:"2022-05-24T12:56:35Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"22+", GitVersion:"v1.22.10-eks-84b4fe6", GitCommit:"cc6a1b4915a99f49f5510ef0667f94b9ca832a8a", GitTreeState:"clean", BuildDate:"2022-06-09T18:24:04Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}

Additional context k8s context file is present in $KUBECONFIG environment variable

[cindymullins-dw]

Hi @bzlom , we’ve released a new version to address this “Deadline exceeded’ error which may help here. Please upgrade to 2.7.6. Also, if you're using a private cluster you’ll need to adjust your firewall rules and we can provide more info on that if relevant.

[bzlom]

@cindymullins-dw hi, sorry for the very late reply. I've tried the same thing with telepresence version 2.12.0. I'm getting the same result but different errors this time around. It works fine when I don't apply any RBAC settings on telepresence side and I'm able to connect and intercept pods just fine. When I apply any RBAC policies on telepresence, same as in my original message I get errors like these:

# telepresence intercept some-random-pod --port 8080:8080 -n oos
telepresence intercept: error: connector.CreateIntercept: request timed out while waiting for agent user-management-client-api-v4.oos to arrive: Events that may be relevant:
AGE   TYPE      REASON                   OBJECT                                              MESSAGE
30s   Warning   FailedCreatePodSandBox   pod/user-management-client-api-v4-6684bf74d-mcv6n   Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "b7b29e195e0bb5913cbf626770fac5b012d7ac9b9d660c87015055af19a9d1d8": plugin type="aws-cni" name="aws-cni" failed (add): add cmd: failed to assign an IP address to container
18s   Warning   FailedCreatePodSandBox   pod/user-management-client-api-v4-6684bf74d-mcv6n   Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f5e6d490198b8c767118205998b2850568afd111b7d085e4cf761576000666f4": plugin type="aws-cni" name="aws-cni" failed (add): add cmd: failed to assign an IP address to container
7s    Warning   FailedCreatePodSandBox   pod/user-management-client-api-v4-6684bf74d-mcv6n   Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "39b296965e86c87155cbfd708b6764ae4f3de78dccc90b4a8e1d32bb9afc318d": plugin type="aws-cni" name="aws-cni" failed (add): add cmd: failed to assign an IP address to container

And after getting the error above (once or a few times) I start getting this one:

telepresence intercept some-random-pod --port 8080:8080 -n oos
telepresence intercept: error: connector.CreateIntercept: request timed out while waiting for agent user-management-client-api-v4.oos to arrive

[cindymullins-dw]

@bzlom , your config looks ok and we would expect this to work. Unfortunately, the error is not informative enough to help. If you can submit logs we'll look into those, and as always we do recommend upgrading to the latest version.

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment, or this will be closed in 7 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 7 days with no activity.