Remote endpoint discovery issue after enabling Ambient: ConfigMap access error

h0tbird commented 3 weeks ago

Is this the right place to submit this?

[x] This is not a security vulnerability or a crashing bug
[x] This is not a question about how to use Istio

Bug Description

In a multi-primary, single-network setup, I'm still using sidecars across the board but decided to enable the ambient profile for future testing. After enabling ambient (CNI, istiod, and ztunnel), I noticed that remote endpoints are no longer being discovered, with the following error appearing in the istiod logs:

{"level":"error","time":"2024-08-18T15:35:15.147793Z","msg":"watch error in cluster pasta-2: failed to list *v1.ConfigMap: configmaps is forbidden: User \"system:serviceaccount:istio-system:istio-reader-service-account\" cannot list resource \"configmaps\" in API group \"\" at the cluster scope"}

Updating the ClusterRole with the following patch resolved the issue for me:

k --context pasta-1 patch clusterrole istio-reader-clusterrole-1-23-0-istio-system --type='json' -p='[{"op": "add", "path": "/rules/1/resources/-", "value": "configmaps"}]'
k --context pasta-2 patch clusterrole istio-reader-clusterrole-1-23-0-istio-system --type='json' -p='[{"op": "add", "path": "/rules/1/resources/-", "value": "configmaps"}]'

Version

$ istioctl --context pasta-1 version
client version: 1.23.0
control plane version: 1.23.0
data plane version: 1.23.0 (6 proxies)
$ k --context pasta-1 version
Client Version: v1.31.0
Kustomize Version: v5.4.2
Server Version: v1.30.3+k3s1

Additional Information

No response

howardjohn commented 3 weeks ago

Thanks for the report. I guess there is a bit of ambiguity in multi-cluster not yet being support with ambient, wrt to whether that means "your ambient workloads won't do cross-cluster things" or "Istiod will blow up entirely if you have both enabled at once".

We should prioritize fixing the later first, and maybe improve some docs around this.

cc @keithmattix

fred-vogt-dod commented 3 weeks ago

ClusterRole istio-reader-clusterrole needed these additional permissions with Ambient interop mode enabled and multi-cluster endpoint discovery active:

  - apiGroups: ["gateway.networking.k8s.io"]
    resources: ["gatewayclasses"]
    verbs: ["get", "watch", "list"]
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["get", "list", "watch"]

howardjohn commented 3 days ago

I think this is just a symptom of https://github.com/istio/istio/issues/42137 not being done, so we can track there. Note ambient does not currently support multi-cluster. The changes to ClusterRole may superficial work but you are in "stuff is randomly happening that is unintended by Istio developers" territory. Its not supported, tested, or even designed to work at all.

istio / istio