search

GoogleCloudPlatform / anthos-service-mesh-packages

Packaged configuration for setting up a Kubernetes cluster with Anthos Service Mesh features enabled
https://cloud.google.com/anthos/service-mesh
Apache License 2.0
134 stars 168 forks source link

Istio-Proxy fails with automountServiceAccountToken: false #1210

Open jacekszlachtass opened 2 years ago

jacekszlachtass commented 2 years ago

According to this issue https://github.com/istio/istio/issues/22193 pods wth "automountServiceAccountToken: false" should work if JWT policy is third-party-jwt and the cluster supports third party tokens:

❯ kubectl get --raw /api/v1 | jq '.resources[] | select(.name | index("serviceaccounts/token"))'

{
  "name": "serviceaccounts/token",
  "singularName": "",
  "namespaced": true,
  "group": "authentication.k8s.io",
  "version": "v1",
  "kind": "TokenRequest",
  "verbs": [
    "create"
  ]
}

❯ k logs argocd-repo-server-6b76cd49fd-v76sg -c istio-proxy -f
2022-06-05T12:37:06.209016Z     info    FLAG: --concurrency="2"
2022-06-05T12:37:06.209052Z     info    FLAG: --domain="argocd.svc.cluster.local"
2022-06-05T12:37:06.209058Z     info    FLAG: --help="false"
2022-06-05T12:37:06.209061Z     info    FLAG: --log_as_json="false"
2022-06-05T12:37:06.209064Z     info    FLAG: --log_caller=""
2022-06-05T12:37:06.209068Z     info    FLAG: --log_output_level="default:info"
2022-06-05T12:37:06.209071Z     info    FLAG: --log_rotate=""
2022-06-05T12:37:06.209074Z     info    FLAG: --log_rotate_max_age="30"
2022-06-05T12:37:06.209077Z     info    FLAG: --log_rotate_max_backups="1000"
2022-06-05T12:37:06.209080Z     info    FLAG: --log_rotate_max_size="104857600"
2022-06-05T12:37:06.209084Z     info    FLAG: --log_stacktrace_level="default:none"
2022-06-05T12:37:06.209097Z     info    FLAG: --log_target="[stdout]"
2022-06-05T12:37:06.209102Z     info    FLAG: --meshConfig="./etc/istio/config/mesh"
2022-06-05T12:37:06.209106Z     info    FLAG: --outlierLogPath=""
2022-06-05T12:37:06.209111Z     info    FLAG: --proxyComponentLogLevel="misc:error"
2022-06-05T12:37:06.209116Z     info    FLAG: --proxyLogLevel="warning"
2022-06-05T12:37:06.209121Z     info    FLAG: --serviceCluster="istio-proxy"
2022-06-05T12:37:06.209126Z     info    FLAG: --stsPort="15463"
2022-06-05T12:37:06.209130Z     info    FLAG: --templateFile=""
2022-06-05T12:37:06.209133Z     info    FLAG: --tokenManagerPlugin="GoogleTokenExchange"
2022-06-05T12:37:06.209146Z     info    FLAG: --vklog="0"
2022-06-05T12:37:06.209158Z     info    Version 1.13.2-asm.5-1394c2b799862444a4bb5f1590b2f6aa7764c94e-Clean
2022-06-05T12:37:06.209392Z     info    Proxy role      ips=[192.168.3.87] type=sidecar id=argocd-repo-server-6b76cd49fd-v76sg.argocd domain=argocd.svc.cluster.local
2022-06-05T12:37:06.209519Z     info    Apply proxy config from env {"discoveryAddress":"istiod-asm-1132-5.istio-system.svc:15012","proxyMetadata":{"CA_PROVIDER":"GoogleCA","GCE_METADATA_HOST":"metadata.google.internal","GCP_METADATA":"fow-sandbox|860502011411|fow-sandbox|europe-west2","GKE_CLUSTER_URL":"https://container.googleapis.com/v1/projects/fow-sandbox/locations/europe-west2/clusters/fow-sandbox","PLUGINS":"GoogleTokenExchange","USE_TOKEN_FOR_CSR":"true"},"meshId":"proj-860502011411"}

2022-06-05T12:37:06.211123Z     info    Effective config: binaryPath: /usr/local/bin/envoy
concurrency: 2
configPath: ./etc/istio/proxy
controlPlaneAuthPolicy: MUTUAL_TLS
discoveryAddress: istiod-asm-1132-5.istio-system.svc:15012
drainDuration: 45s
meshId: proj-860502011411
parentShutdownDuration: 60s
proxyAdminPort: 15000
proxyMetadata:
  CA_PROVIDER: GoogleCA
  GCE_METADATA_HOST: metadata.google.internal
  GCP_METADATA: fow-sandbox|860502011411|fow-sandbox|europe-west2
  GKE_CLUSTER_URL: https://container.googleapis.com/v1/projects/fow-sandbox/locations/europe-west2/clusters/fow-sandbox
  PLUGINS: GoogleTokenExchange
  USE_TOKEN_FOR_CSR: "true"
serviceCluster: istio-proxy
statNameLength: 189
statusPort: 15020
terminationDrainDuration: 5s
tracing:
  zipkin:
    address: zipkin.istio-system:9411

2022-06-05T12:37:06.211155Z     info    JWT policy is third-party-jwt
2022-06-05T12:37:06.211171Z     info    Extract GCP metadata from env variable GCP_METADATA: fow-sandbox|860502011411|fow-sandbox|europe-west2
2022-06-05T12:37:06.297730Z     info    platform detected is GCP
2022-06-05T12:37:06.297774Z     info    stsserver       Start listening on 127.0.0.1:15463
2022-06-05T12:37:06.298168Z     info    CA Endpoint meshca.googleapis.com:443, provider GoogleCA
2022-06-05T12:37:06.298198Z     info    Opening status port 15020
2022-06-05T12:37:06.298872Z     info    ads     All caches have been synced up in 94.245355ms, marking server ready
2022-06-05T12:37:06.299231Z     info    sds     SDS server for workload certificates started, listening on "etc/istio/proxy/SDS"
2022-06-05T12:37:06.299274Z     info    xdsproxy        Initializing with upstream address "istiod-asm-1132-5.istio-system.svc:15012" and cluster "cn-fow-sandbox-europe-west2-fow-sandbox"
2022-06-05T12:37:06.299445Z     error   stsserver       http: Server closed
2022-06-05T12:37:06.299468Z     info    sds     Starting SDS grpc server
2022-06-05T12:37:06.299495Z     info    Status server has successfully terminated
2022-06-05T12:37:06.299521Z     error   accept tcp [::]:15020: use of closed network connection
2022-06-05T12:37:06.300894Z     error   failed to start xds proxy: failed to build TLS dial option to talk to upstream: failed to find root CA cert for XDS: root CA file for XDS does not exist ./var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Error: failed to start xds proxy: failed to build TLS dial option to talk to upstream: failed to find root CA cert for XDS: root CA file for XDS does not exist ./var/run/secrets/kubernetes.io/serviceaccount/ca.crt

Changing to "automountServiceAccountToken: true" fixes the issue.

Any idea why it is not working with "automountServiceAccountToken: false"?

shankgan commented 2 years ago

As of now, the code assumes that the service Account is always mounted at "/var/run/secrets/kubernetes.io". This should not be a hard requirement though. We incorrectly use the "ca.crt" in the above directory to TLS authenticate the Istiod control plane. Looking into fixing this..

tahaozket commented 1 year ago

Hi @shankgan any updates on this? We hit the same issue on Anthos Service Mesh 1.13.7.

lkysow commented 1 year ago

Anyone else running into this you might need to set:

kubectl annotate --overwrite namespace default \
mesh.cloud.google.com/proxy='{"managed":"true"}'