Add support for Istio with Ambient mode

[lapaartis]

Description

With the release of Istio's Ambient mode (Beta), decided to transition from the Istio Sidecars + Istio-CSR setup to Istio Ambient + Istio-CSR. Successfully redeployed Istio with Ambient mode (v1.22.0), but encountered an issue in Istio-CSR related to identity matching. The error occurs when identities are proxied via Ztunnel to Istio-CSR.

Istio Ambient mode Beta version was released and decided try to switch from Istio Sidecars + Istio-CSR to setup with Istio Ambient + Istio-CSR.

Re-deploy Istio with Ambient mode (v1.22.0) went successfully but issue pop out in istio-csr with identity match for spiffe proxied via Ztunnel to istio-csr.

Error in istio-csr pod:

error   klog    [spiffe://cluster.local/ns/istio-system/sa/ztunnel] != [spiffe://cluster.local/ns/example/sa/my-app]: failed to match URIs with identities  logger=grpc-server serving-addr=0.0.0.0:6443 identities=spiffe://cluster.local/ns/istio-system/sa/ztunnel

error in ztunnel pod:

error   access  connection complete src.addr=10.0.0.121:44562 src.workload=my-app-595cf9cf9c-db6x8 src.namespace=example src.identity="spiffe://cluster.local/ns/example/sa/my-app" dst.addr=10.0.0.58:15008 dst.hbone_addr=10.0.0.58:8080 dst.service=my-app.example.svc.cluster.local dst.workload=my-app-78f4dff8d8-fgfp7 dst.namespace=example dst.identity="spiffe://cluster.local/ns/example/sa/my-app" direction="outbound" bytes_sent=0 bytes_recv=0 duration="0ms" error="identity error: signing gRPC error (The request does not have valid authentication credentials): request authenticate failure

Reaching out Istio Ambient team on Slack confirmed that support for Istio-CSR in Ambient mode needs to be implemented in Istio-CSR.

Used Helm values for Istio with sidecars before switch to Ambient mode

cert-manager/istio-csr helm values:

app:
  tls:
    rootCAFile: /var/run/secrets/istio-csr/ca.pem
    certificateDNSNames:
      - istio-csr.cert-manager.svc
      - cert-manager-istio-csr.cert-manager.svc

volumes:
- name: root-ca
  secret:
    secretName: istio-root-ca

volumeMounts:
- name: root-ca
  mountPath: /var/run/secrets/istio-csr

istio/istiod helm values:

global:
  caAddress: cert-manager-istio-csr.cert-manager.svc:443

pilot:
  extraContainerArgs:
    - --tlsCertFile=/etc/cert-manager/tls/tls.crt
    - --tlsKeyFile=/etc/cert-manager/tls/tls.key
    - --caCertFile=/etc/cert-manager/ca/root-cert.pem

  env: 
    ENABLE_CA_SERVER: false

  volumeMounts:
    - mountPath: /etc/cert-manager/tls
      name: cert-manager
      readOnly: true
    - mountPath: /etc/cert-manager/ca
      name: istio-csr-ca-configmap
      readOnly: true

  # Additional volumes to the istiod pod
  volumes: []
    - name: cert-manager
      secret:
        defaultMode: 420
        secretName: istiod-tls
    - configMap:
        defaultMode: 420
        name: istio-ca-root-cert
        optional: true
      name: ca-root-cert

Changes and switch to Ambient mode and Ztunnel

I followed the instructions for installing Ambient Mesh as outlined here: Istio Ambient Mesh Installation. The CA Address was configured in the same manner as the previous sidecar setup.

For ztunnel set that CA Address will be Istio-CSR (same address was used with Istio sidecars):

istio/ztunnel helm values:

caAddress: "cert-manager-istio-csr.cert-manager.svc:443"

Note

Switching CA from Istio-CSR to the default Istiod ambient mode setup without Istio-CSR configuration our ambient mesh started working.

[SpectralHiss]

Hi @lapaartis, thanks for raising this! We are able to reproduce and you are right this is an issue due to the way the authentication works in istio-csr, expecting the workload's own bound service account token to be presented instead of the ztunnel one which obviously differs in Ambient.

I suggest we add an exception to the ztunnel sa, the only difficulty would be that we need to ensure that we have the rright name and in the correct namespace and workloads don't use this to impersonate pods if they don't have a ztunnel installed!

Also @lapaartis , and this is besides the main issue, you can simplify your Istiod config like so:

global:
  caAddress: cert-manager-istio-csr.cert-manager.svc:443
pilot:
  env: 
    ENABLE_CA_SERVER: false

The rest is automatically added since Istio 1.16: https://istio.io/latest/news/releases/1.16.x/announcing-1.16/change-notes/#security

[howardjohn]

@SpectralHiss fwiw in Istio itself we have a CA_TRUSTED_NODE_ACCOUNTS that takes a list of SA's that are authorized to act in this regard (istio-system/ztunnel typically)

[howardjohn]

I put up a draft in https://github.com/cert-manager/istio-csr/pull/335. It 'works' but doesn't do enough auth. But it does prove that adding the auth is all that is blocking things working e2e. I would be happy for someone to carry on the PR or do it myself.

I would want some guidance though on whether it is preferred to just pull in https://github.com/istio/istio/blob/9ca112c308c2c4ab5d595d6ba052a88212dd00f7/security/pkg/server/ca/node_auth.go#L74 or to re-implement it

[paulwilljones]

Thanks @howardjohn. I'm working on #336 to handle impersonation but need to add in node authorization. Looking to do something similar to https://github.com/istio/istio/blob/1.22.1/security/pkg/server/ca/node_auth.go#L110 to check the pod is on the same node as the ztunnel.

[howardjohn]

Nice! I skimmed it really quickly and looks like the right path. I'll put aside some time to make sure to give it a close review and try it out.