Istio mtls support

[sstubbs]

Feature Request

Is your feature request related to a problem? Please describe: I'm using istio 1.6.4 and would like to enable mtls. I've create a tidb cluster with the following:

kubectl create namespace tidb && \
kubectl label namespace tidb istio-injection=enabled && \
kubectl apply -n tidb -f 11-tidb/cluster.yaml

The cluster.yaml looks like the following:

apiVersion: pingcap.com/v1alpha1
kind: TidbCluster
metadata:
  name: cluster-1
spec:
  version: v4.0.1
  timezone: UTC
  pvReclaimPolicy: Retain
  discovery: {}
  pd:
    baseImage: registry.cn-beijing.aliyuncs.com/tidb/pd
    replicas: 1 
    storageClassName: high-performance
    requests:
      storage: "250Gi"
    config: {}
  tikv:
    baseImage: registry.cn-beijing.aliyuncs.com/tidb/tikv
    replicas: 1
    storageClassName: high-performance
    requests:
      storage: "250Gi"
    config: {}
  tidb:
    baseImage: registry.cn-beijing.aliyuncs.com/tidb/tidb
    replicas: 1
    service:
      type: ClusterIP
    config: {}

This works. However once I enable mtls like the following:

kubectl apply -n tidb -f 11-tidb/peer-authentication.yaml

with this peer-authentication.yaml file:

apiVersion: "security.istio.io/v1beta1"
kind: "PeerAuthentication"
metadata:
  name: "default"
spec:
  mtls:
    mode: STRICT

I get this error when running sql statements against it:

ERROR 9001 (HY000): PD server timeout

It would be great if this could be supported.

Describe the feature you'd like: Support for istio mtls

Describe alternatives you've considered: I've considered using the native tls of tidb but would rather use istio mtls as I'm using it elsewhere and it theoretically should be more secure as it's checking the certificate from both sides.

[DanielZhangQD]

PD only supports one CN, not sure if istio supports setting the CN. cc @weekface

[DanielZhangQD]

@sstubbs Did you get istio mtls work with TiDB?

[sstubbs]

I have tried everything I can think of with no luck unfortunately.

[sstubbs]

Not sure if this helps but when I install it with istio sidecar it works as the default is permissive. I use the following:

kubectl create namespace tidb-admin && \
kubectl label namespace tidb-admin istio-injection=enabled && \
helm install tidb-operator pingcap/tidb-operator --namespace=tidb-admin --version=1.1.2 -f 09-tidb-operator/values.yaml && \
kubectl get po -n tidb-admin -l app.kubernetes.io/name=tidb-operator

and

kubectl create namespace tidb && \
kubectl label namespace tidb istio-injection=enabled && \
kubectl apply -n tidb -f 10-tidb/cluster.yaml

Strangely kiali reports an error in this graph like the following even though it seems to work fine.

If I apply this peerauthentication:

apiVersion: "security.istio.io/v1beta1"
kind: "PeerAuthentication"
metadata:
  name: "default"
spec:
  mtls:
    mode: STRICT

to both namespaces of the operator and cluster. The graph doesn't report errors but it doesn't work anymore. I get the following:

[sstubbs]

On second on thought if there is no traffic being allowed through it makes sense it doesn't show errors. I will just use tidb encryption for the time being.

[DanielZhangQD]

@sstubbs Thanks for the reply! Not sure if the PR https://github.com/pingcap/tidb/pull/17539 (released in TiDB v4.0.2, docs update https://github.com/pingcap/docs/pull/2973) can help resolve this issue.

[sstubbs]

I'm having issues with mtls in other application too so I won't be using this anyway.

[dragonly]

@sstubbs Is the issue solved?

according to https://istio.io/latest/docs/tasks/security/authentication/authn-policy, this seems an istio configuration problem, because TiDB cluster components should just send plain texts, and leave all the tls encrypt/decrypt to istio's envoy proxy.