Allow configure TiKV to use IP addresses instead of DNS

[Tema]

Feature Request

Is your feature request related to a problem? We try to use tidb-operator to deploy TiKV cluster w/o TiDB layer, so TiKV client is usually deployed to other k8s cluster and don't have access top internal kube-dns with information about TiKV cluster. Also DNS could be cached at some layers and result in the issues when TiKV/PD node is replaced.

Describe the feature you'd like: We would like to have an ability to configure TiKV to use IP address rather than ${POD_NAME}.${PEER_SERVICE_NAME}.${NAMESPACE}.svc DNS names.

[mikechengwei]

I think this requires tikv to support ip communication natively,tidb-operator can't finish this feature. I think you can try to get through the domain names of two k8s clusters.

[DanielZhangQD]

@Tema As you know that the Pod IP is gonna change whenever you delete the Pod, so if you want to use IP address, do you want to update the config in the clients each time you delete the TiKV Pod or upgrade the TiKV?

[Tema]

I think this requires tikv to support ip communication natively,tidb-operator can't finish this feature.

I don't think the communication part is controlled by operator, it just needs to allow to configure startup script for tikv-server to use IP of the node for --advertise-addr (we use "hostNetwork:true"). https://github.com/pingcap/tidb-operator/blob/d845470d2616a9d54d386856e9fd83c212c03349/charts/tidb-cluster/templates/scripts/_start_tikv.sh.tpl#L33-L34

do you want to update the config in the clients each time you delete the TiKV Pod or upgrade the TiKV?

I would use some external DNS or Load Balancer to discover PD endpoint from the client, but PD would keep track of all TiKV storage nodes using IP addresses and I hope client would get new IP addresses through PD automatically on regular region map update. Or do I miss something and it won't work this way?

[mikechengwei]

I think this requires tikv to support ip communication natively,tidb-operator can't finish this feature.

I don't think the communication part is controlled by operator, it just needs to allow to configure startup script for tikv-server to use IP of the node for --advertise-addr (we use "hostNetwork:true").

https://github.com/pingcap/tidb-operator/blob/d845470d2616a9d54d386856e9fd83c212c03349/charts/tidb-cluster/templates/scripts/_start_tikv.sh.tpl#L33-L34

do you want to update the config in the clients each time you delete the TiKV Pod or upgrade the TiKV?

I would use some external DNS or Load Balancer to discover PD endpoint from the client, but PD would keep track of all TiKV storage nodes using IP addresses and I hope client would get new IP addresses through PD automatically on regular region map update. Or do I miss something and it won't work this way?

I think tikv does not support ip communication, it only supports hostname. You must make the external client of the k8s cluster aware of these hostnames. You can configure the hostname to the dns of the external client, or the local hosts file.

[Tema]

@mikechengwei

I think tikv does not support ip communication, it only supports hostname.

TiKV does support direct IP address configuration: https://docs.pingcap.com/tidb/dev/command-line-flags-for-tikv-configuration#--advertise-addr, bit tidb-operator doesn't allow to configure it.

DNS configuration is much easier, just not as reliable as IP addresses. Connecting by IP address also requires IP address written in the SAN:IP of certificate for TLS communication, which is another problem.

If anyone in the same boat, we found a pretty easy way for cross k8s cluster discovery by creating an ILB for kube-dns on the TiKV k8s cluster:

apiVersion: v1
kind: Service
metadata:
  annotations:
    # this annotation is for GKE only. Other cloud providers have their own annotations.
    cloud.google.com/load-balancer-type: Internal
  name: kube-dns-udp-lb
  namespace: kube-system
spec:
  ipFamilies:
  - IPv4
  ports:
  - name: dns
    port: 53
    protocol: UDP
    targetPort: 53
  selector:
    k8s-app: kube-dns
  sessionAffinity: None
  type: LoadBalancer

then configuring stubDomain of the kube-dns on the client k8s cluster with $EXTERNAL-IP found by running kubectl -n kube-system get service kube-dns-udp-lb:

cat <<EOF | kubectl apply -f -
apiVersion: v1
metadata:
  name: kube-dns
  namespace: kube-system
data:
  stubDomains: |
    {"tidb-cluster.svc.cluster.local" : ["$EXTERNAL-IP"]}
kind: ConfigMap
EOF

Thus a client can connect to PD same way as from inside tikv k8s cluster by <CLUSTER_NAME>-pd-peer.tidb-cluster.svc:2379