tikv client in Kubernetes

[tkalanick]

Hi, this news CNCF to host TiKV in sandbox has been out for some time. can you document how to re-purpose tidb-operator to do TiKV client development in Kubernetes? suppose i don't want the TiDB service. in stead i 'd like to write my own TiKV client according to this doc. can i just deploy my service into the same tidb cluster and expect everything to work ? it certainly looks like that way (client talks to PD, PD forwards to TiKV) but i just want to double check. or better yet, can someone write an example K8S service to demonstrate? i am still trying to figure out how the client to TiKV communication is set up in a K8S environment. how does a client address individual TiKV nodes?

[weekface]

@tkalanick Yes, you should deploy your service into the same tidb cluster.

The PD stores the metadata of the cluster, such as the TiKV address and the region location of a specific key. Therefore, the client can obtain the TiKV address from the PD and then communicate with the TiKV after that.

[gregwebs]

Yes, we should have example usage of TiKV. If you run your service in the cluster, it should work. There is an issue with exposing TiKV outside the cluster right now. Si you would just scale your TiDB down to zero if you aren't using that.

Are you using golang? We also have a rust driver coming out right now.

[tkalanick]

i am using golang. thanks for confirming that the client will just work if i drop my service in the cluster.

in the mean time, i am trying to run dev test cases with a client running on my laptop and TiDB cluster in GCP. i tried to port-forward all services including pd, kv-peer, and pd-peer. but that didn't work. do you know of any work arounds? in order for the client to talk to individual regions all the tikv nodes needs to be exposed, correct? is there a way to set up a service that routes the requests from clients to tikv? we can use that service for development testing only. so you can remove performance requirements.

i need a good set up for development purpose. dind didn't work for me. some nodes simply didn't start. and i thought docker in docker is a set up too complicated for me to debug if it didn't work right away. i didn't try minikube. looks like it doesn't support a lot of the K8S features that we need.

[gregwebs]

dind is a bit hacky :( Minikube is a single node so doesn't work with tidb-operator.

Rather than port-forward, can you deploy your code to GCP? @tennix can you document how to use TiKV form inside K8s?

[tkalanick]

@tennix. can you tell what is wrong?

i deployed a service inside the same tidb namespace of K8S as a deployment.
and in the deployment i did

    d := ti.Driver{}
    store, err := d.Open(fmt.Sprintf("tikv://%s/pd?cluster=1", "127.0.0.1:2379"))

but this fails with the following error: textPayload: "time="2019-01-16T21:02:06Z" level=error msg="[pd] failed to get cluster id: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused\""

in this case, is the client is trying to connect to demo-pd service which is available on the well known port 2379 both inside and outside of the cluster?

[tennix]

@tkalanick When you deploy the tikv client service in the same namespace, you should use tikv://{cluster-name}-pd:2379 instead of 127.0.0.1:2379. And the port-forward doesn't work for TiKV client. Since the client has to connect all pd and tikv via the registered domain names other than IP addresses.

You can run a ycsb test for the tikv cluster using the following command:

kubectl run --namespace=tidb -i --tty --rm --image=pingcap/go-ycsb tikv-ycsb-demo --command -- /go-ycsb shell tikv -p tikv.p
d=demo-pd:2379
If you don't see a command prompt, try pressing enter.
INFO[0001] [pd] leader switches to: http://demo-pd-0.demo-pd-peer.tidb.svc:2379, previous:
INFO[0001] [pd] init cluster id 6645466362752766936
» read a
Read empty for a
» insert a field0=0
Insert a ok
» read a
Read a ok
field0="0"

[tennix]

As @gregwebs said, we can't expose TiKV service like TiDB. TiDB client which just speaks MySQL protocal doesn't care what the backend is, so it's unaware of the PD.

But TiKV client currently has to talk to both PD and TiKV, and to use a stable network ID for the stateful app PD and TiKV on Kubernetes, tidb-operator uses DNS domain name instead of Pod IP address as the advertise address. And the domain name is non-routable outside of the cluster. So we can't simply deploy TiKV clients outside of the Kubernetes cluter right now.

However there is another approach which allows to deploy tikv client application outside of the Kubernetes cluster: Deploy a kv proxy inside the Kubernetes cluster. The proxy connects to both PD and TiKV pods and exposes a kv service. And the application just talks to the exposed kv service.

This appoach requires a lot of work for the kv proxy. But I think this will benefit the whole TiKV community, it makes the TiKV more easy to use. And I hope the TiKV team (@siddontang) and community can provide such a solution.

[tkalanick]

@tennix thank you very much for your help. i changed my pd address to tikv://{cluster-name}-pd:2379 and myservice is now working.

one follow up question - what is the sanctioned TiKV client class (kv.Storage or tikv.RawKVClient) can you describe the difference between the two?

i actually prefer a higher layer KV API, one that handles encoding and even table definitions. i only consider the SQL layer as an overhead. can people build applications on top of the table.Table api ? which layer do those TiKV customers use in production?

[siddontang]

@tkalanick

RawKVClient is for Raw mod (you can think just use common KV API like Get, Put, Delete, etc.), and kv.Storage(actually TxnKVClient internal) is for the distributed transaction. They are both used in some users' production environments. But be remember that don't use them together in the same cluster.

It is not recommended to use table.Table API, either you can use TiDB (the SQL layer), or you can use Raw/Txn client directly. Another choice is to use a Redis layer (there are some open source projects doing it).

[gregwebs]

@tennix can we add a note on connecting to TiKV in the docs now?