search

kopeio / etcd-manager

operator for etcd: moved to https://github.com/kubernetes-sigs/etcdadm
Apache License 2.0
164 stars 45 forks source link

Recommendation for ETCD node type for a large cluster #349

Open jsonmp-k8 opened 4 years ago

jsonmp-k8 commented 4 years ago

This is more of a question than an issue.

We were trying to do an perf test on the ETCD using the ETCDCTL command. This is a KOPS cluster that we use and it keeps failing for a large workload performance test. Do you have any recommendations for the node sizes for the master nodes for a large cluster? Master node is where the etcd manager, ETCD and the control plane components reside.

I used the below command to run a performance test with the workload size l (large).

ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/kube-apiserver/etcd-ca.crt --cert=/etc/kubernetes/pki/kube-apiserver/etcd-client.crt --key=/etc/kubernetes/pki/kube-apiserver/etcd-client.key --endpoints https://127.0.0.1:4001 check perf --load="l"
 60 / 60 Boooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo! 100.00% 1m0s
FAIL: Throughput too low: 7183 writes/s
PASS: Slowest request took 0.074107s
PASS: Stddev is 0.004803s
FAIL

This cluster is created on AWS and the master node size is r5.4xlarge. I do have the following config for the ETCD manager and events

      volumeType: io1
      volumeIops: 3000
      volumeSize: 200

Any recommendations are welcome!