K3S_RESILIENCY

[Jean-Baptiste-Lasselle]

• The cluster uses by default, a datastore that is very fragile : After looking at the logs, I knew needed to set up an external ETCD to hadle that.

This is coherent with what Kubernetes recommends in HA : etcd must be a clusterized one, for resiliency

[Jean-Baptiste-Lasselle]

git clone https://github.com/dwilbraham/docker-compose-etcd
cd docker-compose-etcd/
git clone https://github.com/henszey/etcd-browser/ etcd-browser/
dcoker-compose up -d

• after that, I run the tests :

  
  etcd=$(bin/service_address.sh etcd0 2379)
  curl $etcd/v2/keys
  curl $etcd/v2/keys/foo -XPUT -d value=bar

etcd_all="{$(bin/service_address.sh etcd0 2379),$(bin/service_address.sh etcd1 2379),$(bin/service_address.sh etcd2 2379)}" curl $etcd_all/v2/keys curl $etcd_all/v2/stats/leader

* then I have : 

```bash
~/docker-compose-etcd$ docker ps -a
CONTAINER ID        IMAGE                              COMMAND                  CREATED              STATUS              PORTS                               NAMES
e09a9d2a0a62        docker-compose-etcd_etcd-browser   "nodejs server.js"       About a minute ago   Up About a minute   0.0.0.0:32771->8000/tcp             docker-compose-etcd_etcd-browser_1
de95c862f160        quay.io/coreos/etcd                "/usr/local/bin/etcd…"   About a minute ago   Up About a minute   2380/tcp, 0.0.0.0:32770->2379/tcp   docker-compose-etcd_etcd2_1
3e601e6829c5        quay.io/coreos/etcd                "/usr/local/bin/etcd…"   About a minute ago   Up About a minute   2380/tcp, 0.0.0.0:32768->2379/tcp   docker-compose-etcd_etcd1_1
2d6cb298369c        quay.io/coreos/etcd                "/usr/local/bin/etcd…"   About a minute ago   Up About a minute   2380/tcp, 0.0.0.0:32769->2379/tcp   docker-compose-etcd_etcd0_1

• And the test run to check that the full etcd service is up n running :

  
  jbl@poste-devops-jbl-16gbram:~/docker-compose-etcd$ etcd=$(bin/service_address.sh etcd0 2379)
  jbl@poste-devops-jbl-16gbram:~/docker-compose-etcd$ curl $etcd/v2/keys
  {"action":"get","node":{"dir":true}}
  jbl@poste-devops-jbl-16gbram:~/docker-compose-etcd$ curl $etcd/v2/keys/foo -XPUT -d value=bar
  {"action":"set","node":{"key":"/foo","value":"bar","modifiedIndex":8,"createdIndex":8}}
  jbl@poste-devops-jbl-16gbram:~/docker-compose-etcd$ 
  jbl@poste-devops-jbl-16gbram:~/docker-compose-etcd$ etcd_all="{$(bin/service_address.sh etcd0 2379),$(bin/service_address.sh etcd1 2379),$(bin/service_address.sh etcd2 2379)}"
  jbl@poste-devops-jbl-16gbram:~/docker-compose-etcd$ curl $etcd_all/v2/keys

[1/3]: localhost:32769/v2/keys --> --curl--localhost:32769/v2/keys {"action":"get","node":{"dir":true,"nodes":[{"key":"/foo","value":"bar","modifiedIndex":8,"createdIndex":8}]}}

[2/3]: localhost:32768/v2/keys --> --curl--localhost:32768/v2/keys {"action":"get","node":{"dir":true,"nodes":[{"key":"/foo","value":"bar","modifiedIndex":8,"createdIndex":8}]}}

[3/3]: localhost:32770/v2/keys --> --curl--localhost:32770/v2/keys {"action":"get","node":{"dir":true,"nodes":[{"key":"/foo","value":"bar","modifiedIndex":8,"createdIndex":8}]}} jbl@poste-devops-jbl-16gbram:~/docker-compose-etcd$ jbl@poste-devops-jbl-16gbram:~/docker-compose-etcd$ curl $etcd_all/v2/stats/leader

[1/3]: localhost:32769/v2/stats/leader --> --curl--localhost:32769/v2/stats/leader {"message":"not current leader"} [2/3]: localhost:32768/v2/stats/leader --> --curl--localhost:32768/v2/stats/leader {"leader":"ade526d28b1f92f7","followers":{"cf1d15c5d194b5c9":{"latency":{"current":0.003147,"average":0.006053428571428571,"standardDeviation":0.005520999000107108,"minimum":0.001863,"maximum":0.017602},"counts":{"fail":0,"success":7}},"d282ac2ce600c1ce":{"latency":{"current":0.002747,"average":0.0036824285714285717,"standardDeviation":0.002598938566697833,"minimum":0.001163,"maximum":0.009535},"counts":{"fail":0,"success":7}}}} [3/3]: localhost:32770/v2/stats/leader --> --curl--localhost:32770/v2/stats/leader {"message":"not current leader"}jbl@poste-devops-jbl-16gbram:~/docker-compose-etcd$ jbl@poste-devops-jbl-16gbram:~/docker-compose-etcd$

[Jean-Baptiste-Lasselle]

next step : https://rancher.com/docs/k3s/latest/en/installation/ha/

• add traefik to docker-compose, so that you laod balance across the 3 etcd. and add 127.0.0.1 etcd.ide.io to /etc/hosts on the machine where both the k3d cluster and the etcd datastore service run.
• launch the cluster with added server option --datastore-endpoint=http://etcd.ide.io where it hits traefik and gets load-balanced across the 3 etcd containers
• other example, this time postgres as datastore provider for the k3s Kubernetes cluster load balanced and scale with docker compose : https://github.com/brijesh-deb/docker-compose-sample related to another public repo from the same github user. I'll try n replace the HAProxy by the Traefik v2 load-balancer reverse proxy.
• the scale up approach is th good approach