diotriandika commented 1 week ago

Specs

Melakukan instalasi Kubernetes high availability multi control-plane dengan spesifikasi berikut	Requirement	Spec
Control plane	3
Worker	2
RAM	2GiB
CPU	2
Storage	20 GiB
CRI	Containerd
CNI	Calico
Kubernetes version	Latest
Operating System	Ubuntu Jammy LTS

Todo

[x] Install Kubernetes HA
[x] Install CNI Calico
[x] Label node worker dengan type=app & node-role.kubernetes.io/worker=
[x] Sebar pod CoreDNS hanya di master node
[x] Deploy metric-server
[x] Deploy Kubernetes dashboard
[x] Deploy Ingress nginx
[x] Buat directory workdir pada homedir untuk menyimpan semua manifest
[x] Buat deployment dengan image nginx:1.19 dengan nama btech-testpod-<your_name> dengan 2 replika
[x] Buat SVC dengan type ClusterIP
[ ] Buat ingress dan rewrite ke path /testpath
[x] Test hit ingress dengan curl menggunakan while true dan sleep selama 1 detik (jalankan di tmux)
[x] Cek log deployment test dan simpan ke file di directory workdir
[x] Scale deployment test ke 1 replika
[ ] Install victoria metrics di salah satu node, referensi
[ ] Deploy Victoria metrics operator, referensi
[ ] Deploy NFS External Subdir, referensi
[ ] Create backup schedule, referensi
[ ] Buat monitoring kubernetes menggunakan victoria metrics dan grafana
- [ ] kube-apiserver
- [ ] kube-controller-manager
- [ ] kube-scheduler
- [ ] kube-state-metrics
- [ ] kubelet
- [ ] nginx-ingress-controller
- [ ] etcd
- [ ] coredns
- [ ] ping-exporter
- [ ] blackbox-exporter
- [ ] x509-certificate-exporter

Task dibuat oleh mas @ngurah_bagus_trisna :)

diotriandika commented 1 week ago

Step 1 - Provisioning Cluster

Tasks:

[x] Install Kubernetes HA
[x] Install CNI Calico & CRI Containerd
[x] Label node worker dengan type=app & node-role.kubernetes.io/worker=

Setup Containerd as CRI

CRI merupakan sebuah container engine yang nantinya digunakan oleh kubernetes untuk menjalankan semua container kuberentes.

Jalankan disemua Node

Load module networking yang diperlukan

# Load overlay & br_netfilter module into running kernel.
$ sudo modprobe overlay && sudo modprobe br_netfilter

# Load on boot modules for containerd
$ cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF

Konfigurasi sysctl yang diperlukan dan buat persisten dari reboot.

# Configure sysctl
$ cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

# Apply the configuration and make it persistent on boot.
$ sudo sysctl --system

Selanjutnya install package depedencies, apt key dan repository yang diperlukan untuk instalasi containerd

# Add Docker's offical GPG key: (the same repo as containerd used.)
$ sudo apt-get update
$ sudo apt-get install ca-certificates curl gnupg apt-transport-https -y
$ sudo install -m 0755 -d /etc/apt/keyrings
$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
$ sudo chmod a+r /etc/apt/keyrings/docker.gpg

# Add the repository to Apt sources:
$ echo \
 "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
 $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
 sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

# Update repository & install containerd
$ sudo apt-get update && sudo apt-get install containerd.io -y

Buat default configuration file containerd

# Use root user
$ sudo su -

# Create containerd config directory
mkdir -p /etc/containerd

# Create default configuration file containerd
containerd config default | sudo tee /etc/containerd/config.toml

Selanjutnya rubah file containerd configuration diatas agar mengaktifkan cgroupDriver untuk runc yang mana nantinya diperlukan oleh kubelet.
```
sed -i '/SystemdCgroup/s/false/true/g' /etc/containerd/config.toml
```

Restart service containerd lalu enable on boot.

# estart containerd service
sudo systemctl restart containerd

# Enable automatic startup on system boot
sudo systemctl enable --now containerd

Referensi:

https://www.nocentino.com/posts/2021-12-27-installing-and-configuring-containerd-as-a-kubernetes-container-runtime/

Setup KeepAlived as VRRP

Karena ditask tidak ada node LoadBalancer, maka dari itu saya menggunakan KeepAlived untuk membuat VRRP dan HAProxy sebagai LoadBalancer disemua node control plane.

Jalankan di semua Control Plane Node

Install KeepAlived dan HAProxy

$ sudo apt-get install keepalived haproxy -y

Buat file konfigurasi keepalived di node control-plane-(1-3)

$ sudo nano /etc/keepalived/keepalived.conf

control-plane-1

global_defs {
}
vrrp_script chk_haproxy {
   script "killall -0 haproxy" 
   interval 2 
   weight 2 
}
vrrp_instance haproxy_vip {
   interface ens33
   state MASTER
   priority 300
   virtual_router_id 50
   authentication {
       auth_type PASS
       auth_pass ayamgoreng
   }
virtual_ipaddress {
       172.16.1.15/24
   }
unicast_src_ip 172.16.1.10  # This node
   unicast_peer {
   172.16.1.11          # Other nodes
   172.16.1.12          # Other nodes
   }
track_script {
       chk_haproxy
   }
}

control-plane-2

global_defs {
}
vrrp_script chk_haproxy {
   script "killall -0 haproxy" 
   interval 2 
   weight 2 
}
vrrp_instance haproxy_vip {
   interface ens33
   state BACKUP
   priority 200
   virtual_router_id 50
   authentication {
       auth_type PASS
       auth_pass ayamgoreng
   }
virtual_ipaddress {
       172.16.1.15/24
   }
unicast_src_ip 172.16.1.11  # This node
   unicast_peer {
   172.16.1.10          # Other nodes
   172.16.1.12          # Other nodes
   }
track_script {
       chk_haproxy
   }
}

control-plane-3

global_defs {
}
vrrp_script chk_haproxy {
   script "killall -0 haproxy" 
   interval 2 
   weight 2 
}
vrrp_instance haproxy_vip {
   interface ens33
   state BACKUP
   priority 100
   virtual_router_id 50
   authentication {
       auth_type PASS
       auth_pass ayamgoreng
   }
virtual_ipaddress {
       172.16.1.15/24
   }
unicast_src_ip 172.16.1.12  # This node
   unicast_peer {
   172.16.1.10          # Other nodes
   172.16.1.11          # Other nodes
   }
track_script {
       chk_haproxy
   }
}

Note:

172.16.1.10 adalah IP control-plane-1

172.16.1.11 adalah IP control-plane-2

172.16.1.12 adalah IP control-plane-3

Restart KeepAlived dan cek VRRP

# Restart keepalived service
$ sudo systemctl restart keepalived

# Check VRRP
$ ip a | grep ens33
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
   inet 172.16.1.10/24 brd 172.16.1.255 scope global ens33
   inet 172.16.1.15/24 scope global secondary ens33

Referensi:

https://medium.com/@kemalozz/installation-of-haproxy-and-keepalived-for-high-availability-f1d6e7b8982a

Setup HAProxy as LoadBalancer

Konfigurasi HAProxy untuk menjadi endpoint control plane

Jalankan di semua Node Control Plane

Edit file konfigurasi haproxy dan tambahkan line dibawah

# add these lines below the default configuration
frontend control-plane-frontend
   bind :8443
   mode tcp
   option tcplog
   default_backend control-plane-backend

backend control-plane-backend
   mode tcp
   option tcp-check
   balance roundrobin
   default-server inter 10s downinter 5s
   server control-plane-1 172.16.1.10:6443 check
   server control-plane-2 172.16.1.11:6443 check
   server control-plane-3 172.16.1.12:6443 check

Restart service haproxy
```
$ sudo systemctl restart haproxy
```

Referensi:

https://facsiaginsa.com/kubernetes/haproxy-for-k8s-load-balancer

Kubernetes HA Installation

Dalam provisioning sebuah cluster kubernetes diperlukan 3 tool yang diperlukan disetiap node, yakni kubeadm, kubelet dan kubectl

Jalankan di semua Node

Download public key untuk repository kuberentes serta tambahkan apt repository kuberentes

# Download public signing key kubernetes
$ curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.31/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

# Add Kubernetes apt repository
$ echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.31/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

Update Repository dan Install semua tools

# Update current repository packages
$ sudo apt-get update

# Install kubectl, kubelet and kubeadm
$ sudo apt-get install kubectl kubelet kubeadm -y

# Prevent the package for being automatically updated
$ sudo apt-mark hold kubectl kubeadm kubelet

Enable on boot service kubelet
```
$ sudo systemctl enable --now kubelet
```

Initializing High Availability Kubernetes Cluster

Pastikan semua node bisa mengakses registry.k8s.io dan matikan swap pada setiap node.

Note: Disemua node tambahkan static dns vrrp.k8s.local yang mengarah ke VIP. Sesuaikan dengan network yang dimiliki

Jalankan di control-plane-1

Initialize control-plane-1 untuk membuat cluster

$ sudo kubeadm init --control-plane-endpoint "vrrp.k8s.local:8443" --upload-certs

Tunggu hingga proses pulling image dan deployment selesai lalu copy token dari hasil output yang ada dan simpan terlebih dahulu (bisa dinotepad)

# Token for join as control plane
kubeadm join vrrp.k8s.local:8443 --token y2016n.rmxrx8w7me8sklgs \
       --discovery-token-ca-cert-hash sha256:0f8370aa97282917309491be6505bc20f38c7c31669defadb1f9060dd5fb519b \
       --control-plane --certificate-key 89f51654a4a24ab79723c29685ff959dac4e6635ea626e07b09425e90e637c19

# Token for join as workernode
kubeadm join vrrp.k8s.local:8443 --token y2016n.rmxrx8w7me8sklgs \
       --discovery-token-ca-cert-hash sha256:0f8370aa97282917309491be6505bc20f38c7c31669defadb1f9060dd5fb519b

Copy kubeconfig untuk non-root user

# RUN as NON-ROOT USER
$ mkdir -p $HOME/.kube
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config

Verifikasi dengan list node dalam cluster

$ kubectl get nodes
NAME              STATUS     ROLES           AGE   VERSION
control-plane-1   NotReady   control-plane   21m   v1.31.1

Jalankan di control-plane-1 dan control-plane-2

Masukan token untuk control plane dengan kubeadm

$ sudo kubeadm join vrrp.k8s.local:8443 --token y2016n.rmxrx8w7me8sklgs --discovery-token-ca-cert-hash sha256:0f8370aa97282917309491be6505bc20f38c7c31669defadb1f9060dd5fb519b --control-plane --certificate-key 89f51654a4a24ab79723c29685ff959dac4e6635ea626e07b09425e90e637c19

Copy kubeconfig untuk non-root user

# RUN as NON-ROOT USER
$ mkdir -p $HOME/.kube
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config

Verifikasi dengan list node dalam cluster

$ kubectl get nodes
NAME              STATUS     ROLES           AGE   VERSION
control-plane-1   NotReady   control-plane   34m     v1.31.1
control-plane-2   NotReady   control-plane   111s    v1.31.1
control-plane-3   NotReady   control-plane   2m7s    v1.31.1

Jalankan di semua Worker Node

Masukan token untuk worker node dengan kubeadm

$ sudo kubeadm join vrrp.k8s.local:8443 --token y2016n.rmxrx8w7me8sklgs --discovery-token-ca-cert-hash sha256:0f8370aa97282917309491be6505bc20f38c7c31669defadb1f9060dd5fb519b

Verifikasi di node control plane

$ kubectl get nodes
NAME              STATUS     ROLES           AGE     VERSION
control-plane-1   NotReady   control-plane   34m     v1.31.1
control-plane-2   NotReady   control-plane   111s    v1.31.1
control-plane-3   NotReady   control-plane   2m7s    v1.31.1
worker-1          NotReady   <none>          8m22s   v1.31.1
worker-2          NotReady   <none>          8m27s   v1.31.1

Referensi:

Setup Calico as CNI

CNI atau Container Network Interface adalah sebuah daemon dalam node untuk menyediakan CRI Services untuk kubelet. Ini digunakan untuk memuat plugin CNI agar bisa mengimplementasikan network model kubernetes. CNI bertugas untuk menelola pengalamatan IP, routing network, lain2. Salah satu plugin CNI yang banyak digunakan adalah Calico.

Jalankan disalah satu control plane

Download manifest dari repo resmi calico

# Download calico networking manifest for the Kubernetes API
$ curl https://raw.githubusercontent.com/projectcalico/calico/v3.28.1/manifests/calico.yaml -O

Apply manifest
```
$ kubectl apply -f calico.yaml
```

Tunggu hingga selesai, dan cek status Pod

# Watch calico deployment progress
$ watch kubectl get all -A

# or check pod status
$ kubectl get pod -A
NAMESPACE     NAME                                       READY   STATUS    RESTARTS   AGE
kube-system   calico-kube-controllers-7fbd86d5c5-tllzj   1/1     Running   0          12m
kube-system   calico-node-5mxws                          1/1     Running   0          12m
kube-system   calico-node-95t2f                          1/1     Running   0          12m
kube-system   calico-node-gs4cj                          1/1     Running   0          12m
kube-system   calico-node-m2ztj                          1/1     Running   0          12m
kube-system   coredns-6f6b679f8f-42zgx                   1/1     Running   0          89m
kube-system   coredns-6f6b679f8f-zt789                   1/1     Running   0          89m

Cek status Node

$ kubectl get nodes
NAME              STATUS   ROLES           AGE   VERSION
control-plane-1   Ready    control-plane   58m   v1.31.1
control-plane-2   Ready    control-plane   25m   v1.31.1
control-plane-3   Ready    control-plane   25m   v1.31.1
worker-1          Ready    worker          31m   v1.31.1
worker-2          Ready    worker          31m   v1.31.1

Referensi:

https://docs.tigera.io/calico/latest/getting-started/kubernetes/self-managed-onprem/onpremises

Add label to Node

Tambahkan Label type=app & node-role.kubernetes.io/worker= di worker node

Jalankan di salah satu control plane

Untuk menambahkan label pada node gunakan kubectl label nodes lab

# Add label to all worker node 
$ kubectl label nodes worker-1 type=app
$ kubectl label nodes worker-1 node-role.kubernetes.io/worker=1

Ganti worker-1 dan worker=1 dengan worker 2 untuk label node worker-2

Cek label yang sudah ditambahkan

$ kubectl get nodes --show-labels | grep worker-1
worker-1          NotReady   worker          28m   v1.31.1   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=worker-1,kubernetes.io/os=linux,node-role.kubernetes.io/worker=1,type=app

Referensi:

https://kubernetes.io/docs/tasks/configure-pod-container/assign-pods-nodes/

diotriandika commented 1 week ago

Step 2 - Deployment Additional Applications

Tasks:

[x] Sebar pod CoreDNS hanya di master node
[x] Deploy metric-server
[x] Deploy Kubernetes dashboard
[x] Deploy Ingress nginx
[x] Buat directory workdir pada homedir untuk menyimpan semua manifest

Re-Deploy CoreDNS Pods

Untuk mendeploy CoreDNS hanya pada Master Node, diperlukan matching taints dan tolerations.

Jalankan di Node Control Plane

Cek taint dinode control-plane

$ kubectl describe nodes control-plane-1 | grep Taints
Taints: node-role.kubernetes.io/control-plane:NoSchedule

Tambahkan value true di key node-role.kuberenetes.io/control-plane pada semua node control plane

# Modify taint control plane 1
$ kubectl taint nodes control-plane-1 node-role.kubernetes.io/control-plane=true:No
Schedule --overwrite=true
node/control-plane-1 modified

# do the same thing with the other contol planes

Ubah manifest deployment coredns kemudian cari section spec.nodeSelector lalu sesuaikan label dan taints node control plane.

$ kubectl edit deployment coredns -n kube-system

...
     nodeSelector:
       node-role.kubernetes.io/control-plane: ""            <---- add
     priorityClassName: system-cluster-critical
     restartPolicy: Always
     schedulerName: default-scheduler
     securityContext: {}
     serviceAccount: coredns
     serviceAccountName: coredns
     terminationGracePeriodSeconds: 30
     tolerations:
     - key: CriticalAddonsOnly
       operator: Exists
     - effect: NoSchedule                                   <---- add
       key: node-role.kubernetes.io/control-plane           <---- add
       operator: Equal                                      <---- add
       value: "true"                                        <---- add

Restart deploment coredns dan cek status Pod

# Restart Deployment
$ kubectl rollout restart deployment coredns -n kube-system
deployment.apps/coredns restarted

# Check coredns Pod status
$ kubectl get pod -n kube-system -o wide | grep coredns
coredns-5c65444476-j6299                   1/1     Running   0          14m    192.168.201.193   control-plane-3   <none>           <none>
coredns-5c65444476-rhfrx                   1/1     Running   0          14m    192.168.86.129    control-plane-2   <none>           <none>

Referensi:

Deploying Metric Server

Metrics server sebuah resource metrics monitoring tool untuk kubernetes. Metrics server disini digunakan untuk mengukur penggunaan CPU dan Memory Pods atau Nodes dalam cluster. Metrics server juga bersifat scalable dan efisien.

Jalankan di Node Control Plane

Download manifest lalu edit

# Download metrics-server manifest
$ wget https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/high-availability-1.21+.yaml

# Edit metrics-server manifest
$ nano high-availability-1.21+.yaml

Di bagian deployment metrics-server, tambahkan args baru yakni --kubelet-insecure-tls untuk mendisable certificate verification.

               k8s-app: metrics-server
           namespaces:
           - kube-system
           topologyKey: kubernetes.io/hostname
     containers:
     - args:
       - --cert-dir=/tmp
       - --secure-port=10250
       - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
       - --kubelet-use-node-status-port
       - --metric-resolution=15s
       - --kubelet-insecure-tls     <<--- add this

Apply manifest & cek status deployment metrics-server

# Apply manifest
$ kubectl apply -f high-availability-1.21+.yaml

# Check metrics server deployment
$ kubectl get deployments -n kube-system
NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
calico-kube-controllers   1/1     1            1           124m
coredns                   2/2     2            2           169m
metrics-server            2/2     2            2           18m

Verifikasi dengan menjalankan kubectl top

# Check nodes usage
$ kubectl top nodes
NAME              CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
control-plane-1   148m         7%     1295Mi          70% 
control-plane-2   125m         6%     994Mi           54%
control-plane-3   135m         6%     1009Mi          55%
worker-1          68m          3%     908Mi           49%
worker-2          60m          3%     827Mi           45%

# Check pods usage
$ kubectl top pods -A
NAMESPACE     NAME                                       CPU(cores)   MEMORY(bytes)   
kube-system   calico-kube-controllers-7fbd86d5c5-2bzzc   3m           12Mi            
kube-system   calico-node-4l5m5                          40m          125Mi           
kube-system   calico-node-fghjs                          38m          129Mi           
kube-system   calico-node-m9jmx                          35m          119Mi           
kube-system   calico-node-pkr4h                          38m          135Mi           
kube-system   calico-node-qg97m                          39m          116Mi

Referensi:

Deploying NGINX Ingress

Ingress adalah sebuah objek di Kubernetes yang mendefinisikan rule untuk mengarahkan traffic HTTP/HTTPS ke service dalam cluster. Ingress menyediakan cara untuk mengontrol akses ke service berdasarkan URL, host atau jalur tertentu.

Jalankan di Node Control Plane

Tambahkan repository nginx helm repository

# Add nginx repos
$ helm repo add nginx-stable https://helm.nginx.com/stable

# Update helm repos
$ helm repo update

Install NGINX Ingress Controller

$ helm install nginx-ingress nginx-stable/nginx-ingress --set rbac.create=true

Validasi NGINX Ingress controller

$ kubectl get pod -l app.kubernetes.io/name=nginx-ingress
NAME                                        READY   STATUS    RESTARTS   AGE
nginx-ingress-controller-7c48d7568f-8gvs4   1/1     Running   0          8m24s

Cek Port yang diexpose oleh NGINX Ingress

$ kubectl get svc
NAME                       TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
kubernetes                 ClusterIP      10.96.0.1       <none>        443/TCP                      8h
nginx-ingress-controller   LoadBalancer   10.97.35.127    <pending>     80:31053/TCP,443:32024/TCP   3h46m

Bisa dilihat nginx-ingress-controller mengekspos port 31053 untuk HTTP dan 32024 untuk HTTPS

Bind NodePort yang diexpose oleh ingress di HAProxy

## Execute in all nodes
$ sudo nano /etc/haproxy/haproxy.cfg

## Add these lines below the first backend configuration
frontend ingress-http
       bind *:80
       mode tcp
       default_backend ingress-http_backend

backend ingress-http_backend
       mode tcp
       balance roundrobin
       server control-plane-1 172.16.1.10:31053 check
       server control-plane-2 172.16.1.11:31053 check
       server control-plane-3 172.16.1.12:31053 check

Restart HAproxy
```
$ sudo systemctl restart haproxy
```

Referensi:

Deploying Kuberentes Dashboard

Kubernetes menyediakan sebuah web-based dashboard yang dapat digunakan untuk membuat, monitor hingga memanage sebuah cluster.

Jalankan di Node Control Plane

Install Helm untuk mempermudah instalasi Kubernetes Dashboard.

# Install Helm
$ curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

# Verify installation
$ helm version
version.BuildInfo{Version:"v3.16.1"

Tambahkan Kubernetes Dashboard helm repository.

$ helm repo add kubernetes-dashboard https://kubernetes.github.io/dashboard/

Deploy chart kuberentes dashboard dengan nama kubernetes-dashboard beserta namespacenya.
```
$ helm upgrade --install kubernetes-dashboard kubernetes-dashboard/kubernetes-dashboard --create-namespace --namespace kubernetes-dashboard
```
Secara default k8s dashboard web type servicenya adalah ClusterIP, yang berarti hanya bisa diakses didalam cluster saja. Maka dari itu kita perlu mengekspos menggunakan NodePort atau Ingress.

Rubah service kubernetes-dashboard-kong-proxy ke NodePort

# Create a file named values.yaml with the following content
$ cat << EOF | tee values.yaml
kong:
 proxy:
   type: NodePort
 http:
   enabled: true
EOF

Upgrade helm release kubernetes dashboard dengan file yaml sebelumnya.

$ helm upgrade kubernetes-dashboard kubernetes-dashboard/kubernetes-dashboard -f values.yaml -n kubernetes-dashboard

Cek service kubernetes-dashboard-kong-proxy dan expose portnya

$ kubectl get svc -n kubernetes-dashboard
NAME                                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                         AGE
kubernetes-dashboard-api               ClusterIP   10.104.2.57      <none>        8000/TCP                        13m
kubernetes-dashboard-auth              ClusterIP   10.102.227.161   <none>        8000/TCP                        13m
kubernetes-dashboard-kong-manager      NodePort    10.111.242.82    <none>        8002:31487/TCP,8445:32011/TCP   13m
kubernetes-dashboard-kong-proxy        NodePort    10.98.122.251    <none>        443:30183/TCP                   13m
kubernetes-dashboard-metrics-scraper   ClusterIP   10.107.59.165    <none>        8000/TCP                        13m
kubernetes-dashboard-web               ClusterIP   10.108.251.243   <none>        8000/TCP                        13m

Coba akses ke web browser dengan format https://<virtual-ip>:<exposed-port

Untuk dapat login, kita memerlukan sebuah bearer token yang dapat digenerate melalui sebuah service account. Untuk itu buat manifest untuk user account yang akan digunakan lalu apply.

$ nano admin-dashboard.yaml

## Define ServiceAccount
apiVersion: v1 
kind: ServiceAccount
metadata:  
 name: admin-user  
 namespace: kubernetes-dashboard

---
## Define ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:  
 name: admin-user
roleRef:
 apiGroup: rbac.authorization.k8s.io  
 kind: ClusterRole  
 name: cluster-admin
subjects: 
 - kind: ServiceAccount  
   name: admin-user  
   namespace: kubernetes-dashboard

---
## Define Secret
apiVersion: v1
kind: Secret
metadata:  
 name: admin-user  
 namespace: kubernetes-dashboard  
 annotations:    
   kubernetes.io/service-account.name: "admin-user"
type: kubernetes.io/service-account-token

$ kubectl apply -f admin-dashboard.yaml 
serviceaccount/admin-user created
clusterrolebinding.rbac.authorization.k8s.io/admin-user created
secret/admin-user created

Decode access token dari secret yang sudah dibuat.

$ kubectl get secret admin-user -n kubernetes-dashboard -o jsonpath={".data.token"} | base64 -d
eyJhbGciOiJSUzI1NiIsImtpZCI6IkRMV2NDQmtBWVo4X2hvaF9Ianhab1h5ZGJIblNSc2lfbU5FSFdDbEdpLWcifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJhZG1pbi11c2VyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImFkbWluLXVzZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiIwZTlkNGI5ZS00MDRiLTRjMWEtYmIyNi04YTk3MWQzY2YyNTgiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZXJuZXRlcy1kYXNoYm9hcmQ6YWRtaW4tdXNlciJ9.vO4ninblQTK8p6oDebZhK-WN9UlfM0miOLucYhqnjBOr1KDwiTJ0TpuPCt0ypsZ_thHMvwVerp32o2VkeQavP14_ng_mWRPwAHt5EqxX59rahNvhz5r2f4dmLnqlClEr7qdCmBpmkjsiUXgEw2sC2El4mvT2VMjGc64pfnfd3vHtZ0XukN3EL-PJJwb2rqBjX0C9vg6n6RNffb7ZvWwk1voISC_bAqBU08WZ7RjgVh2RrkXXoplLLje97UZT81509QTt0OwmUd_hM-XoUz8tBhutIHGyP19olK-dhxzb1N3XlaVTj_D0cf2swYGXPu9ejyuaJMaD8oINyU1EA8VzVw

akan muncul random text seperti diatas, ini adalah bearer token dari admin-user.

Gunakan token diatas untuk login ke dashboard

Referensi:

Creating Workdir to Store Manifests

Membuat workdir (working directory) bisa memudahkan kita untuk memangemen semua file manifest yang dibuat.

Jalankan di Node Control Plane

Buat working directory dengan mkdir
```
$ mkdir $HOME/workdir
```
Pindahkan semua manifest yang ada ke workdir
```
$ mv *.yaml $HOME/workdir
```

Verifikasi dengan melist semua file manifest

$ ls -l
total 276
-rw-rw-r-- 1 lnearher lnearher    610 Sep 21 09:09 admin-dashboard.yaml
-rw-rw-r-- 1 lnearher lnearher 253923 Sep 21 05:52 calico.yaml
-rw-rw-r-- 1 lnearher lnearher    351 Sep 21 13:04 dashboard-ingress.yaml
-rw-rw-r-- 1 lnearher lnearher   4876 Sep 21 07:53 high-availability-1.21+.yaml
-rw-rw-r-- 1 lnearher lnearher    356 Sep 21 13:21 ingress-dash.yaml
-rw-rw-r-- 1 lnearher lnearher    677 Sep 21 12:55 test-pod.yaml
-rw-rw-r-- 1 lnearher lnearher     60 Sep 21 09:45 values.yaml

Referensi:

https://www.redhat.com/sysadmin/create-delete-files-directories-linux

diotriandika commented 1 week ago

Step 3 - Deployment Nginx App

[x] Buat deployment dengan image nginx:1.19 dengan nama btech-testpod-<your_name> dengan 2 replika
[x] Buat SVC dengan type ClusterIP
[x] Buat ingress dan rewrite ke path /testpath
[ ] Test hit ingress dengan curl menggunakan while true dan sleep selama 1 detik (jalankan di tmux)
[x] Cek log deployment test dan simpan ke file di directory workdir
[x] Scale deployment test ke 1 replika

Create Nginx Deployment

Deployment adalah sebuah object yang digunakan untuk mengelola Pod dengan lebih fleksibel karena deployment mendefinisikan dan mengelola lifecycle dari Pod dan memastikan bahwa aplikasi berjalan sesuai dengan yang diinginkan atau desired state.

Jalankan di Node Control Plane

Buat manifest deployment nginx bisa menggunakan kubectl create

$ kubectl create deployment btech-testpod-lnearher --image=nginx:1.19 --replicas=2 --port=80 --dry-run -o yaml > btech-testpod-lnearher.yaml

Rapikan manifest

apiVersion: apps/v1
kind: Deployment
metadata:
 labels:
   app: btech-testpod-lnearher
 name: btech-testpod-lnearher
spec:
 replicas: 2
 selector:
   matchLabels:
     app: btech-testpod-lnearher
 template:
   metadata:
     labels:
       app: btech-testpod-lnearher
   spec:
     containers:
     - image: nginx:1.19
       name: nginx
       ports:
       - containerPort: 80

Apply manifest dan cek status deployment

## Deploy Pods
$ kubectl apply -f btech-testpod-lnearher.yaml

## Check deployment status
$ kubectl get deployment
NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
btech-testpod-lnearher     2/2     2            2           14m

Referensi:

https://kubernetes.io/docs/concepts/workloads/controllers/deployment/

Create ClusterIP Service

Service tipe ClusterIP mengekspos service didalam sebuah cluster-internal IP, yang dimana ini membuat aplikasi yang menggunakan service ini hanya bisa diakses dalam cluster saja.

Jalankan di Node Control Plane

Buat manifest service dengan kubectl create

$ kubectl create service clusterip btech-svc --tcp=8080:80 --dry-run -o yaml > btech-testsvc-lnearher.yaml

Rapikan manifest dan ubah selector app agar sesuai dengan label Pod

apiVersion: v1
kind: Service
metadata:
 labels:
   app: btech-svc
 name: btech-svc
spec:
 ports:
 - name: btech-pod
   port: 8080
   protocol: TCP
   targetPort: 80
 selector:
   app: btech-testpod-lnearher
 type: ClusterIP

Apply manifest lalu cek status service

## Deploy Service
$ kubectl apply -f btech-testsvc-lnearher.yaml

## Check the service
$ kubectl get svc
NAME                       TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
btech-svc                  ClusterIP      10.101.209.115   <none>        8080/TCP                     3s

Referensi:

https://kubernetes.io/docs/concepts/services-networking/service/#type-clusterip

Expose Pod and Rewrite Path with NGINX Ingress

Untuk dapat mengekspos service ke external kita memerlukan yang namanya Ingress. Ingress mengekspos rute HTTP dan HTTPS dari luar cluster ke service didalam cluster. Ingress yang sering digunakan adalah NGINX Ingress dan Kong Ingress. Kita juga bisa melakukan rewrite path dengan nginx ingress dengan memanfaatkan annotations nginx.ingress.kubernetes.io/rewrite-target.

Jalankan di Node Control Plane

Buat ConfigMap terlebih dahulu karena kita memerlukan pod nginx memiliki site dipath /testpath

apiVersion: v1
kind: ConfigMap
metadata:
 name: nginx-conf
data:
 nginx.conf: |
   worker_processes 1;

   events {
       worker_connections 1024;
   }

   http {
     include       mime.types;
     default_type application/octet-stream;

     sendfile        on;
     keepalive_timeout  65;

     server {
       listen       80;
       server_name  localhost;

       location /testpath {
         alias /var/www/testpath/;
         index index.html;
       }

       location / {
         root /usr/share/nginx/html/;
         index index.html;
       }

       error_page 404 /404.html;
         location = /404.html {
           root /usr/share/nginx/html;
       }
     }
   }

 index.html: |
   this is btehc /testpath test

Test Ingress and Deployment Logging

Melakukan test hit inggress menggunakan curl dan simpan log dari deployment

Jalankan di Node Control Plane

Tes hit ingress dengan menggunakan while true dan sleep dalam rentang waktu 1 detik melalui tmux

# Create tmux session
$ tmux new-session -s curl-test

# Loop curl
$ while true; do curl http://pod.btech.local/testpath/; sleep 1; done

Detach dari session tmux lalu simpan log ke workdir

## Detach tmux session
--> Hold CTRL and B, release then click D

## Save deployment logs into a file di workdir
$ kubectl logs deployment/btech-testpod-lnearher > workdir/btech-pod_logs.log

Referensi:

Scale down Deployment

kubectl scale dapat digunakan untuk menentukan scale baru dari deployment, replicasets, replication controller atau statefulsets

Jalankan di Node Control Plane

Scale deployment

$ kubectl scale --replicas=1 deployment/btech-testpod-lnearher

Verifikasi replica deployment
```
$ kubectl get deployment
```

Referensi:

https://kubernetes.io/docs/reference/kubectl/generated/kubectl_scale/

diotriandika / learn-kubernetes

Task Provisioning Kubernetes HA Multi Control-Plane and Monitoring Integration #1

Specs

Todo

Step 1 - Provisioning Cluster

Setup Containerd as CRI

Setup KeepAlived as VRRP

Setup HAProxy as LoadBalancer

Kubernetes HA Installation

Initializing High Availability Kubernetes Cluster

Setup Calico as CNI

Add label to Node

Step 2 - Deployment Additional Applications

Re-Deploy CoreDNS Pods

Deploying Metric Server

Deploying NGINX Ingress

Deploying Kuberentes Dashboard

Creating Workdir to Store Manifests

Step 3 - Deployment Nginx App

Create Nginx Deployment

Create ClusterIP Service

Expose Pod and Rewrite Path with NGINX Ingress

Test Ingress and Deployment Logging

Scale down Deployment