Host Site From Home

[anitsh]

TODO:

• [x] Host a simple site from home.
• [ ] Setup https server from home.
• [ ] Setup up analytics for the system.
• [x] Containerize the setup.

RESOURCES: https://www.digitalocean.com/community/questions/can-i-run-two-domain-on-single-ip-address https://support.comodo.com/index.php?/Knowledgebase/Article/View/304/17/if-i-provide-two-domain-names-for-the-same-server-ip-address-can-i-get-two-different-ssl-certificates-one-for-each-domain-name

• [ ] https://github.com/thadd/nuc-cluster Building a Light-weight OpenShift Cluster
• [ ] https://blog.container-solutions.com/bootstrap-as-code-part-1-hardware-is-here-to-stay Setup home system
• [ ] https://www.digitalocean.com/community/tutorials/how-to-configure-bind-as-a-caching-or-forwarding-dns-server-on-ubuntu-14-04

[anitsh]

Basic implementation is done. Steps taken:

• At home router, port forwarded all the requests to from 80 to one of the computer.
• At the domain register, mapped the domain to the public IP.
• Added 'A' name record in the domain registrar. Why? An A record connects your domain name to an IP address and lets web browsers find your website. An A record to point to your hosting account or to create a subdomain. https://www.name.com/support/articles/205188538-Pointing-your-domain-to-hosting-with-A-records
• Removed default CNAME redirect to the domain registrar.

Todo:

• [ ] Setup https to serve the content from Docker
• [ ] https://letsencrypt.org/docs/certificates-for-localhost
• [ ] https://www.freecodecamp.org/news/docker-nginx-letsencrypt-easy-secure-reverse-proxy-40165ba3aee2
• [ ] https://medium.com/faun/docker-letsencrypt-dns-validation-75ba8c08a0d
• [ ] https://www.digitalocean.com/community/tutorials/how-to-secure-a-containerized-node-js-application-with-nginx-let-s-encrypt-and-docker-compose
• [ ] https://medium.com/better-programming/docker-powered-web-development-utilizing-https-and-local-domain-names-a57f129e1c4d
• [ ] https://hackersandslackers.com/caddy-webserver-for-docker-etc
• [ ] Should I run a DNS server from home? DNS is not configured yet. I am not sure if that is needed so need to do some more research on that.
  https://www.wired.com/2010/02/set_up_a_dns_name_server https://opensource.com/article/17/4/build-your-own-name-server https://blog.cpanel.com/how-to-create-your-own-name-servers https://blog.resellerclub.com/how-to-host-a-domain-website-on-your-own

Tools:

• https://github.com/FiloSottile/mkcert
• https://certbot.eff.org/lets-encrypt/ubuntubionic-nginx

[anitsh]

Came to know about Traefik which automates the HTTPS and other processes to run website from locally. Created repo for practice.

[anitsh]

Came to know about Traefik. It seems to automate the all alot of things including HTTPS management with LetsEncrypt.

https://github.com/codeanit/traefik-impl

[anitsh]

Tried to implement from another source

• https://medium.com/@containeroo/traefik-2-0-docker-a-simple-step-by-step-guide-e0be0c17cfa5
• https://medium.com/@containeroo/traefik-2-0-docker-an-advanced-guide-d098b9e9be96

Could not make it workable and the last error is too many failed authorizations recently As this is for development purposes, Let'sEncrypt suggets to use Staging environment.

• https://letsencrypt.org/docs/rate-limits
• https://letsencrypt.org/docs/staging-environment

Added a subdomain. There was issue with authorization and later came to find the ufw had not opened any ports. So opened ports 80 and 443.

I think I need to look into Traefik documents a bit more.

Another simple solution could be using Caddy server https://www.digitalocean.com/community/tutorials/how-to-host-a-website-with-caddy-on-ubuntu-18-04.

[anitsh]

https://docs.traefik.io/v2.0/user-guides/docker-compose/basic-example for basics on docker with Tarefik.

[anitsh]

Docker with LetsEncrypt and NGINX

https://medium.com/@pentacent/nginx-and-lets-encrypt-with-docker-in-less-than-5-minutes-b4b8a60d3a71 https://gist.github.com/ezhulkov/355fff964e54ffed541ed09e80341fd0 https://github.com/wmnnd/nginx-certbot https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-ubuntu-18-04 https://docs.docker.com/compose/install https://github.com/wmnnd/nginx-certbot

[anitsh]

https://www.darkreading.com/network-and-perimeter-security/most-home-routers-are-full-of-vulnerabilities/d/d-id/1332987

[anitsh]

Some tunneling solution

• [ ] https://github.com/anderspitman/awesome-tunneling
• [ ] https://derrick.blog/2020/01/26/open-source-ngrok-alternative

[anitsh]

Cups seems to be running. Stopped and disabled.

sudo systemctl stop cups
sudo systemctl disable cups

CUPS - Print Server The primary mechanism for Ubuntu printing and print services is the Common UNIX Printing System (CUPS). This printing system is a freely available, portable printing layer which has become the new standard for printing in most Linux distributions.

CUPS manages print jobs and queues and provides network printing using the standard Internet Printing Protocol (IPP), while offering support for a very large range of printers, from dot-matrix to laser and many in between. CUPS also supports PostScript Printer Description (PPD) and auto-detection of network printers, and features a simple web-based configuration and administration tool.

[anitsh]

Ran a multi-node Kubernetes cluster.

Control-plane node Protocol	Direction	Port Range	Purpose	Used By
TCP	Inbound	6443*	Kubernetes API server	All
TCP	Inbound	2379-2380	etcd server client API	kube-apiserver, etcd
TCP	Inbound	10250	Kubelet API	Self, Control plane
TCP	Inbound	10251	kube-scheduler	Self
TCP	Inbound	10252	kube-controller-manager	Self

Worker node Protocol	Direction	Port Range	Purpose	Used By
TCP	Inbound	10250	Kubelet API	Self, Control plane
TCP	Inbound	30000-32767	NodePort Services†	All

Coredns pods were running but were not ready. Logged message was [INFO] plugin/ready: Still waiting on: "kubernetes"

So I deleted the pods, and the pods were running fine. kubectl get pods -n kube-system -oname |grep coredns |xargs kubectl delete -n kube-system

I think the other way would be just to redeploy CoreDNS pods kubectl rollout restart -n kube-system deployment/coredns

kubectl logs --tail=100 -f Tail last 100 lines kubectl logs -f

https://github.com/kubernetes/kubernetes/blob/master/cluster/kube-down.sh https://stackoverflow.com/questions/37016546/kubernetes-how-do-i-delete-clusters-and-contexts-from-kubectl-config https://linkerd.io/2.10/tasks/exposing-dashboard https://linkerd.io/2.10/tasks/troubleshooting

To confirm things are working.

The simplest way to test our setup is to create Apache HTTP Server with a pod http-page and expose it via service named http-service with port 80 and NodePort type:

kubectl run http-page --image=httpd --port=80 kubectl expose pod http-page --name=http-service --port=80 --type=NodePort

Verify that it works: kubectl get service http-service kubectl get pods http-page -o wide

Set host IP address to expose it from outerworld. kubectl patch svc kubernetes -p '{"spec":{"externalIPs":["10.2.8.19"]}}' kubectl exec --stdin --tty http-page -- /bin/bash

Run other workload: kubectl create deployment nginx --image=nginx kubectl expose deploy nginx --port 80 --target-port 80 --type NodePort

kubectl exec --stdin --tty ngninx -- sh

And other workload with deployment and add service to it: kubectl apply -f https://k8s.io/examples/service/load-balancer-example.yaml // Deployment name is hello-world kubectl expose deployment hello-world --type=LoadBalancer --name=my-service kubectl scale --replicas=1 deployment my-service kubectl delete services my-service kubectl delete deployment hello-world

kubectl get pods --all-namespaces -o=jsonpath='{range .items[]}{"\n"}{.metadata.name}{":\t"}{range .spec.containers[]}{.image}{", "}{end}{end}' | sort : List container images by pods

Some commands: kubectl get pods kubectl delete pod & kubectl get services kubectl delete service nginx kubectl get deployments kubectl delete deployment nginx kubectl rollout restart deployment [deployment_name] kubectl exec POD_NAME -c CONTAINER_NAME /sbin/killall5 - There are cases when you want to restart a specific container instead of deleting the pod and letting Kubernetes recreate it kubectl scale deployment <> --replicas=0 -n service kubectl exec -it [POD_NAME] -c [CONTAINER_NAME] -- /bin/sh -c "kill 1"

kill -15 5 you're running the kill command to send signal "-15" to the process with the PID 5. This is how you tell a process that you would like for it to terminate (SIGTERM) and have it take the time to clean up any opened resources (temp files, rollback db transactions, close connections, whatever). Contrasted with -9 (SIGKILL), kills the process immediately, not allowing it to clean up any opened resources.

Resource

• https://stackoverflow.com/questions/46123457/restart-container-within-pod
• https://kubernetes.io/docs/tasks/access-application-cluster/list-all-running-container-images

[anitsh]

Setup Linkerd

Installed and enabled dashboard linkerd viz install | kubectl apply -f -

Tried to access the dashboard but it's only accessible in the admin server. And it is not accessible from the admin server's IP. So listed the pods, the dashboard services runs on web pod. Then checked if there are any services that exposes it.

NAMESPACE     NAME                        TYPE           CLUSTER-IP       EXTERNAL-IP     PORT(S)                  AGE
linkerd-viz   web                         ClusterIP      10.98.184.167    <none>          8084/TCP,9994/TCP        170m

There was no external IP set, so set it with the admin's IP. While trying to access it, gave error:
It appears that you are trying to reach this service with a host of 'IP:8084'.
This does not match /^(localhost|127\.0\.0\.1|web\.linkerd-viz\.svc\.cluster\.local|web\.linkerd-viz\.svc|\[::1\])(:\d+)?$/ and has been denied for security reasons.
Please see https://linkerd.io/dns-rebinding for an explanation of what is happening and how to fix it.

The page suggests redeployment of deployment - https://linkerd.io/2.10/tasks/exposing-dashboard/#tweaking-host-requirement

So changed the config

1. Check : kubectl get deployment web -n linkerd-viz -o=json | jq '.spec.template.spec.containers[0].args'
2. Get the deployment config: kubectl get deployment web -n linkerd-viz -o json > linkerd-dashboard.json
3. Update host config by adding IP address
4. Applied the changes: kubectl apply -f linkerd-dashboard.json
5. Set external with the same IP added in the host config: kubectl patch -n linkerd-viz svc web -p '{"spec":{"externalIPs":["192.168.1.100"]}}'
6. Access it : IP:8084

My previous resources(pods, deployments and services, and default cluster) were not injected. So injected Linkerd: kubectl get deploy -o yaml | linkerd inject - | kubectl apply -f -

To test few things generated load with ab. ab -n 100000 -c 10 -H "Accept-Encoding: gzip, deflate; Hello:world" -rk http://IP/

-n: Number of requests
-c: Number of concurrent requests
-H: Add header
—r: flag to not exit on socket receive errors
-k: Use HTTP KeepAlive feature
-p: File containing data to POST
-T: Content-type header to use for POST/PUT data,

Resource

https://linkerd.io/2.10/reference/cli/viz https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#updating-a-deployment https://stackoverflow.com/questions/49860181/how-do-i-update-the-args-for-a-kubernetes-deployment https://linkerd.io/2.10/reference/cli/inject https://diamantidis.github.io/2020/07/15/load-testing-with-apache-bench https://www.sumologic.com/blog/kubectl-logs https://medium.com/tegess/getting-started-with-linkerd-service-mesh-on-private-k8s-cloud-371b532f47c6

[anitsh]

Kubectl command cheatsheet

Command	Description
kubectl get	used to view and find resourcescan output JSON, YAML, or be directly formatted
kubectl describe	retrieve extra information about a resourcesneeds a resource type and (optionally) a resource name
kubectl create	create a resource from a file or standard input
kubectl edit	edit resource
kubectl replace	replace the existing resource with the updated reource
kubectl patch	patch the existing resource
kubectl delete	can delete resources across multiple namespaces
kubectl label	can add/remove/update labels across multiple namespaces
kubectl logs	view container logs for debugging
kubectl exec	execute a command in a running container

Issue: Linkerd Dashboard

Dashboard has some issue. Metrics and Tap Injector pods were failing.

NAME                            READY   STATUS             RESTARTS          AGE
grafana-5c47556496-pn4vw        2/2     Running            0                 16h
metrics-api-86b88fddc-jrdgb     0/1     CrashLoopBackOff   175 (3m34s ago)   10h
prometheus-554c6cc89f-wml4r     1/1     Running            0                 10h
tap-7948bf84c5-plkt6            2/2     Running            0                 16h
tap-injector-5fc98dc7c5-ct548   0/1     CrashLoopBackOff   175 (75s ago)     10h
web-dcb9c6975-cg5fs             2/2     Running            0                 12h

Tap log

Events:
Type     Reason     Age                   From     Message
----     ------     ----                  ----     -------
Warning  Unhealthy  41m (x1652 over 10h)  kubelet  Readiness probe failed: Get "http://10.244.0.18:9995/ready": dial tcp 10.244.0.18:9995: connect: connection refused
Normal   Created    26m (x170 over 10h)   kubelet  Created container tap-injector
Warning  Unhealthy  11m (x520 over 10h)   kubelet  Liveness probe failed: Get "http://10.244.0.18:9995/ping": dial tcp 10.244.0.18:9995: connect: connection refused
Warning  BackOff    84s (x2102 over 10h)  kubelet  Back-off restarting failed container

Metrics log

Events:
Type     Reason     Age                    From     Message
----     ------     ----                   ----     -------
Warning  Unhealthy  36m (x1677 over 10h)   kubelet  Readiness probe failed: Get "http://10.244.0.22:9995/ready": dial tcp 10.244.0.22:9995: connect: connection refused
Warning  Unhealthy  6m32s (x527 over 10h)  kubelet  Liveness probe failed: Get "http://10.244.0.22:9995/ping": dial tcp 10.244.0.22:9995: connect: connection refused
Warning  BackOff    85s (x2110 over 10h)   kubelet  Back-off restarting failed container

Ping works

PING 10.244.0.22 (10.244.0.22) 56(84) bytes of data.
64 bytes from 10.244.0.22: icmp_seq=1 ttl=64 time=0.194 ms
64 bytes from 10.244.0.22: icmp_seq=2 ttl=64 time=0.097 ms
64 bytes from 10.244.0.22: icmp_seq=3 ttl=64 time=0.095 ms

IP route default via 192.168.1.254 dev enp3s0 proto dhcp metric 100 10.244.0.0/24 dev cni0 proto kernel scope link src 10.244.0.1 10.244.1.0/24 via 10.244.1.0 dev flannel.1 onlink 169.254.0.0/16 dev cni0 scope link metric 1000 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 172.18.0.0/16 dev br-940c3874b064 proto kernel scope link src 172.18.0.1 linkdown 172.19.0.0/16 dev br-847829250b8c proto kernel scope link src 172.19.0.1 linkdown 172.20.0.0/16 dev br-6f507f240f54 proto kernel scope link src 172.20.0.1 linkdown 192.168.1.0/24 dev enp3s0 proto kernel scope link src 192.168.1.100 metric 100 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown

IP neigh 10.244.0.8 dev cni0 lladdr 22:e8:b1:87:f9:c1 REACHABLE 10.244.0.21 dev cni0 lladdr f6:3d:7a:45:8c:38 REACHABLE 10.244.0.18 dev cni0 lladdr e2:1a:a6:b3:02:f7 DELAY 192.168.1.64 dev enp3s0 lladdr 28:d2:44:f3:aa:98 REACHABLE 10.244.0.14 dev cni0 lladdr ee:05:a5:fb:f0:04 REACHABLE 10.244.0.11 dev cni0 lladdr 3a:9f:2a:bf:d2:4e REACHABLE 10.244.0.16 dev cni0 lladdr 96:aa:28:03:6d:bb REACHABLE 10.244.0.7 dev cni0 lladdr 2e:7d:d6:9e:0e:4e REACHABLE 10.244.0.12 dev cni0 lladdr 46:75:ae:95:79:b3 REACHABLE 10.244.0.9 dev cni0 lladdr de:99:d3:1b:2f:dd REACHABLE 10.244.0.22 dev cni0 lladdr b2:c6:55:3d:8f:c4 STALE 10.244.0.19 dev cni0 lladdr 56:96:b4:ba:1e:fa REACHABLE 35.224.170.84 dev flannel.1 FAILED 10.244.1.0 dev flannel.1 lladdr 22:27:1c:99:b6:52 PERMANENT 10.244.0.15 dev cni0 lladdr c2:5c:bc:c1:6b:75 REACHABLE 35.232.111.17 dev flannel.1 FAILED 10.244.0.20 dev cni0 lladdr be:8d:0b:f3:86:33 REACHABLE 10.244.0.17 dev cni0 lladdr e6:45:5a:1b:ea:19 STALE 34.122.121.32 dev flannel.1 FAILED 10.244.0.13 dev cni0 lladdr 1a:f0:3e:6f:15:cd REACHABLE 192.168.1.254 dev enp3s0 lladdr 88:b3:62:24:2d:60 REACHABLE 10.244.0.10 dev cni0 lladdr 72:1f:44:6e:26:53 REACHABLE 192.168.1.73 dev enp3s0 lladdr d0:53:49:57:e3:57 REACHABLE fe80::c05c:bcff:fec1:6b75 dev cni0 lladdr c2:5c:bc:c1:6b:75 STALE fe80::e01a:a6ff:feb3:2f7 dev cni0 lladdr e2:1a:a6:b3:02:f7 STALE fe80::94aa:28ff:fe03:6dbb dev cni0 lladdr 96:aa:28:03:6d:bb STALE fe80::dc99:d3ff:fe1b:2fdd dev cni0 lladdr de:99:d3:1b:2f:dd STALE fe80::18f0:3eff:fe6f:15cd dev cni0 lladdr 1a:f0:3e:6f:15:cd STALE fe80::bc8d:bff:fef3:8633 dev cni0 lladdr be:8d:0b:f3:86:33 STALE fe80::5496:b4ff:feba:1efa dev cni0 lladdr 56:96:b4:ba:1e:fa STALE fe80::389f:2aff:febf:d24e dev cni0 lladdr 3a:9f:2a:bf:d2:4e STALE fe80::4475:aeff:fe95:79b3 dev cni0 lladdr 46:75:ae:95:79:b3 STALE fe80::2c7d:d6ff:fe9e:e4e dev cni0 lladdr 2e:7d:d6:9e:0e:4e STALE fe80::1 dev enp3s0 lladdr 88:b3:62:24:2d:60 router STALE fe80::44bf:ff:fee5:312e dev cni0 lladdr 46:bf:00:e5:31:2e STALE fe80::701f:44ff:fe6e:2653 dev cni0 lladdr 72:1f:44:6e:26:53 STALE fe80::e445:5aff:fe1b:ea19 dev cni0 lladdr e6:45:5a:1b:ea:19 STALE fe80::f43d:7aff:fe45:8c38 dev cni0 lladdr f6:3d:7a:45:8c:38 STALE fe80::20e8:b1ff:fe87:f9c1 dev cni0 lladdr 22:e8:b1:87:f9:c1 STALE fe80::ec05:a5ff:fefb:f004 dev cni0 lladdr ee:05:a5:fb:f0:04 STALE fe80::b0c6:55ff:fe3d:8fc4 dev cni0 lladdr b2:c6:55:3d:8f:c4 STALE

ip a

2: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 3c:4a:92:55:34:84 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.100/24 brd 192.168.1.255 scope global dynamic noprefixroute enp3s0
valid_lft 71015sec preferred_lft 71015sec
inet6 2403:3800:322b:603:b91d:9ee4:977f:a62c/64 scope global temporary dynamic 
valid_lft 86215sec preferred_lft 71148sec
inet6 2403:3800:322b:603:5ba0:d7d5:3e24:b976/64 scope global dynamic mngtmpaddr noprefixroute 
valid_lft 86215sec preferred_lft 86215sec
inet6 2403:3800:322b:49a:b91d:9ee4:977f:a62c/64 scope global temporary deprecated dynamic 
valid_lft 70925sec preferred_lft 0sec
inet6 2403:3800:322b:49a:cd66:2849:154f:3080/64 scope global temporary deprecated dynamic 
valid_lft 70925sec preferred_lft 0sec
inet6 2403:3800:322b:49a:bc4a:f344:181e:281f/64 scope global temporary deprecated dynamic 
valid_lft 70925sec preferred_lft 0sec
inet6 2403:3800:322b:49a:4a95:b9ba:ab0b:bd3f/64 scope global deprecated dynamic mngtmpaddr noprefixroute 
valid_lft 70925sec preferred_lft 0sec
inet6 fe80::21f2:2000:9a2a:2e4d/64 scope link noprefixroute 
valid_lft forever preferred_lft forever
3: wlp2s0b1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether e0:2a:82:fb:d5:cc brd ff:ff:ff:ff:ff:ff
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
link/ether 02:42:91:be:07:3f brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
5: br-6f507f240f54: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
link/ether 02:42:ed:92:bd:ab brd ff:ff:ff:ff:ff:ff
inet 172.20.0.1/16 brd 172.20.255.255 scope global br-6f507f240f54
valid_lft forever preferred_lft forever
inet6 fc00:f853:ccd:e793::1/64 scope global tentative 
valid_lft forever preferred_lft forever
inet6 fe80::1/64 scope link tentative 
valid_lft forever preferred_lft forever
6: br-847829250b8c: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
link/ether 02:42:a0:ce:67:5d brd ff:ff:ff:ff:ff:ff
inet 172.19.0.1/16 brd 172.19.255.255 scope global br-847829250b8c
valid_lft forever preferred_lft forever
7: br-940c3874b064: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
link/ether 02:42:33:0a:ed:c9 brd ff:ff:ff:ff:ff:ff
inet 172.18.0.1/16 brd 172.18.255.255 scope global br-940c3874b064
valid_lft forever preferred_lft forever
8: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
link/ether 52:54:00:37:80:14 brd ff:ff:ff:ff:ff:ff
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
valid_lft forever preferred_lft forever
9: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel master virbr0 state DOWN group default qlen 1000
link/ether 52:54:00:37:80:14 brd ff:ff:ff:ff:ff:ff
11: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default 
link/ether ea:e0:0d:4c:85:67 brd ff:ff:ff:ff:ff:ff
inet 10.244.0.0/32 brd 10.244.0.0 scope global flannel.1
valid_lft forever preferred_lft forever
inet6 fe80::e8e0:dff:fe4c:8567/64 scope link 
valid_lft forever preferred_lft forever
12: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
link/ether 5a:e4:c4:12:f7:4a brd ff:ff:ff:ff:ff:ff
inet 10.244.0.1/24 brd 10.244.0.255 scope global cni0
valid_lft forever preferred_lft forever
inet6 fe80::58e4:c4ff:fe12:f74a/64 scope link 
valid_lft forever preferred_lft forever
18: veth58578a24@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default 
link/ether 86:65:e4:05:d0:7c brd ff:ff:ff:ff:ff:ff link-netns cni-9d8ed6c9-9048-2b39-b799-04e828c9de25
inet6 fe80::8465:e4ff:fe05:d07c/64 scope link 
valid_lft forever preferred_lft forever
19: veth83d654aa@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default 
link/ether 02:4e:d9:bb:d7:3e brd ff:ff:ff:ff:ff:ff link-netns cni-bf6a6602-462f-d47a-db7e-4b1fbff87f73
inet6 fe80::4e:d9ff:febb:d73e/64 scope link 
valid_lft forever preferred_lft forever
20: veth8e382bf1@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default 
link/ether ee:4d:03:3c:cf:27 brd ff:ff:ff:ff:ff:ff link-netns cni-4a69d3aa-02b3-3d2d-c082-d04d26e99c26
inet6 fe80::ec4d:3ff:fe3c:cf27/64 scope link 
valid_lft forever preferred_lft forever
21: veth77f4365a@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default 
link/ether 2a:2f:c0:b4:84:6d brd ff:ff:ff:ff:ff:ff link-netns cni-bc4e3554-db66-48a7-6ce5-0927c589f27d
inet6 fe80::282f:c0ff:feb4:846d/64 scope link 
valid_lft forever preferred_lft forever
22: vetha82cf484@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default 
link/ether fe:d7:b7:56:b0:ee brd ff:ff:ff:ff:ff:ff link-netns cni-06885625-eb9b-97b3-4b3a-217597a61263
inet6 fe80::fcd7:b7ff:fe56:b0ee/64 scope link 
valid_lft forever preferred_lft forever
23: vethbe7db6af@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default 
link/ether 42:b9:25:5f:56:db brd ff:ff:ff:ff:ff:ff link-netns cni-99b484b1-c75a-dba2-5229-f42992af32da
inet6 fe80::40b9:25ff:fe5f:56db/64 scope link 
valid_lft forever preferred_lft forever
24: vetheb84669e@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default 
link/ether 36:a6:ad:f5:64:38 brd ff:ff:ff:ff:ff:ff link-netns cni-24276917-be09-891b-40cb-6d93e6f814f5
inet6 fe80::34a6:adff:fef5:6438/64 scope link 
valid_lft forever preferred_lft forever
25: veth7963ccfe@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default 
link/ether 7a:57:8a:19:3c:54 brd ff:ff:ff:ff:ff:ff link-netns cni-6c2ab0f3-d875-ed27-417a-90ef1d054a00
inet6 fe80::7857:8aff:fe19:3c54/64 scope link 
valid_lft forever preferred_lft forever
26: vethc04ca050@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default 
link/ether e6:16:e8:fe:15:90 brd ff:ff:ff:ff:ff:ff link-netns cni-a8537697-481c-c5f6-116a-6565dcfa9c12
inet6 fe80::e416:e8ff:fefe:1590/64 scope link 
valid_lft forever preferred_lft forever
27: veth5267eb7b@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default 
link/ether f2:bd:11:ca:70:f9 brd ff:ff:ff:ff:ff:ff link-netns cni-cad70dd8-55a7-1cd3-c9b1-9af5f4279d88
inet6 fe80::f0bd:11ff:feca:70f9/64 scope link 
valid_lft forever preferred_lft forever
28: veth6e1d8373@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default 
link/ether c6:68:e2:3f:a0:7a brd ff:ff:ff:ff:ff:ff link-netns cni-18e5b9a4-2fde-f8dd-7620-e76fe4e3be47
inet6 fe80::c468:e2ff:fe3f:a07a/64 scope link 
valid_lft forever preferred_lft forever
29: veth65ffc762@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default 
link/ether 86:c4:e1:f3:44:4d brd ff:ff:ff:ff:ff:ff link-netns cni-97c921cb-3ded-027a-5019-b2a628a61583
inet6 fe80::84c4:e1ff:fef3:444d/64 scope link 
valid_lft forever preferred_lft forever
30: veth242d4505@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default 
link/ether 96:c1:12:eb:55:1b brd ff:ff:ff:ff:ff:ff link-netns cni-fb15b669-6eba-a90f-b4f2-fc071b9509d0
inet6 fe80::94c1:12ff:feeb:551b/64 scope link 
valid_lft forever preferred_lft forever
31: veth11d4a14d@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default 
link/ether ca:64:ef:e2:6a:77 brd ff:ff:ff:ff:ff:ff link-netns cni-9eed5228-0d7c-4225-e5d6-d67734f812bd
inet6 fe80::c864:efff:fee2:6a77/64 scope link 
valid_lft forever preferred_lft forever
32: vethe806cfd8@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default 
link/ether 22:7c:ac:09:90:5e brd ff:ff:ff:ff:ff:ff link-netns cni-e75517f4-45f3-1b8e-b598-263cbd6742be
inet6 fe80::207c:acff:fe09:905e/64 scope link 
valid_lft forever preferred_lft forever
33: veth1a143748@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default 
link/ether 22:34:a8:bb:a4:d3 brd ff:ff:ff:ff:ff:ff link-netns cni-cb09e88b-3251-ce51-084a-52eb570c4dec
inet6 fe80::2034:a8ff:febb:a4d3/64 scope link 
valid_lft forever preferred_lft forever
34: veth104a6ba8@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default 
link/ether 3e:17:f2:25:8d:23 brd ff:ff:ff:ff:ff:ff link-netns cni-3538fc5a-13c5-53ae-c24a-81b0c789c3e1
inet6 fe80::3c17:f2ff:fe25:8d23/64 scope link 
valid_lft forever preferred_lft forever

linkerd check

× viz extension self-checkeck
Post "http://localhost:40009/api/v1/SelfCheck": EOF
see https://linkerd.io/2.10/checks/#l5d-viz-metrics-api for hints
‼ linkerd-viz pods are injected
could not find proxy container for metrics-api-86b88fddc-jrdgb pod
see https://linkerd.io/2.10/checks/#l5d-viz-pods-injection for hints
‼ viz extension pods are running
container metrics-api-86b88fddc-jrdgb in pod metrics-api is not ready
see https://linkerd.io/2.10/checks/#l5d-viz-pods-running for hints

This is also not that helpful.

A bit about Readiness and Liveness

The `Readiness` and `Liveness` probes serve slightly different purposes.

The readiness probe controls whether the pod IP is included in the list of endpoints for a service, and so also whether a target for a route when it is exposed via an external URL.

The liveness probe determines whether a pod is still running normally or whether it should be restarted.

Technically an application could still be running fine, but is perhaps backlogged, and so you want to use the readiness probe to temporarily remove it from the set of endpoints for a service to avoid further requests being routed its way and simply being blocked in the request queue for that specific pod when another pod could handle it.

So I personally would agree the duplication seems strange, but it is that way so the different situations can be distinguished.

`kubectl -n linkerd-viz get all` : List all resources in the namespace

Nothing is else comes to mind so uninstall and reninstall.
1. `linkerd viz uninstall | kubectl delete -f -`
2. `linkerd viz install | kubectl apply -f -`

Everything looks good

kubectl -n linkerd-viz get all | less NAME READY STATUS RESTARTS AGE pod/grafana-5c47556496-vnsd6 2/2 Running 0 102s pod/metrics-api-86b88fddc-g9jnv 2/2 Running 0 102s pod/prometheus-554c6cc89f-4vcxz 2/2 Running 0 101s pod/tap-b5f699dcb-7d46j 2/2 Running 0 100s pod/tap-injector-7784dbcfb8-kfv67 2/2 Running 0 98s pod/web-6b8894468b-ghfgk 2/2 Running 0 98s

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/grafana ClusterIP 10.103.156.249 3000/TCP 102s service/metrics-api ClusterIP 10.103.119.148 8085/TCP 103s service/prometheus ClusterIP 10.108.254.177 9090/TCP 101s service/tap ClusterIP 10.110.124.32 8088/TCP,443/TCP 101s service/tap-injector ClusterIP 10.110.82.222 443/TCP 100s service/web ClusterIP 10.97.146.27 8084/TCP,9994/TCP 99s

NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/grafana 1/1 1 1 102s deployment.apps/metrics-api 1/1 1 1 103s deployment.apps/prometheus 1/1 1 1 101s deployment.apps/tap 1/1 1 1 100s deployment.apps/tap-injector 1/1 1 1 99s deployment.apps/web 1/1 1 1 99s

NAME DESIRED CURRENT READY AGE replicaset.apps/grafana-5c47556496 1 1 1 102s replicaset.apps/metrics-api-86b88fddc 1 1 1 103s replicaset.apps/prometheus-554c6cc89f 1 1 1 101s replicaset.apps/tap-b5f699dcb 1 1 1 100s replicaset.apps/tap-injector-7784dbcfb8 1 1 1 99s replicaset.apps/web-6b8894468b 1 1 1 99s

Config now has been reset

kubectl get deployment web -n linkerd-viz -o=json | jq '.spec.template.spec.containers[0].args' [ "-linkerd-controller-api-addr=linkerd-controller-api.linkerd.svc.cluster.local:8085", "-linkerd-metrics-api-addr=metrics-api.linkerd-viz.svc.cluster.local:8085", "-cluster-domain=cluster.local", "-grafana-addr=grafana.linkerd-viz.svc.cluster.local:3000", "-controller-namespace=linkerd", "-viz-namespace=linkerd-viz", "-log-level=info", "-enforced-host=^(localhost|127\.0\.0\.1|web\.linkerd-viz\.svc\.cluster\.local|web\.linkerd-viz\.svc|\[::1\])(:\d+)?$" ]

Set the external IP `kubectl patch -n linkerd-viz svc web -p '{"spec":{"externalIPs":["192.168.1.100"]}}'`

Everything is working fine.

# Resource
- https://stackoverflow.com/questions/44309291/kubernetes-readiness-probe-execution-after-container-started

[anitsh]

Deploying Emojivoto and use Linkerd CLI tap, top, edges, and stat commands.

Emojivoto is a gRPC application that has three services:

• web: the frontend that users interact with
• emoji: provides API endpoints to list emoji
• voting: provides API endpoints to vote for emoji
• There is also a fourth component, vote-boti, that generates simulated traffic to the application

Deploy the Emojivoto application to your cluster: kubectl apply -f https://run.linkerd.io/emojivoto.yml

Review the yaml file and saw ServiceAccount.

kubectget all -n emojivoto

NAME                            READY   STATUS    RESTARTS   AGE
pod/emoji-66ccdb4d86-dgq5v      1/1     Running   0          13m
pod/vote-bot-69754c864f-lb9rf   1/1     Running   0          13m
pod/voting-f999bd4d7-2g8hz      1/1     Running   0          13m
pod/web-79469b946f-b82rv        1/1     Running   0          13m

NAME                 TYPE        CLUSTER-IP       EXTERNAL-IP     PORT(S)             AGE
service/emoji-svc    ClusterIP   10.105.211.215   <none>         8080/TCP,8801/TCP   13m
service/voting-svc   ClusterIP   10.105.176.45    <none>          8080/TCP,8801/TCP   13m
service/web-svc      ClusterIP   10.108.91.59      <none>          80/TCP              13m

NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/emoji      1/1     1            1           13m
deployment.apps/vote-bot   1/1     1            1           13m
deployment.apps/voting     1/1     1            1           13m
deployment.apps/web        1/1     1            1           13m

NAME                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/emoji-66ccdb4d86      1         1         1       13m
replicaset.apps/vote-bot-69754c864f   1         1         1       13m
replicaset.apps/voting-f999bd4d7      1         1         1       13m
replicaset.apps/web-79469b946f        1         1         1       13m

Default setup to access is to port forward but as are accessing NOT from the host so we need to update the External IP to the service but before that lets check the current service status. kubectl -n emojivoto port-forward svc/web-svc 8080:80 kubectl describe service web-svc -n emojivoto

Name:              web-svc
Namespace:         emojivoto
Labels:            <none>
Annotations:       <none>
Selector:          app=web-svc
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.108.91.59
IPs:               10.108.91.59
Port:              http  80/TCP
TargetPort:        8080/TCP
Endpoints:         10.244.1.21:8080
Session Affinity:  None
Events:            <none>

kubectl patch service web-svc -n emojivoto -p '{"spec":{"externalIPs":["192.168.1.100"]}}'

Injecting the Emojivoto deployments with the Linkerd proxy. First, annotate the emojivoto namespace: kubectl annotate ns emojivoto linkerd.io/inject=enabled

This annotation is all we need to inform Linkerd to inject the proxies into pods in this namespace. However, simply adding the annotation won't affect existing resources. We'll also need to restart the Emojivoto deployments: kubectl rollout restart deploy -n emojivoto

kubectl scale deploy/web -n emojivoto --replicas=2

Tap all the traffic from the web deployment: linkerd tap deploy/web -n emojivoto

tap traffic from deploy/web and output as JSON: linkerd tap deployment/web --namespace emojivoto --to deployment/voting --path /emojivoto.v1.VotingService/VoteDoughnut -o json

Use linkerd top to view traffic sorted by the most popular paths: linkerd top deploy/web -n emojivoto

See the golden metrics, the latencies, success/error rates, and requests per second that you saw in the dashboard, except this time, you want to see them in the terminal. linkerd stat deploy -n emojivoto

Let's narrow the query to focus in on the traffic from the web deployment to the emoji deployment: linkerd viz stat -n emojivoto deploy/web --to deploy/emoji -o wide

Let's take a look at the traffic between the web and voting deployments to investigate linkerd stat -n emojivoto deploy/web --to deploy/voting -o wide

View the metrics for traffic to all deployments in the emojivoto namespace that comes from the web deployment: linkerd viz stat -n emojivoto deploy --from deploy/web

The Linkerd control plane is not only "aware" of the services that are meshed, but it also is aware of which services communicate with each other.

Get the edges for all the deployments in the emojivoto namespace: linkerd edges -n emojivoto deploy

Zoom in on the service graph by looking at the edges between pods rather than deployments, like this: linkerd edges -n emojivoto po

Notes

ServiceAccount A service account provides an identity for processes that run in a Pod. When you (a human) access the cluster (for example, using kubectl), you are authenticated by the apiserver as a particular User Account (currently this is usually admin, unless your cluster administrator has customized your cluster). Processes in containers inside pods can also contact the apiserver. When they do, they are authenticated as a particular Service Account (for example, default)/

Ports Kubernetes does not manage loose containers but manages them in units called a Pod. A Pod could be a single container or set of containers launched as a service.

“containerPort” defines the port on which app can be reached out inside the container.

targetPort and containerPort must be identical most of the time because whatever port is open for your application in a container that would be the same port you will wish to send traffic from service via targetPort.

The Service object of Kubernetes exposes a Pod/s to the outer world with nodePort via static IP. Furthermore, it also load-balances among the Pods and also acts as a firewall.

• Port exposes the Kubernetes service on the specified port within the cluster. Other pods within the cluster can communicate with this server on the specified port.
• TargetPort is the port on which the service will send requests to, that your pod will be listening on. Your application in the container will need to be listening on this port also.
• NodePort exposes a service externally to the cluster by means of the target nodes IP address and the NodePort. NodePort is the default setting if the port field is not specified.

apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    app: hello-world
spec:
  containers:
  - name: nginx
    image: nginx
    ports:
      - containerPort: 80
 ---
apiVersion: v1
kind: Service
metadata:
  name: hello-world
spec:
  type: NodePort
  selector:
    app: hello-world
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 80
      nodePort: 30036

Create a pod running nginx that listens at port 80. Create a service which will be exposed internally to other applications in the same cluster at port 8080 and externally to the cluster at port 30036. It will also forward requests to pods with the label “app: hello-world” on port 80.

Resource

• https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account
• https://github.com/linkerd/linkerd2/issues/2585
• https://sysdig.com/blog/debug-kubernetes-crashloopbackoff
• https://linkerd.io/2.10/tasks/troubleshooting

[anitsh]

How to get per-route metrics using service profiles

By creating a ServiceProfile for a Kubernetes service, we can specify the routes available to the service and collect data.

Use the Linkerd dashboard and linkerd routes command to view per-route metrics for services in the Emojivoto application.

A Linkerd service profile is implemented by instantiating a Kubernetes Custom Resource Definitions (CRD) called, as you might expect, ServiceProfile. The ServiceProfile enumerates the routes that Linkerd should expect for that service.

Use the linkerd profile command to generate ServiceProfile definitions. If the services in your application have OpenAPI or protobuf definitions, you can use those to create ServiceProfile definitions. Alternatively, Linkerd can observe requests in real time to generate the service profile for you. Finally, you can write the profile by hand.

We create a per-route metric for UI with linkerd profile -h

Viewing Per-Route Metrics with the Linkerd CLI linkerd routes deploy/emoji -n emojivoto linkerd routes deploy/web -n emojivoto linkerd routes deploy/voting -n emojivoto

linkerd viz top deploy/voting -n emojivoto

[anitsh]

Linkerd can help ensure reliability through features like traffic splitting, load balancing, retries, and timeouts. Each of these plays an important role in enhancing the overall reliability of the system.

But what kind of reliability can these features add, exactly? The answer is that, ultimately, what Linkerd (or any service mesh!) can help protect against, is transient failures. If a service is completely down, or consistently returns a failure, or consistently hangs, no amount of retrying or load balancing can help. But if one single instance of a service is having an issue, or if the underlying issue is just a temporary one, well, that's where Linkerd can be useful. Happily—or unhappily?—these sorts of partial, transient failures are endemic to distributed systems!

There are two important things to understand when it comes to Linkerd's core reliability featureset of load balancing, retries, and timeouts (traffic splitting, also a reliability feature, is a bit different—we'll be addressing that in a later chapter). They are:

• All these techniques happen on the client side: the proxy on the side that makes the call is the one that enacts these features, not the one on the server side. One implication is that if your server is meshed, but your client isn't, these features will not be enabled for calls between the two!
• These three techniques work best together. Timeouts have little value without retries; retries have little value without load balancing.

Linkerd's actual and effective metrics can diverge in the presence of retries or timeouts, but the actual numbers represent what actually hit the server, and the effective numbers represent what the client effectively got in response to its request, after Linkerd's reliability logic did its duty.

Requests with the HTTP POST method are not retryable in Linkerd today. This is for implementation reasons: POST requests almost always contain data in the request body, and retrying a request means that the proxy must store that data in memory. So, to maintain minimal memory usage, the proxy does not store POST request bodies, and they cannot be retried.

As we discussed in earlier chapters, Linkerd considers only 5XX status codes in responses as errors. Both 2XX and 4XX are recognized as successful status codes. The difference is subtle, but important: 4XX status codes indicate that the server looked but couldn't find the resource—correct behavior on the part of the server. 5XX status codes indicate that the server ran into an error while processing the request—incorrect behavior.

[anitsh]

From the last setup, further research was done on Flannel, Linkerd, security of Kubernetes overall.

Some conclusions made were:

• Flannel needs to be enhanced further to provide better end-to-end encryption security.
• Flannel does not work service meshes, so need to research further on Flannel and Linkerd usages.
• There was no end-to-end logging and monitoring, so need to setup a Security Information and Event Management (SIEM). And configure clusters to forward all audit and log events to it.

The next I thought to move ahead with to use the setup and make it more practical would be run example Microservices applications completely with full build and deploy process. And for that a CI/CD system would be needed. Also, as I am trying to move away from Docker, I looked into tools to build images. Kaniko, Buildah/Podman seems to be the best options, with Kaniko fitting more with it's native feature to build image inside container, it will be used.

Then, there was the case of local image registry was looked for and Harbor seems to be a good tool. Quay also seems to be good tool, Podman and Docker as well. But for now Harbor will be tried at first as an image registry. For trial purpose, Dockerhub public image repository will be used.

For CI, Jenkins running on Kubernetes will implemented, and probably ArgoCD for CD. So the first step was to run Jenkins in Kubernetes. To run Jenkins on Kubernetes, requires Persistent Volume Claim. A better understanding of PVC was needed.

The steps would be:

1. Run Jenkins in Kubernetes
2. Setup CI to be fired with Github's webhook
3. Run Kaniko in Kubernetes to build image
4. Pushes image to Dockerhub
5. The application should be deployed on the respective Kubernetes cluster

Note

https://goharbor.io Harbor is an open source registry that secures artifacts with policies and role-based access control, ensures images are scanned and free from vulnerabilities, and signs images as trusted. Harbor, a CNCF Graduated project, delivers compliance, performance, and interoperability to help you consistently and securely manage artifacts across cloud native compute platforms like Kubernetes and Docker.

https://www.projectquay.io Quay is a container image registry that enables you to build, organize, distribute, and deploy containers. Quay gives you security over your repositories with image vulnerability scanning and robust access controls. Project Quay provides a scalable open source platform to host container images across any size organization.

https://argoproj.github.io/argo-cd Argo CD is a declarative, GitOps continuous delivery tool for Kubernetes.

Kubernetes Storage

A PersistentVolume (PV) is a piece of storage in the cluster that has been provisioned by an administrator. It is a resource in the cluster just like a node is a cluster resource. PVs are volume plugins like Volumes, but have a lifecycle independent of any individual pod that uses the PV. This API object captures the details of the implementation of the storage, be that NFS, iSCSI, or a cloud-provider-specific storage system. A PersistentVolumeClaim (PVC) is a request for storage by a user. It is similar to a pod. Pods consume node resources and PVCs consume PV resources. Pods can request specific levels of resources (CPU and Memory). Claims can request specific size and access modes (e.g., can be mounted once read/write or many times read-only).

hostPath volume Type The hostPath volume type is single-node only, meaning that a pod on one node cannot access the hostPath volume on another node. One way to get around this limitation may be to either create a StatefulSet or Daemonset which could force pods to always deploy to the same node(s), or force a deployment's pods to always be deployed to the same node via a nodeSelector.

If you are circumventing the hostPath single-node limitation via nodeSelector while using multiple pods on the same node beware of the following issue:

• Multiple Kubernetes pods sharing the same host-path/pvc will duplicate output

Alternative Volume Types If you do not wish to circumvent the limitation of the hostPath volume type, you should look into other volume types such as NFS or Gluster, both of which you can setup locally, but require some additional configuration and setup.

If you have only one drive which you can attach to one node, I think you should use the basic NFS volume type as it does not require replication.

If however, you can afford another drive to plug in to the second node, you can take advantage of GlusterFS's replication feature.

Volume Types

• NFS: https://kubernetes.io/docs/concepts/storage/volumes/#nfs
• GlusterFS: https://kubernetes.io/docs/concepts/storage/volumes/#glusterfs

Converting a drive to a volume: As for making your hard-drive become a persistent volume, I would separate that into 2 tasks.

• You need mount your physical drive to make it available at a specific path within your operating system.
• Refer to the path of the mounted drive when configuring NFS, GlusterFS, or hostPath.

Resource

• https://thenewstack.io/kaniko-builds-container-images-without-the-docker-daemon
• https://github.com/GoogleContainerTools/kaniko#running-kaniko-in-a-kubernetes-cluster
• https://codefresh.io/devops/docker-images-without-docker-practical-guide
• https://docs.gitlab.com/ee/ci/docker/using_kaniko.html
• https://www.baeldung.com/ops/kaniko
• https://cloud.google.com/blog/products/containers-kubernetes/introducing-kaniko-build-container-images-in-kubernetes-and-google-container-builder-even-without-root-access
• https://thenewstack.io/tutorial-host-a-local-podman-image-registry
• https://docs.docker.com/registry/insecure
• https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry
• https://github.com/containerd/cri/blob/master/docs/registry.md
• https://support.cloudbees.com/hc/en-us/articles/360031223512-What-you-need-to-know-when-using-Kaniko-from-Kubernetes-Jenkins-Agents
• https://www.linux.com/audience/enterprise/set-cicd-pipeline-kubernetes-part-1-overview
• https://www.opcito.com/blogs/how-to-successfully-run-jenkins-inside-a-kubernetes-cluster
• https://devopscube.com/setup-jenkins-on-kubernetes-cluster
• https://vocon-it.com/2018/12/20/kubernetes-local-persistent-volumes
• https://cloudian.com/guides/kubernetes-storage/kubernetes-storage-101-concepts-and-best-practices
• https://blog.tentamen.eu/jenkins-and-github-integration-using-webhooks
• https://stackoverflow.com/questions/6957219/connect-github-to-secured-jenkins-via-http-post-commit-hook
• https://plugins.jenkins.io/github
• https://www.fourkitchens.com/blog/article/trigger-jenkins-builds-pushing-github
• https://plugins.jenkins.io/kubernetes

[anitsh]

Security Status

Spend some time on understanding security status of the Kubernetes cluster. Found a good tool: Kubescape

Kubescape is the first tool for testing if Kubernetes is deployed securely as defined in Kubernetes Hardening Guidance by to NSA and CISA Tests are configured with YAML files, making this tool easy to update as test specifications evolve.

Installed and did a security scan:

kubescape scan framework nsa --exclude-namespaces kube-system,kube-public

Summary - Passed:18   Failed:0   Total:18
+-------------------------------------------------+------------------+---------------+-----------+
|                  CONTROL NAME                   | FAILED RESOURCES | ALL RESOURCES | % SUCCESS |
+-------------------------------------------------+------------------+---------------+-----------+
| Allow privilege escalation                      | 0                | 18            | 100%      |
| Allowed hostPath                                | 0                | 18            | 100%      |
| Applications credentials in configuration files | 11               | 26            | 57%       |
| Automatic mapping of service account            | 20               | 20            | 0%        |
| Cluster-admin binding                           | 14               | 149           | 90%       |
| Control plane hardening                         | 0                | 18            | 100%      |
| Dangerous capabilities                          | 0                | 18            | 100%      |
| Exec into container                             | 2                | 149           | 98%       |
| Exposed dashboard                               | 0                | 35            | 100%      |
| Host PID/IPC privileges                         | 0                | 18            | 100%      |
| Immutable container filesystem                  | 18               | 18            | 0%        |
| Insecure capabilities                           | 0                | 18            | 100%      |
| Linux hardening                                 | 18               | 18            | 0%        |
| Non-root containers                             | 17               | 18            | 5%        |
| Privileged container                            | 0                | 18            | 100%      |
| Resource policies                               | 18               | 18            | 0%        |
| hostNetwork access                              | 0                | 18            | 100%      |
+-------------------------------------------------+------------------+---------------+-----------+
|                       17                        |       118        |      595      |    80%    |
+-------------------------------------------------+------------------+---------------+-----------+

The summary listed out many issues.

The configuration used to access the was admin's which was created while creating clusters using kubeadmin , i.e. /etc/kubernetes/admin.conf. I wanted to know if the result would change if another user with lower privileged would do the scan.

Current context

kubectl config current-context 
kubernetes-admin@kubernetes

So the question - How can I create a user and restrict user to access only one namespace in Kubernetes?

Kubernetes gives us a way to regulate access to Kubernetes clusters and resources based on the roles of individual users through a feature called Role-based access control (RBAC) via rbac.authorization.k8s.io/v1 API.

The RBAC API declares four kinds of Kubernetes object: Role, ClusterRole, RoleBinding and ClusterRoleBinding. You can describe objects, or amend them, using tools such as kubectl, just like any other Kubernetes object.

Therefore to achieve a complete isolation in Kubernetes, the concepts on namespaces and role based access control is used.

• Role: A role contains rules that represent a set of permissions. A role is used to grant access to resources within a namespace.
• RoleBinding: A role binding is used to grant the permissions defined in a role to a user or set of users. It holds a list of subjects (users, groups, or service accounts), and a reference to the role being granted.
• Service Account: Account meant for processes, which run in pods. The idea behind service accounts is based on the principle of least privilege. An account is created for specific tasks.

Create and Limit Service account to a namespace in Kubernetes:

• Create a demo namespace kubectl create namespace demo

• Create a service account

  cat <<EOF | kubectl apply -f -
  apiVersion: v1
  kind: ServiceAccount
  metadata:
  name: admin
  namespace: demo
  EOF

• Create a role that will give full access to the demo namespace

  
  cat <<EOF | kubectl apply -f -
  kind: Role
  apiVersion: rbac.authorization.k8s.io/v1
  metadata:
  name: admin
  namespace: demo
  rules:

• apiGroups: ["", "extensions", "apps"] resources: [""] verbs: [""]

• apiGroups: ["batch"] resources:

  • jobs
  • cronjobs verbs: ["*"] EOF

    
    Confirm it - `kubectl get roles -n demo`

• Bind the role to a user

  
  cat <<EOF | kubectl apply -f -
  kind: RoleBinding
  apiVersion: rbac.authorization.k8s.io/v1
  metadata:
  name: admin
  namespace: demo
  subjects:

• kind: ServiceAccount name: admin namespace: demo roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: admin EOF

  
  Confirm it - `kubectl get rolebindings --namespace demo`

• Check user token name kubectl describe sa admin -n demo

• Get service account token to be used to access Kubernetes on dashboard or through kubectl command line.

  
  export NAMESPACE="demo"
  export K8S_USER="demo-user"
  kubectl -n ${NAMESPACE} describe secret $(kubectl -n ${NAMESPACE} get secret | (grep ${K8S_USER} || echo "$_") | awk '{print $1}') | grep token: | awk '{print $2}'\n

or

kubectl get secret demo-user-token-xxxxx -n demo -o "jsonpath={.data.token}" | base64 -D

- Get certificate data, the Certificate Authority.
`kubectl  -n ${NAMESPACE} get secret `kubectl -n ${NAMESPACE} get secret | (grep ${K8S_USER} || echo "$_") | awk '{print $1}'` -o "jsonpath={.data['ca\.crt']}"`

or 
`kubectl get secret demo-user-token-xxxxx -n demo -o "jsonpath={.data['ca\.crt']}"`

- Create kubectl configuration .config file with the secret. 

apiVersion: v1 kind: Config preferences: {}

Define the cluster

clusters:

• cluster: certificate-authority-data: PLACE CERTIFICATE HERE

  You'll need the API endpoint of your Cluster here:

  server: https://YOUR_KUBERNETES_API_ENDPOINT name: demo

Define the user

users:

• name: admin user: as-user-extra: {} client-key-data: PLACE CERTIFICATE HERE token: PLACE USER TOKEN HERE

Define the context: linking a user to a cluster

contexts:

• context: cluster: demo namespace: demo user: admin name: demo

Define current context

current-context: demo

- Set the context to this
`kubectl config --kubeconfig=$HOME/.kube/demo-config set-context demo`

Need to confirm if this has to be done manually or it there a CLI way to do it?

kubectl config set-cluster default-cluster --server=https://${MASTER} \ --certificate-authority=/path/to/ca.pem

kubectl config set-credentials default-admin \ --certificate-authority=/path/to/ca.pem \ --client-key=/path/to/admin-key.pem \ --client-certificate=/path/to/admin.pem

kubectl config set-credentials default-admin \ --certificate-authority=/path/to/ca.pem \ --client-key=/path/to/admin-key.pem \ --client-certificate=/path/to/admin.pem

kubectl config set-context default-system --cluster=default-cluster --user=default-admin kubectl config use-context default-system

An example of role with limited permissions:

cat <<EOF | kubectl apply -f - kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: namespace: demo name: deployment-admin rules:

• apiGroups: ["", "extensions", "apps"] resources: ["deployments", "replicasets", "pods", "services", "ingresses"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] # You can also use ["*"] EOF

RBAC Role vs ClusterRole

If you want to define a role within a namespace, use a Role; if you want to define a role cluster-wide, use a ClusterRole.

An RBAC Role or ClusterRole contains rules that represent a set of permissions. Permissions are purely additive (there are no "deny" rules).

A Role always sets permissions within a particular namespace; when you create a Role, you have to specify the namespace it belongs in.

ClusterRole, by contrast, is a non-namespaced resource. The resources have different names (Role and ClusterRole) because a Kubernetes object always has to be either namespaced or not namespaced; it can't be both.

ClusterRoles have several uses. You can use a ClusterRole to:

define permissions on namespaced resources and be granted within individual namespace(s)
define permissions on namespaced resources and be granted across all namespaces
define permissions on cluster-scoped resources

A ClusterRole can be used to grant the same permissions as a Role. Because ClusterRoles are cluster-scoped, you can also use them to grant access to:

• cluster-scoped resources (like nodes)
• non-resource endpoints (like /healthz)
• namespaced resources (like Pods), across all namespaces. For example: you can use a ClusterRole to allow a particular user to run kubectl get pods --all-namespaces

Here's an example Role in the "default" namespace that can be used to grant read access to pods:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: default
  name: pod-reader
rules:
- apiGroups: [""] # "" indicates the core API group
  resources: ["pods"]
  verbs: ["get", "watch", "list"]

Here is an example of a ClusterRole that can be used to grant read access to secrets in any particular namespace, or across all namespaces (depending on how it is bound):

apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata:

"namespace" omitted since ClusterRoles are not namespaced

name: secret-reader rules:

• apiGroups: [""] #

  at the HTTP level, the name of the resource for accessing Secret

  objects is "secrets"

  resources: ["secrets"] verbs: ["get", "watch", "list"]

Resource

• https://github.com/armosec/kubescape
• https://learnk8s.io/how-many-clusters
• https://kubernetes.io/docs/reference/access-authn-authz/rbac/#role-and-clusterrole
• https://kubernetes.io/docs/reference/kubectl/cheatsheet
• https://github.com/kubernetes-sigs/kubespray/issues/257
• http://docs.shippable.com/deploy/tutorial/create-kubeconfig-for-self-hosted-kubernetes-cluster
• https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm

[anitsh]

Objectives

• [x] Security
• [ ] CI/CD

Top lists a process ran by user nobody. Found out the process is for Prometheus. https://askubuntu.com/questions/329714/what-is-the-purpose-of-the-nobody-user https://www.educba.com/nfs-in-linux

Security

https://www.youtube.com/watch?v=tGg_IjPLB20 https://kubernetes.io/docs/tasks/configure-pod-container/security-context https://kubernetes.io/docs/concepts/configuration/overview https://snyk.io/blog/10-kubernetes-security-context-settings-you-should-understand https://kubernetes.io/docs/tutorials/clusters/apparmor https://kubernetes.io/docs/tutorials/clusters/seccomp https://kubernetes.io/docs/tasks/debug-application-cluster https://kubernetes.io/docs/concepts/cluster-administration/addons https://stuartleeks.com/posts/working-with-multiple-kubernetes-contexts https://github.com/kubernetes/examples https://github.com/search?o=desc&q=microservices+examples&s=stars&type=Repositories https://kublr.com/blog/monitor-pods-in-kubernetes-with-heapster-influxdb-grafana, https://stackoverflow.com/questions/50704244/kubectl-top-nodes-shows-error-metrics-not-available-yet

Kubernetes Metrics Server

Metrics Server is a scalable, efficient source of container resource metrics for Kubernetes built-in autoscaling pipelines.

Metrics Server collects resource metrics from Kubelets and exposes them in Kubernetes apiserver through Metrics API for use by Horizontal Pod Autoscaler and Vertical Pod Autoscaler. Metrics API can also be accessed by kubectl top, making it easier to debug autoscaling pipelines.

Metrics Server is not meant for non-autoscaling purposes. For example, don't use it to forward metrics to monitoring solutions, or as a source of monitoring solution metrics. In such cases please collect metrics from Kubelet /metrics/resource endpoint directly.

Metrics Server offers: A single deployment that works on most clusters (see Requirements) Fast autoscaling, collecting metrics every 15 seconds. Resource efficiency, using 1 mili core of CPU and 2 MB of memory for each node in a cluster. Scalable support up to 5,000 node clusters.

https://github.com/kubernetes-sigs/metrics-server http://www.mtitek.com/tutorials/kubernetes/install-kubernetes-metrics-server.php

apt-mark

apt-mark hold package: apt hold back packages from update and upgrade apt-mark showhold: list hold packages apt-mark unhold pacakgeName: unhold https://www.cyberciti.biz/faq/apt-get-hold-back-packages-command

https://github.com/kubernetes/kubernetes/tree/master/test/images https://github.com/kubernetes/examples https://kubernetes.io/docs/tutorials/stateful-application https://olinux.net/wp-content/uploads/2019/01

[anitsh]

• [x] Setup Jenkins in Kubernetes Cluster

Issue: Pods/Deployment not ready, and log from deployment unavailable `k logs deployment/jenkins -n jenkins error: a container name must be specified for pod jenkins-79cf9b4b8f-hhnkx, choose one of: [jenkins linkerd-proxy] or one of the init containers: [linkerd-init]

''' The reason is because Istio adds a second container to deployments via the istio-sidecar-injector. This is what adds the envoy proxy in front of all of your pods traffic.

$ kubectl logs productpage-v1-84f77f8747-8zklx -c productpage ''' Solution: k logs deployment/jenkins -n jenkins -c jenkins

Deleted the entire namespace and recreated new. The resources created by default after new namespace is created are:

k apply -f namespace.yaml 
kubectl get all,cm,secret,ing -n jenkins
NAME                         DATA   AGE
configmap/kube-root-ca.crt   1      2m23s

NAME                         TYPE                                  DATA   AGE
secret/default-token-xg2j2   kubernetes.io/service-account-token   3      2m23s

Create ServiceAccount

k apply -f serviceAccount.yaml 
serviceaccount/jenkins created
clusterrole.rbac.authorization.k8s.io/jenkins created
clusterrolebinding.rbac.authorization.k8s.io/jenkins created

Create storage volume

k apply -f volume.yaml 
storageclass.storage.k8s.io/local-storage unchanged
persistentvolume/jenkins-pv-volume unchanged
persistentvolumeclaim/jenkins-pv-claim created

Create deployment

k apply -f deployment.yaml 
deployment.apps/jenkins created

List images and containers from Containerd

sudo ctr -n k8s.io images ls
sudo ctr -n k8s.io containers ls

Inject Linerd to the namespace cat deployment.yaml | linkerd inject - | kubectl apply -f -

Set external IP kubectl patch -n jenkins svc jenkins-service -p '{"spec":{"externalIPs":["192.168.1.100"]}}'

kubectl get all,cm,secret,ing -A: Get most of the resources - pod, service, daemonset, deployment ,replicaset, statefulset, job, configmap, secret, ingress.

Conclusion

First attempt worked. I wanted to learned each steps more and redid the steps which did not work out.

Resource

• [ ] https://devopscube.com/setup-jenkins-on-kubernetes-cluster
• [ ] https://www.jenkins.io/doc/book/installing/kubernetes
• [ ] https://raw.githubusercontent.com/jenkins-infra/jenkins.io/master/content/doc/tutorials/kubernetes/installing-jenkins-on-kubernetes/jenkins-sa.yaml
• [ ] https://raw.githubusercontent.com/jenkins-infra/jenkins.io/master/content/doc/tutorials/kubernetes/installing-jenkins-on-kubernetes/jenkins-volume.yaml
• [ ] https://raw.githubusercontent.com/jenkins-infra/jenkins.io/master/content/doc/tutorials/kubernetes/installing-jenkins-on-kubernetes/jenkins-deployment.yaml
• [ ] https://devopscube.com/kubernetes-ingress-tutorial
• [ ] https://devopscube.com/setup-ingress-kubernetes-nginx-controller

[anitsh]

• [x] Setup Jenkins in Kubernetes Cluster
• [ ] Setup CI/CD

Conclusion

Re-installation worked. Default manifests used from https://devopscube.com/setup-jenkins-on-kubernetes-cluster. An issue with volume mount was as the volume was on created on node and given RW permission.

https://www.magalix.com/blog/create-a-ci/cd-pipeline-with-kubernetes-and-jenkins https://loft.sh/blog/kubernetes-namespaces-vs-virtual-clusters https://kubernetes.io/docs/concepts/storage https://kubernetes.io/docs/concepts/configuration https://www.jenkins.io/blog/2021/08/23/jenkins-operator-security-work-report https://jenkinsci.github.io/kubernetes-operator/docs/getting-started/latest/schema https://docs.openshift.com/container-platform/4.1/nodes/containers/nodes-containers-init.html

[anitsh]

Objectives

• [x] Understand more Kubernetes native CI/CD solution.

• https://tekton.dev, https://github.com/tektoncd/pipeline Tekton is a cloud-native solution for building CI/CD systems. It consists of Tekton Pipelines, which provides the building blocks, and of supporting components, such as Tekton CLI and Tekton Catalog, that make Tekton a complete ecosystem.

• https://jenkins-x.io Uses Tekton's Pipeline.

• https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions

• https://landscape.cd.foundation

• https://github.com/ortelius/ortelius Ortelius simplifies the implementation of microservices. By providing a central catalog of services with their deployment specs, application teams can easily consume and deploy services across cluster. Ortelius tracks application versions based on service updates and maps their service dependencies eliminating confusion and guess work.

Next Steps

• [ ] Implement Tekton and move to JenkinsX.
• [ ] Use https://goharbor.io, https://github.com/hashicorp/vault

Harbor is an open source registry that secures artifacts with policies and role-based access control, ensures images are scanned and free from vulnerabilities, and signs images as trusted. Harbor, a CNCF Graduated project, delivers compliance, performance, and interoperability to help you consistently and securely manage artifacts across cloud native compute platforms like Kubernetes and Docker.

[anitsh]

• [ ] Setup CI/CD pipeline in Jenkins
  • [ ] Run Tests, Linters, Unit, Functional
  • [ ] Notify if tests fails, email, slack
  • [x] Build image with Kaniko
  • [x] Deploy image to Docker registry
  • [ ] Apply in environment, production

Continuing from the last installed state of Jenkins.

Change namespace to jenkins kubectl config set-context --current --namespace=jenkins

List the resources in the namespace

kubectl get all,cm,secret,ing,pvc
NAME                           READY   STATUS    RESTARTS   AGE
pod/jenkins-559d8cd85c-wgj8z   1/1     Running   0          2d6h

NAME                      TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
service/jenkins-service   NodePort   10.98.160.40   <none>        8080:32000/TCP   2d6h

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/jenkins   1/1     1            1           2d6h

NAME                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/jenkins-559d8cd85c   1         1         1       2d6h

NAME                         DATA   AGE
configmap/kube-root-ca.crt   1      2d6h

NAME                               TYPE                                  DATA   AGE
secret/default-token-wvg5h         kubernetes.io/service-account-token   3      2d6h
secret/jenkins-admin-token-tqfhk   kubernetes.io/service-account-token   3      2d6h

NAME                                     STATUS   VOLUME              CAPACITY   ACCESS MODES   STORAGECLASS    AGE
persistentvolumeclaim/jenkins-pv-claim   Bound    jenkins-pv-volume   10Gi       RWO            local-storage   2d6h

Set external IP to access UI kubectl patch service/jenkins-service -p '{"spec":{"externalIPs":["192.168.1.100"]}}'

Build image with Kaniko

Build image with Kaniko and deployed to DockerHub. For demo the process was implemented as a Kubernetes Job.

To send image to Dockerhub, authentication is required. kubectl create secret docker-registry regcred --docker-server=https://index.docker.io/v2/ --docker-username=codeanit --docker-password=TOKEN --docker-email=codeanit@gmail.com

apiVersion: batch/v1
kind: Job
metadata:
  name: kaniko
spec:
  template:
    spec:
      containers:
      - name: kaniko
        image: gcr.io/kaniko-project/executor:latest
        args: ["--dockerfile=Dockerfile",
               "--context=git://github.com/kubernetes-utilities/kubernetes-kaniko.git#refs/heads/master",
               "--destination=codeanit/demo-cicd-go:v0.0.1"]
        volumeMounts:
        - name: kaniko-secret
          mountPath: "/kaniko/.docker"
      restartPolicy: Never
      volumes:
      - name: kaniko-secret
        secret:          
          secretName: regcred
          items:
          - key: .dockerconfigjson
            path: config.json

Although there wasn't any configuration set for AWS, an error was logged. But this did not stop from

k logs  pod/kaniko--1-q4xfx
Enumerating objects: 62, done.
Counting objects: 100% (15/15), done.
Compressing objects: 100% (12/12), done.
Total 62 (delta 5), reused 12 (delta 3), pack-reused 47
INFO[0001] GET KEYCHAIN                                 
INFO[0001] running on kubernetes ....                   
E0831 10:44:58.578978       1 aws_credentials.go:77] while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated.
        For verbose messaging see aws.Config.CredentialsChainVerboseErrors
INFO[0010] Retrieving image manifest nginx              
INFO[0010] Retrieving image nginx from registry index.docker.io 
INFO[0010] GET KEYCHAIN                                 
INFO[0014] Built cross stage deps: map[]                
INFO[0014] Retrieving image manifest nginx              
INFO[0014] Returning cached image manifest              
INFO[0014] Executing 0 build triggers                   
INFO[0014] Unpacking rootfs as cmd RUN echo 'This is version v0.0.1' > /usr/share/nginx/html/index.html requires it. 
INFO[0029] RUN echo 'This is version v0.0.1' > /usr/share/nginx/html/index.html 
INFO[0029] Taking snapshot of full filesystem...        
INFO[0033] cmd: /bin/sh                                 
INFO[0033] args: [-c echo 'This is version v0.0.1' > /usr/share/nginx/html/index.html] 
INFO[0033] Running: [/bin/sh -c echo 'This is version v0.0.1' > /usr/share/nginx/html/index.html] 
INFO[0033] Taking snapshot of full filesystem...        
INFO[0033] GET KEYCHAIN                                 
INFO[0033] Pushing image to codeanit/demo-cicd-go:v0.0.1 
INFO[0038] Pushed image to 1 destinations 

The tutorial, https://betterprogramming.pub/how-to-build-containers-in-a-kubernetes-cluster-with-kaniko-2d01cd3242a7, used FluxCD for deployment. FluxCD uses a push based GitOps, i.e. No Github webhooks are required, FluxCD pulls or checks the changes like cron jobs.

Resource

• [ ] https://betterprogramming.pub/how-to-build-containers-in-a-kubernetes-cluster-with-kaniko-2d01cd3242a7
• [ ] https://betterprogramming.pub/how-to-continuously-deliver-kubernetes-applications-with-flux-cd-502e4fb8ccfe
• [ ] https://github.com/codeanit/til/issues/668 GitOps
• [ ] https://blog.csanchez.org/2020/09/15/building-docker-images-with-kaniko-pushing-to-docker-registries/
• [ ] https://devopscube.com/build-docker-image-kubernetes-pod/
• [ ] https://dev.to/ankitbansal/building-custom-image-in-kubernetes-cluster-with-kaniko-11km
• [ ] https://cloud.google.com/blog/products/containers-kubernetes/introducing-kaniko-build-container-images-in-kubernetes-and-google-container-builder-even-without-root-access
• [ ] https://blog.alexellis.io/building-containers-without-docker/
• [ ] https://medium.com/hootsuite-engineering/building-docker-images-inside-kubernetes-42c6af855f25
• [ ] https://www.ibm.com/cloud/blog/build-a-container-image-inside-a-kubernetes-cluster-and-push-it-to-ibm-cloud-container-registry
• [ ] https://bmuschko.com/blog/go-on-jenkins/
• [ ] https://anupamgogoi.medium.com/writing-a-kubernetes-operator-from-zero-to-hero-8ca5dc2462b7
• [ ] https://medium.com/artisans-of-tech/building-a-ci-system-for-go-with-jenkins-4ab04d4bacd0
• [ ] https://www.jenkins.io/doc/book/getting-started/
• [ ] https://bmuschko.com/blog/go-on-jenkins/
• [ ] https://devopscube.com/setup-jenkins-on-kubernetes-cluster
• [ ] https://plugins.jenkins.io/kubernetes/
• [ ] https://betterprogramming.pub/how-to-continuously-deliver-kubernetes-applications-with-flux-cd-502e4fb8ccfe
• [ ] https://levelup.gitconnected.com/automating-build-and-test-of-go-application-with-jenkins-9f96879b5479

[anitsh]

After further reading on DevOps practices based on Kubernetes, JenkinsX seemed more relevant. Most example of service mesh included Istio, also in Google example: https://github.com/GoogleCloudPlatform/microservices-demo. So using Istio for this purpose.

Uninstall Linkerd: https://linkerd.io/2.10/tasks/uninstall, https://linkerd.io/2.10/reference/cli/uninject

• To remove Linkerd Viz

  linkerd viz uninstall | kubectl delete -f -

• To remove Linkerd Jaeger

  linkerd jaeger uninstall | kubectl delete -f -

• To remove Linkerd Multicluster

  linkerd multicluster uninstall | kubectl delete -f -

• Uninject from pods/deploymnets

  kubectl get deploy -o yaml | linkerd uninject - | kubectl apply -f -

A pod was stuck on terminating state so delete it forcefully: kubectl delete pod xxx --now/force https://stackoverflow.com/questions/35453792/pods-stuck-in-terminating-status, https://github.com/kubernetes/kubernetes/issues/51835,

Also deleted tekton-pipelines ns as it was not useful but it was also stuck in terminating status. Tired another approach for that. kubectl get namespace tekton-pipelines -o json > tekton-pipelines.json

Removed kubernetes value from

"spec": {
        "finalizers": [
        ]
    },

And reapplied:kubectl replace --raw "/api/v1/namespaces/tekton-pipelines/finalize" -f tekton-pipelines.json

• [ ] https://tech.ebayinc.com/research/kubernetes-secrets-a-secure-credential-store-for-jenkins, https://github.com/eBay/kube-credentials-plugin
• [ ] https://tech.ebayinc.com/engineering/how-ebay-leverages-kubernetes-helm-charts-and-jenkins-pipelines-to-deliver-high-quality-software
• [ ] https://tech.ebayinc.com/engineering/grit-a-protocol-for-distributed-transactions-across-microservices
• [ ] https://itnext.io/k8s-native-jenkins-x-and-tekton-pipelines-e2b5a61a1d22, https://itnext.io/jenkins-x-istio-flagger-canary-deployment-9d5e187c2334

Flagger https://docs.flagger.app

Flagger is a progressive delivery tool that automates the release process for applications running on Kubernetes. It reduces the risk of introducing a new software version in production by gradually shifting traffic to the new version while measuring metrics and running conformance tests. Flagger implements several deployment strategies (Canary releases, A/B testing, Blue/Green mirroring) using a service mesh (App Mesh, Istio, Linkerd, Open Service Mesh) or an ingress controller (Contour, Gloo, NGINX, Skipper, Traefik) for traffic routing. For release analysis, Flagger can query Prometheus, Datadog, New Relic, CloudWatch or Graphite and for alerting it uses Slack, MS Teams, Discord and Rocket. Flagger can be configured with Kubernetes custom resources and is compatible with any CI/CD solutions made for Kubernetes. Since Flagger is declarative and reacts to Kubernetes events, it can be used in GitOps pipelines together with tools like Flux, JenkinsX, Carvel, Argo, etc.

[anitsh]

State

The laptops with K8s installed were turned off. Kubelet was not active probably due to swapp was on. Re-instantiated Kubelet by executing shell script:

https://github.com/SystemUtilities/utils/blob/main/k8s-init.sh

Kubelet active. Node joined the cluster.

Resource

• [x] https://phoenixnap.com/kb/how-to-install-kubernetes-on-a-bare-metal-server

Note: This issue is has become too long. Need to make a note from this and close it.

[anitsh]

Setting up systems with primary domain and subdomains

• https://serverfault.com/questions/270683/forward-all-subdomains-to-an-ip-in-bind
• https://serverfault.com/questions/473284/wildcard-dns-redirect-to-root-as-well
• https://www.namecheap.com/support/knowledgebase/article.aspx/597/2237/how-can-i-set-up-a-catchall-wildcard-subdomain
• https://webmasters.stackexchange.com/questions/126973/redirect-all-subdomains-from-one-domain-to-the-equivalent-subdomain-of-another
• https://blog.logrocket.com/how-to-build-web-app-with-multiple-subdomains-nginx
• https://srimalfernando.medium.com/how-to-host-multiple-servers-with-different-sub-domains-inside-one-digitalocean-droplet-f6d5887538fa