.. contents:: Contents
:depth: 4
Try Kubernetes on separated machines
Install
Based on http://tracpath.com/works/devops/how-to-install-the-kubernetes-kubeadm/
(VirtualBox) Add an internal network between VMs::
http://qiita.com/areaz_/items/c9075f7a0b3e147e92f2#%E3%82%B2%E3%82%B9%E3%83%88os%E3%81%AE%E5%8B%95%E4%BD%9C%E7%A2%BA%E8%AA%8D
Shutdown a VM
Setting -> Network -> Adapter 2
Check "Enable Network Adapter"
Attached to: "Internal Network"
Reboot the VM
SSH into the VM (kube-host01 should be 172.168.0.2)
$ sudo vi /etc/network/interfaces
- auto enp0s8
- iface enp0s8 inet static
- address 172.168.0.1
-
netmask 255.255.255.0
Reboot the VM
Operate the following installation on both kube-master and kube-host01 (nfs-common is for Subpath)::
$ sudo su -
apt-get update && apt-get install -y apt-transport-https
echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" > /etc/apt/sources.list.d/kubernetes.list
add-apt-repository ppa:gluster/glusterfs-4.1
apt-get update
apt-get install -y docker.io nfs-common glusterfs-client
To install the latest packages of Kubernetes::
apt-get install -y kubelet kubeadm kubectl
If you want to install previous release of Kubernetes, check available releases with::
apt-cache policy kubelet
kubelet:
Installed: (none)
Candidate: 1.10.0-00
Version table:
1.10.0-00 500
500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
1.9.6-00 500
500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
1.9.5-00 500
[..]
#
Then you can specify the release like::
apt-get install -y kubelet=1.9.6-00 kubeadm=1.9.6-00 kubectl=1.9.6-00
Initialization of kube-master
(Flannel) Operate the following commands::
kubeadm init --pod-network-cidr=10.244.0.0/16
[..]
Your Kubernetes master has initialized successfully!
[..]
You can now join any number of machines by running the following on each node
as root:
kubeadm join --token 22ac74.4d061109507a992b 172.27.138.55:6443
10.244.0.0/16 comes from kube-flannel.yml which contains::
"Network": "10.244.0.0/16",
(Other) Operate the following commands::
kubeadm init
[..]
Your Kubernetes master has initialized successfully!
[..]
You can now join any number of machines by running the following on each node
as root:
kubeadm join --token 22ac74.4d061109507a992b 172.27.138.55:6443
#
The above output needs to be operated on kube-host01 to join into the cluster.
If using VirtualBox, need to specify the internal ip address like::
kubeadm init --apiserver-advertise-address 172.168.0.1
Operate the following commands::
$ sudo cp /etc/kubernetes/admin.conf $HOME/
$ sudo chown $(id -u):$(id -g) $HOME/admin.conf
$ export KUBECONFIG=$HOME/admin.conf
$ echo "export KUBECONFIG=$HOME/admin.conf" >> $HOME/.bashrc
(Flannel) Configure network setting for pod2pod communication::
$ wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
$ kubectl apply -f kube-flannel.yml
(Weave) Configure network setting for pod2pod communication::
$ kubectl apply -f https://git.io/weave-kube-1.6
Check the valid installation::
$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system etcd-kube-manager 1/1 Running 0 1h
kube-system kube-apiserver-kube-manager 1/1 Running 0 1h
kube-system kube-controller-manager-kube-manager 1/1 Running 0 1h
kube-system kube-dns-692378583-3gbgp 0/3 ContainerCreating 0 1h
kube-system kube-proxy-4rbvg 1/1 Running 0 1h
kube-system kube-scheduler-kube-manager 1/1 Running 0 1h
kube-system weave-net-cjf25 2/2 Running 0 51s
$
Confirm the STATUS becomes Ready::
$ kubectl get nodes
NAME STATUS AGE VERSION
k8s-master Ready 1m v1.7.3
Make the manager schedulable::
$ kubectl describe nodes | grep Tain
Taints: node-role.kubernetes.io/master:NoSchedule
$ kubectl taint nodes node-role.kubernetes.io/master:NoSchedule-
node "k8s-master" untainted
$ kubectl describe nodes | grep Tain
Taints:
$
Initialization of a k8s node
To make the mount propagation work proerly, edit /etc/systemd/system/multi-user.target.wants/docker.service like::
[Service]
- MountFlags=slave
- MountFlags=shared
This is required to pass e2e test "[sig-storage] CSI Volumes CSI plugin test using CSI driver: hostPath".
To add a node into k8s cluster, operate the following command on a node (not manager)::
kubeadm join --token 22ac74.4d061109507a992b 172.27.138.55:6443
Check the node joins into the cluster with the command on the manager::
$ kubectl get nodes
NAME STATUS AGE VERSION
kube-host01 Ready 51s v1.6.6
kube-manager Ready 1h v1.6.6
$
Retrive the way to add a node
Get a kubeadm token on k8s-master::
$ TOKEN=sudo kubeadm token list | grep authentication | awk '{print $1}'
$ echo $TOKEN
c3cf19.89e62945a88d7a91
If you cannot get a token, need to recreate with::
$ sudo kubeadm token create
Get a discovery token on k8s-master::
$ DISCOVERY_TOKEN=openssl x509 -pubkey \ -in /etc/kubernetes/pki/ca.crt | openssl rsa \ -pubin -outform der 2>/dev/null | openssl dgst \ -sha256 -hex | sed 's/^.* //'
$ echo $DISCOVERY_TOKEN
b3bb83c24673649bf1909e9144929a64569b1a7988df97323a9a3449c3b4c1e6
Get an endpoint on k8s-master::
$ ENDPOINT=cat admin.conf | grep server | sed s@" server: https://"@@
$ echo $ENDPOINT
192.168.1.105:6443
Use the token and the discovery token on k8s-node to add a new node on the node::
TOKEN=c3cf19.89e62945a88d7a91
DISCOVERY_TOKEN=b3bb83c24673649bf1909e9144929a64569b1a7988df97323a9a3449c3b4c1e6
ENDPOINT=192.168.1.105:6443
#
kubeadm join --token ${TOKEN} ${ENDPOINT} \
--discovery-token-ca-cert-hash sha256:${DISCOVERY_TOKEN}
Enable metrics-server for HPA
Install metrics-server on k8s-master::
$ git clone https://github.com/kubernetes-incubator/metrics-server
$ cd metrics-server/
$ kubectl create -f deploy/kubernetes/
Integrate standalone-cinder of the external cloud-provider-openstack for Dynamic Volume Provisioning
NOTE: It is not necessary to add options (--cloud-provider, --cloud-config) to kube-controller-manager and other processes at all.
Use manifests as samples from https://github.com/oomichi/try-kubernetes/tree/master/manifests/standalone-cinder-external
Add RBAC for standalone-cinder deployment::
$ kubectl create -f rbac.yaml
Change hostAliases, OS_AUTHURL and other OS*** env values of deployment.yaml for your environment.
Deploy standalone-cinder::
$ kubectl create -f deployment.yaml
Add default StorageClass::
$ kubectl create -f storage-class.yaml
Verify Dynamic Volume Provisioning works fine::
$ kubectl create -f pvc.yaml
$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
cinder-claim Bound pvc-af01ada4-9cf4-11e8-a146-fa163e420595 1Gi RWO gold 31s
$
Enable MetricsGrabber
Change the listening address of kube-scheduler to 0.0.0.0::
$ sudo vi /etc/kubernetes/manifests/kube-scheduler.yaml
--- ./kube-scheduler.yaml.orig 2018-08-22 02:30:16.060204589 +0000
+++ /etc/kubernetes/manifests/kube-scheduler.yaml 2018-08-22 02:30:38.160555932 +0000
@@ -13,7 +13,7 @@
containers:
- command:
- kube-scheduler
- --kubeconfig=/etc/kubernetes/scheduler.conf
- --leader-elect=true
image: k8s.gcr.io/kube-scheduler-amd64:v1.11.1
Change the listening address of kube-controller-manager to 0.0.0.0::
$ sudo vi /etc/kubernetes/manifests/kube-controller-manager.yaml
--- kube-controller-manager.yaml 2019-01-08 23:39:48.721525219 +0000
+++ /etc/kubernetes/manifests/kube-controller-manager.yaml 2019-01-08 23:40:09.266003213 +0000
@@ -13,7 +13,7 @@
containers:
- command:
- kube-controller-manager
- --allocate-node-cidrs=true
- --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf
- --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf
Enable StorageObjectInUseProtection plugin on admission controller
StorageObjectInUseProtection sets the PV protection finalizer flag before
a PV object appears on k8s API. "PV protection" e2e test requires this behavior.
Add StorageObjectInUseProtection to --enable-admission-plugins option::
$ sudo vi /etc/kubernetes/manifests/kube-apiserver.yaml
- --advertise-address=192.168.1.102
- --allow-privileged=true
- --client-ca-file=/etc/kubernetes/pki/ca.crt
-
- --enable-admission-plugins=NodeRestriction
-
- --enable-admission-plugins=NodeRestriction,StorageObjectInUseProtection
- --enable-bootstrap-token-auth=true
- --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
- --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
How to see REST API operation on kubectl command
Just specify '--v=8' option on kubectl command like::
$ kubectl --v=8 get nodes
[..] GET https://172.27.138.55:6443/api/v1/nodes
[..] Request Headers:
[..] Accept: application/json
[..] User-Agent: kubectl/v1.6.6 (linux/amd64) kubernetes/7fa1c17
[..] Response Status: 200 OK in 21 milliseconds
[..] Response Headers:
[..] Content-Type: application/json
[..] Date: Wed, 28 Jun 2017 00:33:39 GMT
[..] Response Body: {"kind":"NodeList","apiVersion":"v1",
"metadata":{"selfLink":"/api/v1/nodes","resourceVersion":"7254"},
"items":[{"metadata":{"name":"kube-host01","selfLink":"/api/v1/nodeskube-host01",
"uid":"a354969d-5b98-11e7-9e55-1866da463eb0",
"resourceVersion":"7244","creationTimestamp":"2017-06-28T00:27:59Z",
"labels":{"beta.kubernetes.io/arch":"amd64",
"beta.kubernetes.io/os":"linux",
"kubernetes.io/hostname":"kube-host01"} ..
How to know resource workload for each node
This requires metrics-server is enabled.
How to know resource work load for each node is::
$ kubectl top node
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
k8s-master 113m 5% 1271Mi 33%
k8s-node01 116m 5% 902Mi 23%
Run e2e test
Prepare
Need to install golang 1.13.7 which is not provided from ubuntu 18.04 as the
default. So we need to do the following process for that::
$ wget https://go.dev/dl/go1.18.2.linux-amd64.tar.gz
$ sudo tar -C /usr/local/ -xzf go1.18.2.linux-amd64.tar.gz
$ export PATH=$PATH:/usr/local/go/bin
$ echo "export PATH=$PATH:/usr/local/go/bin" >> $HOME/.bashrc
$ sudo ln -s /usr/local/go/bin/go /usr/local/bin/go
Set GOPATH as parmanent setting::
$ mkdir ${HOME}/go
$ echo "export GOPATH=${HOME}/go" >> ${HOME}/.bashrc
Install some building packages::
$ sudo apt-get install -y docker.io gcc make
Build e2e test binary
Download k8s source code::
$ go get k8s.io/test-infra
$ go get k8s.io/kubernetes
Check out the same version as the target k8s cluster::
$ cd $GOPATH/src/k8s.io/kubernetes
$ kubectl version
Client Version: version.Info{
Major:"1", Minor:"11", GitVersion:"v1.11.1",
GitCommit:"b1b29978270dc22fecc592ac55d903350454310a",
GitTreeState:"clean", BuildDate:"2018-07-17T18:53:20Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1",
GitCommit:"b1b29978270dc22fecc592ac55d903350454310a",
GitTreeState:"clean", BuildDate:"2018-07-17T18:43:26Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
$
$ git tag -l
v0.10.0
..
v1.11.1
..
$
$ git checkout refs/tags/v1.11.1
$ git checkout -b tag-v1.11.1
Build e2e test binary.
(NOTE: When changing the e2e code, we need to build the binary again to apply the changes)::
The docker daemon runs as root user, not docker user. So it is necessary to specify sudo
$ cd ${HOME}/go/src/k8s.io/test-infra/kubetest
$ GO111MODULE=on go install ./kubetest
$ export PATH=${PATH}:${HOME}/go/bin
$ cd ${HOME}/go/src/k8s.io/kubernetes
$ sudo ${HOME}/go/bin/kubetest --build
Run e2e test
Run e2e test::
$ export KUBECONFIG=$HOME/admin.conf
$ export KUBERNETES_CONFORMANCE_TEST=true
$ kubetest --provider=skeleton --test --test_args="--ginkgo.focus=[Conformance]"
Confirm which tests will run without actual tests (Need to specify provider to avoid an error)::
$ kubetest --provider=skeleton --test --test_args="--ginkgo.dryRun=true --ginkgo.focus=[Conformance]"
[..]
[k8s.io] Docker Containers
should use the image defaults if command and args are blank [Conformance]
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/docker_containers.go:35
~SS
[k8s.io] EmptyDir volumes
should support (non-root,0644,tmpfs) [Conformance] [sig-storage]
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/empty_dir.go:85
~SS
[sig-apps] ReplicaSet
should serve a basic image on each replica with a public image [Conformance]
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apps/replica_set.go:82
~S
[sig-network] Services
should provide secure master service [Conformance]
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/service.go:71
~
Ran 149 of 652 Specs in 0.072 seconds
SUCCESS! -- 0 Passed | 0 Failed | 0 Pending | 503 Skipped PASS
Ginkgo ran 1 suite in 519.123083ms
Test Suite Passed
2017/08/09 15:38:12 util.go:133: Step './hack/ginkgo-e2e.sh --ginkgo.dryRun=true --ginkgo.focus=[Conformance]' finished in 937.615925ms
2017/08/09 15:38:12 e2e.go:80: Done
$
Specify a single test with regex::
$ kubetest --provider=skeleton --test --test_args="--ginkgo.focus=1\spod\sto\s2\spods"
If changing e2e code, we need to specify --check-version-skew=false to skip checking versions of both server and e2e client::
$ kubetest --provider=skeleton --test --test_args="--ginkgo.focus=from\s3\sto\s5$" --check-version-skew=false
Setup dev env
Install bazel::
$ sudo apt-get install pkg-config zip g++ zlib1g-dev unzip python
$ wget https://github.com/bazelbuild/bazel/releases/download/0.21.0/bazel-0.21.0-installer-linux-x86_64.sh
$ chmod 755 bazel-0.21.0-installer-linux-x86_64.sh
$ ./bazel-0.21.0-installer-linux-x86_64.sh --user
$
$ export PATH="$PATH:$HOME/bin"
$ echo "export PATH=$PATH:$HOME/bin" >> $HOME/.bashrc
Run unit tests on kubernetes/test-infra::
$ bazel test //..
Run unit test
with make::
$ make test
with bazel::
$ bazel test //...
Helm & Spinnaker
Install Helm
As https://github.com/kubernetes/helm#install ::
$ wget https://storage.googleapis.com/kubernetes-helm/helm-v2.9.1-linux-amd64.tar.gz
$ tar -zxvf helm-v2.9.1-linux-amd64.tar.gz
$ sudo mv linux-amd64/helm /usr/local/bin/
$ helm init
Verify helm::
$ helm version
Client: &version.Version{SemVer:"v2.9.1", GitCommit:"20adb27c7c5868466912eebdf6664e7390ebe710", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.9.1", GitCommit:"20adb27c7c5868466912eebdf6664e7390ebe710", GitTreeState:"clean"}
$
Add permission to deploy tiller::
$ kubectl create serviceaccount --namespace kube-system tiller
$ kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
$ kubectl patch deploy --namespace kube-system tiller-deploy -p '{"spec":{"template":{"spec":{"serviceAccount":"tiller"}}}}'
Install Spinnaker
Install Spinnaker::
$ wget https://raw.githubusercontent.com/kubernetes/charts/master/stable/spinnaker/values.yaml
$ helm install -n kubelive -f values.yaml stable/spinnaker
Error: timed out waiting for the condition
$
$ helm ls --all kubelive
NAME REVISION UPDATED STATUS CHART NAMESPACE
kubelive 1 Tue May 15 21:36:52 2018 FAILED spinnaker-0.4.1 default
$
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
kubelive-create-bucket-j97wn 0/1 CrashLoopBackOff 5 10m
kubelive-jenkins-86bcb6c4b5-h4bqx 0/1 Pending 0 10m
kubelive-minio-5d78b95d9c-pkpss 0/1 Pending 0 10m
kubelive-redis-5667b84965-k4nmz 0/1 Pending 0 10m
kubelive-spinnaker-clouddriver-85997f4b64-q97qq 0/1 Running 0 10m
kubelive-spinnaker-deck-86c48f7594-vxmnt 1/1 Running 0 10m
kubelive-spinnaker-echo-8ccc9956c-prk58 1/1 Running 0 10m
kubelive-spinnaker-front50-6859bf64bb-cn9bd 0/1 CrashLoopBackOff 6 10m
kubelive-spinnaker-gate-5468cccbc7-n2ncw 0/1 CrashLoopBackOff 6 10m
$
$ kubectl logs kubelive-create-bucket-j97wn
mc: Unable to initialize new config from the provided credentials.
Get http://kubelive-minio:9000/probe-bucket-sign/?location=: dial tcp: lookup kubelive-minio on 10.96.0.10:53: no such host
$
Operate something
Sort instances with --sort-by
Easy one::
$ kubectl get pods -n=default
NAME READY STATUS RESTARTS AGE
pod-00 1/1 Running 0 51s
pod-01 1/1 Running 0 1m
pod-name 1/1 Running 0 18m
$
$ kubectl get pods --sort-by=.status.startTime -n=default
NAME READY STATUS RESTARTS AGE
pod-name 1/1 Running 0 18m
pod-01 1/1 Running 0 55s
pod-00 1/1 Running 0 42s
$
$ kubectl get pods --sort-by=.metadata.name -n=default
NAME READY STATUS RESTARTS AGE
pod-00 1/1 Running 0 2m
pod-01 1/1 Running 0 2m
pod-name 1/1 Running 0 20m
$
Create a pod
Easy one::
$ kubectl create -f manifests/pod-01.yaml
Create a pod with some changes by edit without any chages of the original manifest file::
$ kubectl create -f manifests/pod-01.yaml --edit -o json
Create a deployment
Create a deployment with external network access::
$ kubectl run nginx --image nginx --replicas=3
$ kubectl expose deployment nginx --port=80 --target-port=80
$ kubectl create -f manifests/ingress-nginx.yaml
$ kubectl describe ingress
Name: test-ingress
Namespace: default
Address:
Default backend: nginx:80 (10.244.0.25:80,10.244.0.26:80,10.244.0.27:80)
Rules:
Host Path Backends
On this environment, ingress-nginx-controller is used and the setting is::
$ kubectl get services -n ingress-nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default-http-backend ClusterIP 10.102.0.178 80/TCP 2h
ingress-nginx NodePort 10.101.145.191 80:31454/TCP,443:31839/TCP 2h
$
So NodePort is configured and the host's 31454/TCP is proxied to 80/TCP of the ingress.
You can get nginx page like::
$ curl http://localhost:31454
<!DOCTYPE html>
Welcome to nginx!
..
Create a snapshot of etcd
-------------------------
On this environment, etcd is running as a pod on kube-system namespace::
$ kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
etcd-k8s-v109-flannel-master 1/1 Running 0 1d
..
$
The manifest is /etc/kubernetes/manifests/etcd.yaml and we can see the endpoint (http://127.0.0.1:2379) in this case::
$ sudo cat /etc/kubernetes/manifests/etcd.yaml
..
- command:
- etcd
- --data-dir=/var/lib/etcd
- --listen-client-urls=http://127.0.0.1:2379
- --advertise-client-urls=http://127.0.0.1:2379
..
Install etcdctl command (The ubuntu package is too old and doesn't support the snapshot feature)::
$ mkdir foo
$ cd foo
$ wget https://github.com/coreos/etcd/releases/download/v3.2.18/etcd-v3.2.18-linux-amd64.tar.gz
$ tar -zxvf etcd-v3.2.18-linux-amd64.tar.gz
$ cd etcd-v3.2.18-linux-amd64
Create a snapshot::
$ ETCDCTL_API=3 ./etcdctl --endpoints http://127.0.0.1:2379 snapshot save snapshot.db
Snapshot saved at snapshot.db
$
Make a node tainted and pods go away from the node
--------------------------------------------------
Check pods where live and the node::
$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
nginx-foo-74cd78d68f-4jwsq 1/1 Running 0 1m 10.244.0.30 k8s-v109-flannel-master
nginx-foo-74cd78d68f-5jl55 1/1 Running 0 1m 10.244.1.7 k8s-v109-flannel-worker
nginx-foo-74cd78d68f-9cts2 1/1 Running 0 1m 10.244.1.5 k8s-v109-flannel-worker
nginx-foo-74cd78d68f-9gtwx 1/1 Running 0 1m 10.244.1.6 k8s-v109-flannel-worker
nginx-foo-74cd78d68f-b7zmx 1/1 Running 0 1m 10.244.1.4 k8s-v109-flannel-worker
nginx-foo-74cd78d68f-d97pw 1/1 Running 0 1m 10.244.0.29 k8s-v109-flannel-master
nginx-foo-74cd78d68f-j27qf 1/1 Running 0 1m 10.244.0.28 k8s-v109-flannel-master
nginx-foo-74cd78d68f-j45c8 1/1 Running 0 1m 10.244.1.2 k8s-v109-flannel-worker
nginx-foo-74cd78d68f-l4mwq 1/1 Running 0 1m 10.244.0.31 k8s-v109-flannel-master
nginx-foo-74cd78d68f-wnb4c 1/1 Running 0 1m 10.244.1.3 k8s-v109-flannel-worker
$
$ kubectl describe node k8s-v109-flannel-worker | grep Taints
Taints:
$
Even if making the node tainted with NoSchedule, the pods still exist in the node::
$ kubectl taint nodes k8s-v109-flannel-worker key=value:NoSchedule
node "k8s-v109-flannel-worker" tainted
$ kubectl describe node k8s-v109-flannel-worker | grep Taints
Taints: key=value:NoSchedule
$
$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
nginx-foo-74cd78d68f-4jwsq 1/1 Running 0 5m 10.244.0.30 k8s-v109-flannel-master
nginx-foo-74cd78d68f-5jl55 1/1 Running 0 5m 10.244.1.7 k8s-v109-flannel-worker
nginx-foo-74cd78d68f-9cts2 1/1 Running 0 5m 10.244.1.5 k8s-v109-flannel-worker
nginx-foo-74cd78d68f-9gtwx 1/1 Running 0 5m 10.244.1.6 k8s-v109-flannel-worker
nginx-foo-74cd78d68f-b7zmx 1/1 Running 0 5m 10.244.1.4 k8s-v109-flannel-worker
nginx-foo-74cd78d68f-d97pw 1/1 Running 0 5m 10.244.0.29 k8s-v109-flannel-master
nginx-foo-74cd78d68f-j27qf 1/1 Running 0 5m 10.244.0.28 k8s-v109-flannel-master
nginx-foo-74cd78d68f-j45c8 1/1 Running 0 5m 10.244.1.2 k8s-v109-flannel-worker
nginx-foo-74cd78d68f-l4mwq 1/1 Running 0 5m 10.244.0.31 k8s-v109-flannel-master
nginx-foo-74cd78d68f-wnb4c 1/1 Running 0 5m 10.244.1.3 k8s-v109-flannel-worker
$
After making the node tainted with NoExecute, the pods go away from the node::
$ kubectl taint nodes k8s-v109-flannel-worker key=value:NoExecute
node "k8s-v109-flannel-worker" tainted
$ kubectl describe node k8s-v109-flannel-worker | grep Taints
Taints: key=value:NoExecute
$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
nginx-foo-74cd78d68f-48q4p 1/1 Running 0 17s 10.244.0.37 k8s-v109-flannel-master
nginx-foo-74cd78d68f-4jwsq 1/1 Running 0 8m 10.244.0.30 k8s-v109-flannel-master
nginx-foo-74cd78d68f-9q6f8 1/1 Running 0 17s 10.244.0.34 k8s-v109-flannel-master
nginx-foo-74cd78d68f-d97pw 1/1 Running 0 8m 10.244.0.29 k8s-v109-flannel-master
nginx-foo-74cd78d68f-j27qf 1/1 Running 0 8m 10.244.0.28 k8s-v109-flannel-master
nginx-foo-74cd78d68f-jlxng 1/1 Running 0 17s 10.244.0.36 k8s-v109-flannel-master
nginx-foo-74cd78d68f-k5rl9 1/1 Running 0 17s 10.244.0.32 k8s-v109-flannel-master
nginx-foo-74cd78d68f-l4mwq 1/1 Running 0 8m 10.244.0.31 k8s-v109-flannel-master
nginx-foo-74cd78d68f-sg52l 1/1 Running 0 17s 10.244.0.33 k8s-v109-flannel-master
nginx-foo-74cd78d68f-vzspf 1/1 Running 0 17s 10.244.0.35 k8s-v109-flannel-master
$
Remove the taint after this try::
$ kubectl taint nodes k8s-v109-flannel-worker key-
Create a secret and use it from a pod
-------------------------------------
Encode a plain password with base64::
$ echo -n "mypassword" | base64
bXlwYXNzd29yZA==
$
Create a secret::
$ cat manifests/secret-01.yaml
apiVersion: v1
kind: Secret
metadata:
name: secret-01
type: Opaque
data:
password: bXlwYXNzd29yZA==
$
$ kubectl create -f manifests/secret-01.yaml
Create a pod with the secret as a file::
$ kubectl create -f manifests/pod-using-secret-as-file.yaml
Confirm the password in the pod::
$ kubectl exec -it pod-using-secret-as-file /bin/bash
(login the pod)
#
# ls /etc/foo/
password
# cat /etc/foo/password
mypassword
Create a pod with the secret as a variable::
$ kubectl create -f manifests/pod-using-secret-as-variable.yaml
Confirm the password in the pod::
$ kubectl exec -it pod-using-secret-as-variable /bin/bash
(login the pod)
#
# echo $SECRET_PASSWORD
mypassword
Rolling-upgrade for a deployment
--------------------------------
Create a deployment with a little old nginx (v1.7.9)::
$ kubectl create -f manifests/nginx-deployment.yaml
$ kubectl describe deployment/nginx-deployment | grep Image
Image: nginx:1.7.9
$
Check the strategy (in this case (the default), that is RollingUpdate and the upgrade happens immediately just after setting the image)::
$ kubectl describe deployment/nginx-deployment | grep StrategyType
StrategyType: RollingUpdate
$
Check the ReplicaSet name and the pod names::
$ kubectl get rs
NAME DESIRED CURRENT READY AGE
nginx-deployment-75675f5897 3 3 3 6s
$
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-deployment-75675f5897-9mhmv 1/1 Running 0 36s
nginx-deployment-75675f5897-kpgtr 1/1 Running 0 36s
nginx-deployment-75675f5897-plq92 1/1 Running 0 36s
$
Set a newer nginx image (v1.9.1)::
$ kubectl set image deployment/nginx-deployment nginx=nginx:1.9.1
$ kubectl describe deployment/nginx-deployment | grep Image
Image: nginx:1.9.1
$
Then check the status of the upgrade::
$ kubectl rollout status deployment/nginx-deployment
Waiting for rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for rollout to finish: 2 old replicas are pending termination...
Waiting for rollout to finish: 1 old replicas are pending termination...
Waiting for rollout to finish: 1 old replicas are pending termination...
deployment "nginx-deployment" successfully rolled out
$
Conform new created ReplicaSet and pods. The old ReplicaSet doesn't have
any pods now and new pods only exist::
$ kubectl get rs
NAME DESIRED CURRENT READY AGE
nginx-deployment-75675f5897 0 0 0 3m
nginx-deployment-c4747d96c 3 3 3 1m
$
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-deployment-c4747d96c-fbsw6 1/1 Running 0 2m
nginx-deployment-c4747d96c-gvqg2 1/1 Running 0 1m
nginx-deployment-c4747d96c-jfvvl 1/1 Running 0 1m
$
Rolling-back of a deployment
----------------------------
Check the history of a deployment::
$ kubectl rollout history deployment/nginx-deployment
deployments "nginx-deployment"
REVISION CHANGE-CAUSE
1
2
$
Show the detail of each revision::
$ kubectl rollout history deployment/nginx-deployment --revision=2
deployments "nginx-deployment" with revision #2
Pod Template:
Labels: app=nginx
pod-template-hash=1520898311
Containers:
nginx:
Image: nginx:1.9.1
Port: 80/TCP
Environment:
Mounts:
Volumes:
$
$ kubectl rollout history deployment/nginx-deployment --revision=1
deployments "nginx-deployment" with revision #1
Pod Template:
Labels: app=nginx
pod-template-hash=2710681425
Containers:
nginx:
Image: nginx:1.7.9
Port: 80/TCP
Environment:
Mounts:
Volumes:
$
Rolling-back the deployment::
$ kubectl rollout undo deployment/nginx-deployment
Confirm the rolling-back succeeded::
$ kubectl rollout history deployment/nginx-deployment
deployments "nginx-deployment"
REVISION CHANGE-CAUSE
2
3
$ kubectl rollout history deployment/nginx-deployment --revision=3
deployments "nginx-deployment" with revision #3
Pod Template:
Labels: app=nginx
pod-template-hash=2710681425
Containers:
nginx:
Image: nginx:1.7.9
Port: 80/TCP
Environment:
Mounts:
Volumes:
$
$ kubectl describe deployment/nginx-deployment | grep Image
Image: nginx:1.7.9
$
Verify DNS works for Services and Pods
--------------------------------------
https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/
Check what service works on the cluster::
$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 443/TCP 2d
nginx-deployment ClusterIP 10.99.52.90 80/TCP 24s
$
Create a pod for verifying DNS works::
$ kubectl create -f manifests/pod-busybox.yaml
$ kubectl exec -it pod-busybox sh
(login the pod)
wget http://nginx-deployment
Connecting to nginx-deployment (10.99.52.90:80)
index.html 100% |********************************************************************************************************************************************| 612 0:00:00 ETA
/ # cat index.html
Welcome to nginx!
..
#
As the above, DNS works fine and the service nginx-deployment can be looked up from a pod as the same name.
A pod also can be looked up by "pod-ip-address.my-namespace.pod.cluster.local" like::
$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
pod-01 1/1 Running 0 19m 10.244.0.48 k8s-v109-flannel-master
$ kubectl exec -it pod-busybox sh
/ #
/ # ping 10-244-0-48.default.pod.cluster.local
PING 10-244-0-48.default.pod.cluster.local (10.244.0.48): 56 data bytes
64 bytes from 10.244.0.48: seq=0 ttl=64 time=0.033 ms
64 bytes from 10.244.0.48: seq=1 ttl=64 time=0.064 ms
Use init-containers
-------------------
https://kubernetes.io/docs/concepts/workloads/pods/init-containers/
Create a pod with init-containers::
$ kubectl create -f manifests/pod-init-container.yaml
Check the pod status, it waits for end of init process::
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
pod-init-container 0/1 Init:0/2 0 30s
$
Check logs of each containers, init-containers start on the order of the manifest. That means 2nd init-container also wait for 1st one's finishes::
$ kubectl logs pod-init-container -c myapp-container
Error from server (BadRequest): container "myapp-container" in pod "pod-init-container" is waiting to start: PodInitializing
$
$ kubectl logs pod-init-container -c init-myservice
waiting for myservice
nslookup: can't resolve 'myservice'
Server: 10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
waiting for myservice
nslookup: can't resolve 'myservice'
Server: 10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
waiting for myservice
$
$ kubectl logs pod-init-container -c init-mydb
Error from server (BadRequest): container "init-mydb" in pod "pod-init-container" is waiting to start: PodInitializing
$
Create services for making end of init process::
$ kubectl create -f manifests/services-for-init-containers.yaml
service "myservice" created
service "mydb" created
$
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
pod-init-container 0/1 PodInitializing 0 4m
$
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
pod-init-container 1/1 Running 0 5m
$
Then the pod outputs the message to show the end as its command in the manifest::
$ kubectl logs pod-init-container
The app is running!
$
Create a DaemonSet
------------------
Create a daemonset::
$ kubectl create -f manifests/daemonset.yaml
Check the existence::
$ kubectl get ds -n kube-system
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
fluentd-elasticsearch 1 1 1 1 1 1m
..
$
Troubleshooting
===============
(Non-recommended way) Enforce kubelet boot on an environment with swap::
$ sudo diff -u /etc/systemd/system/kubelet.service.d/10-kubeadm.conf.orig /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
sudo: unable to resolve host k8s-v109-flannel-worker
--- /etc/systemd/system/kubelet.service.d/10-kubeadm.conf.orig 2018-04-05 21:28:10.278748887 +0000
+++ /etc/systemd/system/kubelet.service.d/10-kubeadm.conf 2018-04-05 21:32:14.191449307 +0000
@@ -6,5 +6,6 @@
Environment="KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt"
Environment="KUBELET_CADVISOR_ARGS=--cadvisor-port=0"
Environment="KUBELET_CERTIFICATE_ARGS=--rotate-certificates=true --cert-dir=/var/lib/kubelet/pki"
+Environment="KUBELET_SWAP_ARGS=--fail-swap-on=false"
ExecStart=
-ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CADVISOR_ARGS $KUBELET_CERTIFICATE_ARGS $KUBELET_EXTRA_ARGS
+ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CADVISOR_ARGS $KUBELET_CERTIFICATE_ARGS $KUBELET_EXTRA_ARGS $KUBELET_SWAP_ARGS
$
$ sudo reboot
Swapoff on lxcfs (lxcfs is a simple file system to implement nest-cgroup
for systemd environments which are defact init of Linux kernel today)::
$ diff -u /usr/share/lxcfs/lxc.mount.hook.orig /usr/share/lxcfs/lxc.mount.hook
--- /usr/share/lxcfs/lxc.mount.hook.orig 2018-04-05 21:55:21.626302043 +0000
+++ /usr/share/lxcfs/lxc.mount.hook 2018-04-05 21:57:05.956673664 +0000
@@ -7,6 +7,7 @@
if [ -d /var/lib/lxcfs/proc/ ]; then
for entry in /var/lib/lxcfs/proc/*; do
[ -e "${LXC_ROOTFS_MOUNT}/proc/$(basename $entry)" ] || continue
+ [ $entry != "swap" ] || continue
mount -n --bind $entry ${LXC_ROOTFS_MOUNT}/proc/$(basename $entry)
done
fi