ymmt2005
commented
3 years ago Test steps and results in a dctest environment. Result: SUCCESS
- Prepare coil v2
$ cd cybozu-go/coil/coil-migrator
$ go build .
$ cd ../v2
$ make certs- bootstrap dctest and copy coil v2 to
boot-0
$ cd cybozu-go/neco/dctest
$ make placemat
$ make test SUITE=bootstrap
$ ./dcscp .../coil/v2/coil-migrator/coil-migrator boot-0:
$ ./dcscp -r .../coil/v2 boot-0:
$ ./dcssh boot-0- Prepare migration
boot-0:~$ ckecli kubernetes issue --ttl=3h > $HOME/.kube/config
boot-0:~$ ckecli etcd root-issue --output=file
cert files: etcd-ca.crt etcd-root.crt etcd-root.key
boot-0:~$ kubectl run pod1 --image=quay.io/cybozu/testhttpd:0
boot-0:~$ kubectl run pod2 --image=quay.io/cybozu/testhttpd:0
boot-0:~$ kubectl run pod3 --image=quay.io/cybozu/testhttpd:0
boot-0:~$ kubectl run pod4 --image=quay.io/cybozu/testhttpd:0
boot-0:~$ kubectl get pods -A -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE
default pod1 1/1 Running 0 19s 10.64.0.128 10.69.0.197
default pod2 1/1 Running 0 16s 10.64.0.160 10.69.0.196
default pod3 1/1 Running 0 12s 10.64.0.33 10.69.2.68
default pod4 1/1 Running 0 8s 10.64.0.1 10.69.0.4
internet-egress squid-75c5d8bdbf-drtwk 2/2 Running 0 7m17s 172.19.0.0 10.69.0.197
internet-egress squid-75c5d8bdbf-p8tnv 2/2 Running 0 7m17s 172.19.0.2 10.69.0.4
internet-egress unbound-559dfc758f-7j2hr 1/1 Running 0 7m17s 172.19.0.3 10.69.0.196
internet-egress unbound-559dfc758f-jn8dl 1/1 Running 0 7m17s 172.19.0.1 10.69.2.68
kube-system cluster-dns-89b9b9b49-q7ffn 1/1 Running 0 7m18s 10.64.0.0 10.69.0.4
kube-system cluster-dns-89b9b9b49-xqvcz 1/1 Running 0 7m18s 10.64.0.32 10.69.2.68
kube-system coil-controllers-6766cfdcc8-vm4k9 1/1 Running 0 7m16s 10.69.0.196 10.69.0.196
kube-system coil-node-9jgs7 1/1 Running 0 7m34s 10.69.0.196 10.69.0.196
kube-system coil-node-ghlgw 1/1 Running 0 7m34s 10.69.2.68 10.69.2.68
kube-system coil-node-h9vv4 1/1 Running 0 7m34s 10.69.0.197 10.69.0.197
kube-system coil-node-nbd74 1/1 Running 0 7m34s 10.69.0.4 10.69.0.4
kube-system coil-node-qhf62 1/1 Running 0 7m34s 10.69.0.6 10.69.0.6
kube-system node-dns-2w2wc 2/2 Running 0 7m41s 10.69.0.196 10.69.0.196
kube-system node-dns-6ls89 2/2 Running 0 7m41s 10.69.0.6 10.69.0.6
kube-system node-dns-gd4vg 2/2 Running 0 7m41s 10.69.0.4 10.69.0.4
kube-system node-dns-mz6ll 2/2 Running 0 7m41s 10.69.0.197 10.69.0.197
kube-system node-dns-slgh4 2/2 Running 0 7m41s 10.69.2.68 10.69.2.68- Remove Coil resources from CKE
boot-0:~$ ckecli resource delete /usr/share/neco/coil-deploy.yml
boot-0:~$ ckecli resource delete /usr/share/neco/coil-rbac.yml
boot-0:~$ ckecli resource list
ClusterRole/psp:restricted
ClusterRoleBinding/default:psp:restricted
ConfigMap/internet-egress/local-unbound
ConfigMap/internet-egress/squid
ConfigMap/internet-egress/unbound
Deployment/internet-egress/squid
Deployment/internet-egress/unbound
Namespace/internet-egress
PodDisruptionBudget/internet-egress/squid-pdb
PodDisruptionBudget/internet-egress/unbound-pdb
PodSecurityPolicy/coil
PodSecurityPolicy/restricted
PodSecurityPolicy/squid
Role/internet-egress/psp:squid
RoleBinding/internet-egress/squid:psp:squid
Service/internet-egress/squid
Service/internet-egress/unbound
ServiceAccount/internet-egress/squid
ServiceAccount/internet-egress/unboundPodSecurityPolicy/coil was defined separately, so it remains.
Coil v2 names PSP differently, so this will not cause conflicts.
- Run
coil-migrator dump
boot-0:~$ CPNODE=$(kubectl get nodes | grep master | head -1 | awk '{print $1}')
boot-0:~$ echo $CPNODE
10.69.0.196
boot-0:~$ ./coil-migrator dump --etcd-tls-ca=etcd-ca.crt --etcd-tls-cert=etcd-root.crt --etcd-tls-key=etcd-root.key --etcd-endpoints=$CPNODE:2379 > data.yaml
uninstalling Coil v1 ...
deleted *v1.DaemonSet coil-node
deleted *v1.Deployment coil-controllers
deleted *v1.ConfigMap coil-config
deleted *v1.ClusterRoleBinding coil-node
deleted *v1.ClusterRoleBinding coil-controllers
deleted *v1.ClusterRole coil-node
deleted *v1.ClusterRole coil-controllers
deleted *v1.ServiceAccount coil-node
deleted *v1.ServiceAccount coil-controller
annotated namespace internet-egressThis generated the following YAML:
---
apiVersion: coil.cybozu.com/v2
kind: AddressPool
metadata:
creationTimestamp: null
name: internet-egress
spec:
blockSizeBits: 0
subnets:
- ipv4: 172.19.0.0/28
---
apiVersion: coil.cybozu.com/v2
kind: AddressPool
metadata:
creationTimestamp: null
name: default
spec:
blockSizeBits: 5
subnets:
- ipv4: 10.64.0.0/14
---
apiVersion: coil.cybozu.com/v2
index: 5
ipv4: 10.64.0.160/27
kind: AddressBlock
metadata:
creationTimestamp: null
labels:
coil.cybozu.com/node: 10.69.0.196
coil.cybozu.com/pool: default
coil.cybozu.com/reserved: "true"
name: default-5-v1
---
apiVersion: coil.cybozu.com/v2
index: 3
ipv4: 172.19.0.3/32
kind: AddressBlock
metadata:
creationTimestamp: null
labels:
coil.cybozu.com/node: 10.69.0.196
coil.cybozu.com/pool: internet-egress
coil.cybozu.com/reserved: "true"
name: internet-egress-3-v1
---
apiVersion: coil.cybozu.com/v2
index: 4
ipv4: 10.64.0.128/27
kind: AddressBlock
metadata:
creationTimestamp: null
labels:
coil.cybozu.com/node: 10.69.0.197
coil.cybozu.com/pool: default
coil.cybozu.com/reserved: "true"
name: default-4-v1
---
apiVersion: coil.cybozu.com/v2
index: 0
ipv4: 172.19.0.0/32
kind: AddressBlock
metadata:
creationTimestamp: null
labels:
coil.cybozu.com/node: 10.69.0.197
coil.cybozu.com/pool: internet-egress
coil.cybozu.com/reserved: "true"
name: internet-egress-0-v1
---
apiVersion: coil.cybozu.com/v2
index: 0
ipv4: 10.64.0.0/27
kind: AddressBlock
metadata:
creationTimestamp: null
labels:
coil.cybozu.com/node: 10.69.0.4
coil.cybozu.com/pool: default
coil.cybozu.com/reserved: "true"
name: default-0-v1
---
apiVersion: coil.cybozu.com/v2
index: 2
ipv4: 172.19.0.2/32
kind: AddressBlock
metadata:
creationTimestamp: null
labels:
coil.cybozu.com/node: 10.69.0.4
coil.cybozu.com/pool: internet-egress
coil.cybozu.com/reserved: "true"
name: internet-egress-2-v1
---
apiVersion: coil.cybozu.com/v2
index: 1
ipv4: 10.64.0.32/27
kind: AddressBlock
metadata:
creationTimestamp: null
labels:
coil.cybozu.com/node: 10.69.2.68
coil.cybozu.com/pool: default
coil.cybozu.com/reserved: "true"
name: default-1-v1
---
apiVersion: coil.cybozu.com/v2
index: 1
ipv4: 172.19.0.1/32
kind: AddressBlock
metadata:
creationTimestamp: null
labels:
coil.cybozu.com/node: 10.69.2.68
coil.cybozu.com/pool: internet-egress
coil.cybozu.com/reserved: "true"
name: internet-egress-1-v1- Apply Coil v2 CRDs
boot-0:~$ cd v2/config/crd/
boot-0:~/v2/config/crd$ kustomize build . | kubectl apply -f -
customresourcedefinition.apiextensions.k8s.io/addressblocks.coil.cybozu.com created
customresourcedefinition.apiextensions.k8s.io/addresspools.coil.cybozu.com created
customresourcedefinition.apiextensions.k8s.io/blockrequests.coil.cybozu.com created
customresourcedefinition.apiextensions.k8s.io/egresses.coil.cybozu.com created
boot-0:~/v2/config/crd$ cd- Apply
data.yaml
boot-0:~$ kubectl apply -f data.yaml
addresspool.coil.cybozu.com/internet-egress created
addresspool.coil.cybozu.com/default created
addressblock.coil.cybozu.com/default-5-v1 created
addressblock.coil.cybozu.com/internet-egress-3-v1 created
addressblock.coil.cybozu.com/default-4-v1 created
addressblock.coil.cybozu.com/internet-egress-0-v1 created
addressblock.coil.cybozu.com/default-0-v1 created
addressblock.coil.cybozu.com/internet-egress-2-v1 created
addressblock.coil.cybozu.com/default-1-v1 created
addressblock.coil.cybozu.com/internet-egress-1-v1 created- Edit Coil v2
kustomization.yamlandnetconf.json, then apply it
Enable PSP, calico compatibility, the image tag, and set MTU for dctest environment.
boot-0:~$ cd v2
boot-0:~/v2$ sed -i -E 's,^#(- config/default/pod_security_policy.yaml),\1, ; s,^#(- config/pod/compat_calico.yaml),\1, ; s,newTag:.*$,newTag: test,' kustomization.yaml
boot-0:~/v2$ cat >netconf.json <<'EOF'
{
"cniVersion": "0.4.0",
"name": "k8s-pod-network",
"plugins": [
{
"type": "coil",
"socket": "/run/coild.sock"
},
{
"type": "tuning",
"mtu": 1400
},
{
"type": "portmap",
"capabilities": {
"portMappings": true
}
}
]
}
EOF
boot-0:~/v2$ kustomize build . | kubectl apply -f -
customresourcedefinition.apiextensions.k8s.io/addressblocks.coil.cybozu.com configured
customresourcedefinition.apiextensions.k8s.io/addresspools.coil.cybozu.com configured
customresourcedefinition.apiextensions.k8s.io/blockrequests.coil.cybozu.com configured
customresourcedefinition.apiextensions.k8s.io/egresses.coil.cybozu.com configured
serviceaccount/coil-controller created
serviceaccount/coil-router created
serviceaccount/coild created
podsecuritypolicy.policy/coil-controller created
podsecuritypolicy.policy/coil-egress created
podsecuritypolicy.policy/coil-router created
podsecuritypolicy.policy/coild created
role.rbac.authorization.k8s.io/coil-leader-election created
clusterrole.rbac.authorization.k8s.io/coil-controller created
clusterrole.rbac.authorization.k8s.io/coil-egress created
clusterrole.rbac.authorization.k8s.io/coil-router created
clusterrole.rbac.authorization.k8s.io/coild created
clusterrole.rbac.authorization.k8s.io/coilv2-addressblock-viewer-role created
clusterrole.rbac.authorization.k8s.io/coilv2-addresspool-viewer-role created
clusterrole.rbac.authorization.k8s.io/coilv2-blockrequest-viewer-role created
clusterrole.rbac.authorization.k8s.io/coilv2-egress-viewer-role created
clusterrole.rbac.authorization.k8s.io/psp-coil-controller created
clusterrole.rbac.authorization.k8s.io/psp-coil-egress created
clusterrole.rbac.authorization.k8s.io/psp-coil-router created
clusterrole.rbac.authorization.k8s.io/psp-coild created
rolebinding.rbac.authorization.k8s.io/coil-leader-election created
clusterrolebinding.rbac.authorization.k8s.io/coil-controller created
clusterrolebinding.rbac.authorization.k8s.io/coil-egress created
clusterrolebinding.rbac.authorization.k8s.io/coil-router created
clusterrolebinding.rbac.authorization.k8s.io/coild created
clusterrolebinding.rbac.authorization.k8s.io/psp-coil-controller created
clusterrolebinding.rbac.authorization.k8s.io/psp-coil-egress created
clusterrolebinding.rbac.authorization.k8s.io/psp-coil-router created
clusterrolebinding.rbac.authorization.k8s.io/psp-coild created
configmap/coil-config-22b825t954 created
secret/coilv2-webhook-server-cert created
service/coilv2-webhook-service created
deployment.apps/coil-controller created
daemonset.apps/coild created
mutatingwebhookconfiguration.admissionregistration.k8s.io/coilv2-mutating-webhook-configuration created
validatingwebhookconfiguration.admissionregistration.k8s.io/coilv2-validating-webhook-configuration created- Run a new pod to see Coil v2 is working
boot-0:~/v2$ kubectl run newpod1 --image=quay.io/cybozu/testhttpd:0
boot-0:~/v2$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
newpod1 1/1 Running 0 5s 10.64.0.64 10.69.2.68
pod1 1/1 Running 0 10m 10.64.0.128 10.69.0.197
pod2 1/1 Running 0 10m 10.64.0.160 10.69.0.196
pod3 1/1 Running 0 10m 10.64.0.33 10.69.2.68
pod4 1/1 Running 0 10m 10.64.0.1 10.69.0.4- Run
coil-migrator restart
boot-0:~$ ./coil-migrator replace
DO YOU WANT TO PROCEED? [y/N] y
Deleting pods on node 10.69.0.4...
deleting default/pod4
deleting internet-egress/squid-75c5d8bdbf-p8tnv
deleting kube-system/cluster-dns-89b9b9b49-q7ffn
Deleting pods on node 10.69.2.68...
deleting default/newpod1
deleting default/pod3
deleting internet-egress/unbound-559dfc758f-jn8dl
deleting kube-system/cluster-dns-89b9b9b49-xqvcz
Deleting pods on node 10.69.0.197...
deleting default/pod1
deleting internet-egress/squid-75c5d8bdbf-drtwk
Deleting pods on node 10.69.0.196...
deleting default/pod2
deleting internet-egress/unbound-559dfc758f-7j2hr
deleting reserved address blocks- Check remaining address blocks
boot-0:~$ kubectl get addressblocks
NAME NODE POOL IPV4 IPV6
default-2 10.69.0.197 default 10.64.0.64/27
default-3 10.69.0.4 default 10.64.0.96/27
internet-egress-4 10.69.2.68 internet-egress 172.19.0.4/32
internet-egress-5 10.69.0.4 internet-egress 172.19.0.5/32
internet-egress-6 10.69.0.4 internet-egress 172.19.0.6/32
internet-egress-7 10.69.2.68 internet-egress 172.19.0.7/32
This PR elaborates on the migration steps and implements the necessary tools and changes. Specifically,
coil-migratoris implemented.