nickatnceas
commented
1 year ago I added the two new VMs to the K8s-dev cluster and pushed all running pods to them. Everything appears to be working properly.
outin@halt-29904:~$ kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
docker-dev-ucsb-1 Ready control-plane,master 2y256d v1.23.3 128.111.85.190 <none> Ubuntu 18.04.6 LTS 5.4.0-126-generic docker://20.10.6
docker-dev-ucsb-2 Ready,SchedulingDisabled <none> 2y256d v1.23.3 128.111.85.191 <none> Ubuntu 18.04.6 LTS 5.4.0-126-generic docker://20.10.6
k8s-dev-node-2 Ready <none> 37m v1.23.4 128.111.85.213 <none> Ubuntu 22.04.1 LTS 5.15.0-50-generic docker://20.10.12
k8s-dev-node-3 Ready <none> 5m21s v1.23.4 128.111.85.227 <none> Ubuntu 22.04.1 LTS 5.15.0-50-generic docker://20.10.12
outin@halt-29904:~$ kubectl get pods -A -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
bookkeeper bookkeeper-589b4559b-595qj 1/1 Running 0 28m 192.168.19.199 k8s-dev-node-2 <none> <none>
bookkeeper postgres-56dbc7b9c7-6tj8c 1/1 Running 0 28m 192.168.19.202 k8s-dev-node-2 <none> <none>
ceph-csi-cephfs ceph-csi-cephfs-csi-cephfsplugin-5x7s4 3/3 Running 0 38m 128.111.85.213 k8s-dev-node-2 <none> <none>
ceph-csi-cephfs ceph-csi-cephfs-csi-cephfsplugin-gmhzg 3/3 Running 7 (6d3h ago) 356d 128.111.85.191 docker-dev-ucsb-2 <none> <none>
ceph-csi-cephfs ceph-csi-cephfs-csi-cephfsplugin-tsqgc 3/3 Running 20 (6d2h ago) 356d 128.111.85.190 docker-dev-ucsb-1 <none> <none>
ceph-csi-cephfs ceph-csi-cephfs-csi-cephfsplugin-vrvc6 3/3 Running 0 7m7s 128.111.85.227 k8s-dev-node-3 <none> <none>
ceph-csi-cephfs ceph-csi-cephfs-provisioner-5f465c8b64-cnxgl 6/6 Running 0 356d 192.168.19.194 k8s-dev-node-2 <none> <none>
ceph-csi-cephfs ceph-csi-cephfs-provisioner-5f465c8b64-dt8m4 6/6 Running 5568 (6d2h ago) 356d 192.168.3.148 docker-dev-ucsb-1 <none> <none>
ceph-csi-cephfs ceph-csi-cephfs-provisioner-5f465c8b64-m2nws 6/6 Running 0 28m 192.168.73.194 k8s-dev-node-3 <none> <none>
ceph-csi-rbd ceph-csi-rbd-csi-cephfsplugin-7rhj6 3/3 Running 20 (6d2h ago) 345d 128.111.85.190 docker-dev-ucsb-1 <none> <none>
ceph-csi-rbd ceph-csi-rbd-csi-cephfsplugin-ppqvl 3/3 Running 0 38m 128.111.85.213 k8s-dev-node-2 <none> <none>
ceph-csi-rbd ceph-csi-rbd-csi-cephfsplugin-t49pn 3/3 Running 7 (6d3h ago) 345d 128.111.85.191 docker-dev-ucsb-2 <none> <none>
ceph-csi-rbd ceph-csi-rbd-csi-cephfsplugin-wmpdw 3/3 Running 0 7m7s 128.111.85.227 k8s-dev-node-3 <none> <none>
ceph-csi-rbd ceph-csi-rbd-provisioner-6b9f54d9df-4fh64 7/7 Running 0 345d 192.168.19.193 k8s-dev-node-2 <none> <none>
ceph-csi-rbd ceph-csi-rbd-provisioner-6b9f54d9df-62qv4 7/7 Running 0 28m 192.168.73.193 k8s-dev-node-3 <none> <none>
ceph-csi-rbd ceph-csi-rbd-provisioner-6b9f54d9df-mx99b 7/7 Running 9265 (6d2h ago) 345d 192.168.3.138 docker-dev-ucsb-1 <none> <none>
cert-manager cert-manager-6c576bddcf-hk6lz 1/1 Running 0 28m 192.168.19.200 k8s-dev-node-2 <none> <none>
cert-manager cert-manager-cainjector-669c966b86-b7ql7 1/1 Running 0 28m 192.168.19.203 k8s-dev-node-2 <none> <none>
cert-manager cert-manager-webhook-7ff944cc49-cgj6f 1/1 Running 0 28m 192.168.19.201 k8s-dev-node-2 <none> <none>
dataone-solr dataone-solr-0 1/1 Running 1 (25m ago) 27m 192.168.3.157 docker-dev-ucsb-1 <none> <none>
dataone-solr dataone-solr-1 1/1 Running 7 (6d2h ago) 181d 192.168.3.134 docker-dev-ucsb-1 <none> <none>
dataone-solr dataone-solr-2 1/1 Running 0 27m 192.168.19.219 k8s-dev-node-2 <none> <none>
dataone-solr dataone-solr-zookeeper-0 1/1 Running 0 27m 192.168.19.207 k8s-dev-node-2 <none> <none>
dataone-solr dataone-solr-zookeeper-1 1/1 Running 5 (6d2h ago) 181d 192.168.3.135 docker-dev-ucsb-1 <none> <none>
dataone-solr dataone-solr-zookeeper-2 1/1 Running 0 27m 192.168.19.208 k8s-dev-node-2 <none> <none>
gnis gnis-74c6f7c6df-gbfwk 2/4 Running 0 28m 192.168.19.206 k8s-dev-node-2 <none> <none>
ingress-nginx ingress-nginx-controller-74f577fbf9-2l927 1/1 Running 5 (6d2h ago) 246d 192.168.3.191 docker-dev-ucsb-1 <none> <none>
jing d1index-idxworker-6f66bbcf8-5d7tj 0/1 Pending 0 26h <none> <none> <none> <none>
jing d1index-idxworker-6f66bbcf8-vjrjh 0/1 Pending 0 26h <none> <none> <none> <none>
jing d1index-idxworker-6f66bbcf8-vph2n 0/1 Pending 0 26h <none> <none> <none> <none>
jing d1index-rabbitmq-0 1/1 Running 0 26h 192.168.3.139 docker-dev-ucsb-1 <none> <none>
jing d1index-solr-0 1/1 Running 3 (22m ago) 26h 192.168.3.190 docker-dev-ucsb-1 <none> <none>
jing d1index-solr-1 1/1 Running 0 27m 192.168.19.220 k8s-dev-node-2 <none> <none>
jing d1index-solr-2 1/1 Running 0 27m 192.168.19.217 k8s-dev-node-2 <none> <none>
jing d1index-zookeeper-0 1/1 Running 0 26h 192.168.3.149 docker-dev-ucsb-1 <none> <none>
jing d1index-zookeeper-1 1/1 Running 0 27m 192.168.19.211 k8s-dev-node-2 <none> <none>
jing d1index-zookeeper-2 1/1 Running 0 27m 192.168.19.215 k8s-dev-node-2 <none> <none>
jones d1index-idxworker-844b59cc7c-79kjp 0/1 Pending 0 12d <none> <none> <none> <none>
jones d1index-idxworker-844b59cc7c-bjztl 0/1 Pending 0 12d <none> <none> <none> <none>
jones d1index-idxworker-844b59cc7c-w2vlm 0/1 Pending 0 12d <none> <none> <none> <none>
jones d1index-rabbitmq-0 1/1 Running 0 27m 192.168.19.214 k8s-dev-node-2 <none> <none>
jones d1index-solr-0 1/1 Running 0 27m 192.168.19.221 k8s-dev-node-2 <none> <none>
jones d1index-solr-1 1/1 Running 1 (6d2h ago) 12d 192.168.3.150 docker-dev-ucsb-1 <none> <none>
jones d1index-solr-2 1/1 Running 0 27m 192.168.19.218 k8s-dev-node-2 <none> <none>
jones d1index-zookeeper-0 1/1 Running 1 (6d2h ago) 12d 192.168.3.164 docker-dev-ucsb-1 <none> <none>
jones d1index-zookeeper-1 1/1 Running 0 27m 192.168.19.213 k8s-dev-node-2 <none> <none>
jones d1index-zookeeper-2 1/1 Running 1 (6d2h ago) 12d 192.168.3.179 docker-dev-ucsb-1 <none> <none>
kube-system calico-kube-controllers-6fd7b9848d-7wrr9 1/1 Running 1823 (6d2h ago) 214d 192.168.3.160 docker-dev-ucsb-1 <none> <none>
kube-system calico-node-5swzc 1/1 Running 0 39m 128.111.85.213 k8s-dev-node-2 <none> <none>
kube-system calico-node-78ttp 1/1 Running 2 (6d3h ago) 214d 128.111.85.191 docker-dev-ucsb-2 <none> <none>
kube-system calico-node-hzznf 1/1 Running 38 (6d2h ago) 214d 128.111.85.190 docker-dev-ucsb-1 <none> <none>
kube-system calico-node-v42ng 1/1 Running 0 7m48s 128.111.85.227 k8s-dev-node-3 <none> <none>
kube-system coredns-78fcd69978-qktrm 1/1 Running 7 (6d2h ago) 422d 192.168.3.175 docker-dev-ucsb-1 <none> <none>
kube-system coredns-78fcd69978-xjqjz 1/1 Running 7 (6d2h ago) 422d 192.168.3.158 docker-dev-ucsb-1 <none> <none>
kube-system etcd-docker-dev-ucsb-1 1/1 Running 55 (6d2h ago) 49d 128.111.85.190 docker-dev-ucsb-1 <none> <none>
kube-system kube-apiserver-docker-dev-ucsb-1 1/1 Running 653 (6d2h ago) 21d 128.111.85.190 docker-dev-ucsb-1 <none> <none>
kube-system kube-controller-manager-docker-dev-ucsb-1 1/1 Running 2243 (5d10h ago) 49d 128.111.85.190 docker-dev-ucsb-1 <none> <none>
kube-system kube-proxy-7hmdm 1/1 Running 3 (6d3h ago) 422d 128.111.85.191 docker-dev-ucsb-2 <none> <none>
kube-system kube-proxy-9q4fl 1/1 Running 0 7m48s 128.111.85.227 k8s-dev-node-3 <none> <none>
kube-system kube-proxy-lrhqm 1/1 Running 0 39m 128.111.85.213 k8s-dev-node-2 <none> <none>
kube-system kube-proxy-x54mw 1/1 Running 7 (6d2h ago) 422d 128.111.85.190 docker-dev-ucsb-1 <none> <none>
kube-system kube-scheduler-docker-dev-ucsb-1 1/1 Running 565 (5d10h ago) 49d 128.111.85.190 docker-dev-ucsb-1 <none> <none>
metadig metadig-controller-8fdcfb97b-rh44v 1/1 Running 0 28m 192.168.19.205 k8s-dev-node-2 <none> <none>
metadig metadig-postgres-84f47997d6-6rqxp 2/2 Running 6 (6d2h ago) 135d 192.168.3.168 docker-dev-ucsb-1 <none> <none>
metadig metadig-rabbitmq-0 1/1 Running 0 27m 192.168.19.212 k8s-dev-node-2 <none> <none>
metadig metadig-rabbitmq-1 1/1 Running 3 (6d2h ago) 135d 192.168.3.154 docker-dev-ucsb-1 <none> <none>
metadig metadig-rabbitmq-2 1/1 Running 0 27m 192.168.19.210 k8s-dev-node-2 <none> <none>
metadig metadig-scheduler-6584589b77-28p6r 1/1 Running 2 (6d2h ago) 135d 192.168.3.170 docker-dev-ucsb-1 <none> <none>
metadig metadig-scorer-85664c564-8k7vr 1/1 Running 0 28m 192.168.19.209 k8s-dev-node-2 <none> <none>
metadig metadig-solr-0 1/1 Running 0 27m 192.168.19.216 k8s-dev-node-2 <none> <none>
metadig metadig-worker-65c4948d6f-hjpsl 1/1 Running 7 (6d2h ago) 135d 192.168.3.173 docker-dev-ucsb-1 <none> <none>
olm catalog-operator-8dc86744b-rwtfc 1/1 Running 3 (6d2h ago) 86d 192.168.3.137 docker-dev-ucsb-1 <none> <none>
olm olm-operator-6d88f56b99-mg4mz 1/1 Running 3 (6d2h ago) 86d 192.168.3.136 docker-dev-ucsb-1 <none> <none>
polder crawl-27758880--1-mcbvw 0/1 Completed 0 2d 192.168.3.130 docker-dev-ucsb-1 <none> <none>
polder dev-gleaner-8b6b6c4c9-shklp 3/3 Running 3 (6d2h ago) 6d6h 192.168.3.187 docker-dev-ucsb-1 <none> <none>
polder dev-polder-7b7ffb57c9-p7gps 1/1 Running 8 (12h ago) 6d6h 192.168.3.133 docker-dev-ucsb-1 <none> <none>
polder setup-gleaner--1-qcdgf 0/1 Completed 0 6d2h 192.168.3.165 docker-dev-ucsb-1 <none> <none>
polder setup-gleaner--1-zw5jf 0/1 Init:Error 0 6d6h <none> docker-dev-ucsb-1 <none> <none>
purser purser-6cf78cf647-jqqkz 1/1 Running 0 28m 192.168.19.196 k8s-dev-node-2 <none> <none>
slinky redis-6cdf5d7d9d-gl4lf 1/1 Running 3 (6d2h ago) 62d 192.168.3.181 docker-dev-ucsb-1 <none> <none>
slinky rqdashboard-6dc69d7678-9zm7l 1/1 Running 3 (6d2h ago) 62d 192.168.3.162 docker-dev-ucsb-1 <none> <none>
slinky rqscheduler-75cf7bbdd8-n5m89 1/1 Running 0 28m 192.168.19.197 k8s-dev-node-2 <none> <none>
slinky virtuoso-7787cbf65f-c8t25 1/1 Running 3 (6d2h ago) 62d 192.168.3.151 docker-dev-ucsb-1 <none> <none>
slinky web-6dcc99cd8c-hnz4s 1/1 Running 0 28m 192.168.19.198 k8s-dev-node-2 <none> <none>
slinky worker-dataset-584b7c4ccc-hdpwf 1/1 Running 0 28m 192.168.19.195 k8s-dev-node-2 <none> <none>
slinky worker-default-5cd8d84957-p6ns2 1/1 Running 0 28m 192.168.19.204 k8s-dev-node-2 <none> <none>
I setup two new node VMs (k8s-dev-node-2 and k8s-dev-node-3) so there will be less downtime when upgrading and rebooting dev nodes.
These need to be added to K8s using the instructions at https://github.com/DataONEorg/k8s-cluster/blob/main/admin/admin.md