Martin-Weiss
commented
2 years ago harvester1:/usr/local/.state/var-lib-rancher.bind/rke2/agent/containerd # du -sh *
4.0K bin
5.1M containerd-2022-07-22T03-13-52.486.log.gz
5.2M containerd-2022-07-26T21-39-24.727.log.gz
5.1M containerd-2022-07-29T03-01-49.658.log.gz
8.1M containerd.log
18G io.containerd.content.v1.content
442M io.containerd.grpc.v1.cri
8.0K io.containerd.grpc.v1.introspection
12M io.containerd.metadata.v1.bolt
4.0K io.containerd.runtime.v1.linux
3.6M io.containerd.runtime.v2.task
4.0K io.containerd.snapshotter.v1.btrfs
8.0K io.containerd.snapshotter.v1.native
20G io.containerd.snapshotter.v1.overlayfs
4.0K lib
4.0K tmpmountscrictl images
IMAGE TAG IMAGE ID SIZE
docker.io/bitnami/kubectl 1.18.6 5dacb5d170d3a 133MB
docker.io/csiplugin/snapshot-controller v4.0.0 f1d8a00ae690f 47.8MB
docker.io/kubevip/kube-vip-cloud-provider 0.1 63fdd5f25e1d5 89.5MB
docker.io/kubevip/kube-vip-cloud-provider v0.0.1 5d5b371eade4b 89.5MB
docker.io/library/alpine 3 0ac33e5f5afa7 5.86MB
docker.io/library/alpine 3.12 24c8ece58a1aa 5.87MB
docker.io/library/busybox 1.32.0 219ee5171f800 1.45MB
docker.io/longhornio/backing-image-manager v2_20210820 54872706d1b0f 288MB
docker.io/longhornio/csi-attacher v3.2.1 6de272f181374 54.7MB
docker.io/longhornio/csi-node-driver-registrar v2.3.0 368ee7d6e60fb 20MB
docker.io/longhornio/csi-provisioner v2.1.2 0f0a0f7990768 52.9MB
docker.io/longhornio/csi-resizer v1.2.0 0aa9629e1508b 55.2MB
docker.io/longhornio/csi-snapshotter v3.0.3 000846ee53356 49.1MB
docker.io/longhornio/longhorn-engine v1.2.3 d74b4661fc487 319MB
docker.io/longhornio/longhorn-engine v1.2.4 f0c119aae83c5 320MB
docker.io/longhornio/longhorn-instance-manager v1_20211210 38c8612419a4b 294MB
docker.io/longhornio/longhorn-instance-manager v1_20220303 92291586da370 317MB
docker.io/longhornio/longhorn-manager v1.2.3 b9dc7b95eb9c7 266MB
docker.io/longhornio/longhorn-manager v1.2.4 6561caaf4d7e6 267MB
docker.io/longhornio/longhorn-share-manager v1_20211020 2882a087491b9 179MB
docker.io/longhornio/longhorn-ui v1.2.3 8696717eb0410 46.5MB
docker.io/longhornio/longhorn-ui v1.2.4 085dbcd29a7cd 51.9MB
docker.io/rancher/fleet-agent v0.3.9 25ddd5cd4c116 156MB
<none> <none> c059bfaa849c4 5.87MB
docker.io/plndr/kube-vip v0.3.7 f0740292dca9d 41.7MB
docker.io/rancher/fleet v0.3.8 64012dadc3ba8 73.6MB
docker.io/rancher/fleet v0.3.9 fd7b1e2489a67 73.6MB
docker.io/rancher/gitjob v0.1.23 dc6c08cf3137d 62.2MB
docker.io/rancher/gitjob v0.1.26 f1482136b088d 62.6MB
docker.io/rancher/hardened-calico v3.20.2-build20211119 2568133718dcc 528MB
docker.io/rancher/hardened-calico v3.21.4-build20220228 3d20d59cd627c 575MB
docker.io/rancher/hardened-cluster-autoscaler v1.8.5-build20211119 0e6424d22fd5c 120MB
docker.io/rancher/hardened-cni-plugins v0.9.1-build20211119 a01dc94a8ae18 155MB
docker.io/rancher/hardened-coredns v1.8.5-build20211119 4eece06adefa5 147MB
docker.io/rancher/hardened-coredns v1.9.1-build20220318 a9c20758166a0 137MB
docker.io/rancher/hardened-dns-node-cache 1.21.2-build20211119 910f1302e587f 145MB
docker.io/rancher/hardened-etcd v3.4.13-k3s3-build20211118 cbeb6a8bed654 136MB
docker.io/rancher/hardened-etcd v3.4.18-k3s1-build20220118 f346e2e50af47 136MB
docker.io/rancher/hardened-flannel v0.15.1-build20211119 f4754f700d099 279MB
docker.io/rancher/hardened-flannel v0.17.0-build20220317 f776f3ce534ab 275MB
docker.io/rancher/hardened-k8s-metrics-server v0.5.0-build20211119 57533a88f34ca 143MB
docker.io/rancher/hardened-kubernetes v1.21.11-rke2r1-build20220316 607ea2d30c1cd 737MB
docker.io/rancher/hardened-multus-cni v3.7.1-build20211119 e9e92ffb1e664 253MB
docker.io/rancher/harvester-cluster-repo v1.0.0 650d18d621215 146MB
docker.io/rancher/harvester-cluster-repo v1.0.1 a55a85d01bab8 181MB
docker.io/rancher/harvester-cluster-repo v1.0.2 2ca92bfb490ae 182MB
docker.io/rancher/harvester-load-balancer v0.1.2 ea34dd666ed75 77.6MB
docker.io/rancher/harvester-network-controller v0.2.6 56b5012a07c49 54.1MB
docker.io/rancher/harvester-network-controller v0.2.7 7d39048091020 172MB
docker.io/rancher/harvester-network-helper v0.2.6 8bd2cab235abf 47.8MB
docker.io/rancher/harvester-network-helper v0.2.7 18dd6c2c232ef 88.8MB
docker.io/rancher/harvester-node-disk-manager v0.4.0 2e3a719683993 48.4MB
docker.io/rancher/harvester-node-disk-manager v0.4.1 49beb36f611fd 203MB
docker.io/rancher/harvester-node-disk-manager v0.4.3 569f93064c6b7 203MB
docker.io/rancher/harvester-upgrade v1.0.0 a9b6a69c6cc78 342MB
docker.io/rancher/harvester-upgrade v1.0.1 c3947c4edcd18 305MB
docker.io/rancher/harvester-upgrade v1.0.2 eba8e4b996314 304MB
docker.io/rancher/harvester-webhook v1.0.0 98e3d6bba5a32 68.5MB
docker.io/rancher/harvester-webhook v1.0.1 3d1c491af8ad7 230MB
docker.io/rancher/harvester-webhook v1.0.2 4c40bb2967788 231MB
docker.io/rancher/harvester v1.0.0 12e578e92109e 201MB
docker.io/rancher/harvester v1.0.1 fcfed22de0206 292MB
docker.io/rancher/harvester v1.0.2 970e09b318b01 294MB
docker.io/rancher/klipper-helm v0.6.6-build20211022 194c895f8d63f 242MB
docker.io/rancher/klipper-helm v0.7.0-build20220315 d5cb491e1a53a 239MB
docker.io/rancher/kubectl v1.20.2 b119c4dfc78c8 40.2MB
docker.io/rancher/kubectl v1.21.5 43e8b9390c725 46.7MB
docker.io/rancher/mirrored-directxman12-k8s-prometheus-adapter v0.8.4 9d7c9987f24e8 66.1MB
docker.io/rancher/mirrored-grafana-grafana 6.7.4 be4c69a1aae8d 237MB
docker.io/rancher/mirrored-grafana-grafana 7.5.11 6cfe8ab94353f 206MB
docker.io/rancher/mirrored-grafana-grafana 7.5.8 6f31769d2f324 206MB
docker.io/rancher/mirrored-ingress-nginx-kube-webhook-certgen v1.0 17e55ec30f203 48MB
docker.io/rancher/mirrored-ingress-nginx-kube-webhook-certgen v1.1.1 c41e9fcadf5a2 49.1MB
docker.io/rancher/mirrored-jettech-kube-webhook-certgen v1.5.2 a573628e4199d 30.9MB
docker.io/rancher/mirrored-kiwigrid-k8s-sidecar 1.12.2 2ec7ad04d6cd6 95.1MB
docker.io/rancher/mirrored-kiwigrid-k8s-sidecar 1.12.3 7cd5e6c23e4dc 95.2MB
docker.io/rancher/mirrored-kube-state-metrics-kube-state-metrics v2.0.0 def5ee20a468c 34.7MB
docker.io/rancher/mirrored-kube-state-metrics-kube-state-metrics v2.2.0 8ceaed6e0b5dd 39.6MB
docker.io/rancher/mirrored-library-busybox 1.31.1 1c35c44120825 1.44MB
docker.io/rancher/mirrored-library-busybox 1.32.1 388056c9a6838 1.45MB
docker.io/rancher/mirrored-library-nginx 1.21.1-alpine 1318bf5f63b46 24.4MB
docker.io/rancher/mirrored-prometheus-adapter-prometheus-adapter v0.9.0 e45f535a68cc6 68.9MB
docker.io/rancher/mirrored-prometheus-alertmanager v0.22.2 bed86c08a78ad 25MB
docker.io/rancher/mirrored-prometheus-node-exporter v1.1.2 c19ae228f0699 27.4MB
docker.io/rancher/mirrored-prometheus-node-exporter v1.2.2 0fafea1498594 22.6MB
docker.io/rancher/mirrored-prometheus-operator-prometheus-config-reloader v0.48.0 918e0dc58ce03 13.8MB
docker.io/rancher/mirrored-prometheus-operator-prometheus-config-reloader v0.50.0 972e89374460e 13.8MB
docker.io/rancher/mirrored-prometheus-operator-prometheus-operator v0.48.0 3001bc29fbefa 45.7MB
docker.io/rancher/mirrored-prometheus-operator-prometheus-operator v0.50.0 e8718944b25c5 48.6MB
docker.io/rancher/mirrored-prometheus-prometheus v2.27.1 86ea6f86fc575 187MB
docker.io/rancher/mirrored-prometheus-prometheus v2.28.1 9dfc442be98c8 190MB
docker.io/rancher/nginx-ingress-controller nginx-1.0.2-hardened4 ccc8ebe076751 589MB
docker.io/rancher/pause 3.6 6270bb605e12e 686kB
docker.io/rancher/rancher-agent v2.6.5 fb844ee23a116 453MB
docker.io/rancher/rancher-agent v2.6.6 b09d97dcb88ca 459MB
docker.io/rancher/rancher-webhook v0.2.2 d662c1eed858b 49.4MB
docker.io/rancher/rancher-webhook v0.2.5 e4f71593519f0 50.5MB
docker.io/rancher/rancher v2.6.3-harvester1 4b047114a03f6 1.19GB
docker.io/rancher/rancher v2.6.4-harvester3 9e4db6ed500b1 1.43GB
docker.io/rancher/rke2-cloud-provider v0.0.3-build20211118 029e4a095f53d 143MB
docker.io/rancher/rke2-runtime v1.21.11-rke2r1 c289ec075218c 293MB
docker.io/rancher/rke2-runtime v1.21.7-rke2r1 0b733ec5b03fe 293MB
docker.io/rancher/shell v0.1.14 2a129a26351d1 172MB
docker.io/rancher/shell v0.1.16 9dab16c68e6b8 170MB
docker.io/rancher/shell v0.1.8 4921f8cf3ec44 151MB
docker.io/rancher/support-bundle-kit v0.0.5 2e25c11c65171 85.8MB
docker.io/rancher/support-bundle-kit v0.0.7 27c127e763a32 205MB
docker.io/rancher/support-bundle-kit v0.0.8 e94cd69519eba 74.1MB
docker.io/rancher/system-agent v0.1.1-suc 4382d3c706e71 81.6MB
docker.io/rancher/system-agent v0.2.3-suc 17e412e38b2de 82.5MB
docker.io/rancher/system-upgrade-controller v0.8.1 410ce405fc4ff 29.8MB
ghcr.io/kube-vip/kube-vip v0.4.3 c379ecd00c21d 39MB
ghcr.io/kube-vip/kube-vip v0.4.4 10a711aa9888d 37.7MB
registry.suse.com/harvester-beta/virt-api 0.49.0-2 b6ccb14473e21 180MB
registry.suse.com/harvester-beta/virt-controller 0.49.0-2 1e58906c673b2 180MB
registry.suse.com/harvester-beta/virt-handler 0.49.0-2 5b8b28a7f3051 339MB
registry.suse.com/harvester-beta/virt-launcher 0.49.0-2 0de0705b81ed0 590MB
registry.suse.com/harvester-beta/virt-operator 0.49.0-2 a30583fd6e840 179MB
registry.suse.com/harvester-beta/vmdp latest 2f0b445da1f4e 16.9MB
registry.suse.com/suse/sles/15.3/libguestfs-tools 0.45.0-8.4.3 f6b914f5b4acc 506MB
registry.suse.com/suse/sles/15.3/virt-api 0.45.0-8.4.3 42db6d2ef5c12 178MB
registry.suse.com/suse/sles/15.3/virt-controller 0.45.0-8.4.3 2536d353d5c41 178MB
registry.suse.com/suse/sles/15.3/virt-launcher 0.45.0-8.4.3 c5d061cac2e36 584MB
registry.suse.com/suse/sles/15.3/virt-operator 0.45.0-8.4.3 ecde4e18a75a7 177MB
It seems the disk space is used by images - especially due to previous upgrades where the "old" images did not seem to get removed.. but as the images are automatically pre-loaded during the harvester installation and upgrade I am not sure if it is a valid workaround to remove the "old" images i.e. with crictl rmi --prune <- would that be allowed with harvester before the upgrade to free up enough disk space in /usr/local?
bk201
Describe the bug Based on the harvester requirements documentation harvester was installed on a 200 GB disk for the OS and a 300 GB disk for additional data. Now when doing the upgrade from 1.0.2 to 1.0.3 we are verifying the free disk space on /usr/local and see that this is only 14 GB instead of the documented >25 GB that are required.
-> how can we increase the disk space for /usr/local or how can we free up disk space in /usr/local so that we can follow the requirements of the upgrade?
To Reproduce Steps to reproduce the behavior:
Expected behavior Default disk layout should be sufficient to allow upgrades. In case it is documnented that there needs to be enough free disk space in /usr/local it is also required to document what to do in case there is not sufficient disk space available.
Support bundle supportbundle_0ed34b76-5048-40e2-a2ac-da9798f550fb_2022-08-16T13-46-49Z.zip
Environment