search

sealerio / sealer

Build, Share and Run Both Your Kubernetes Cluster and Distributed Applications (Project under CNCF)
http://sealer.cool
Apache License 2.0
2.06k stars 362 forks source link

sealer0.91版本安装kubernetes v1.20.4失败，提示找不到kubeadm.yml文件 #2044

Open Jackmeipx opened 1 year ago

Jackmeipx commented 1 year ago

What happen?

问题：6个节点安装kubernetes v1.20.4失败，提示找不到/var/lib/sealer/data/my-cluster/rootfs/etc/kubeadm.yml文件，查看安装工具未创建目录/var/lib/sealer/data/my-cluster/rootfs/etc/和拷贝相应的文体，检查镜像中有etc目录和kubeadm.yml文件。

使用kubernetes v.1.22.15版本安装存在相同的报错，无法安装成功。

[root@test-shiyan ~]# sealer run docker.io/sealerio/kubernetes:v1.20.4-sealerio-2 --masters 172.20.57.108,172.20.58.189,172.20.58.88 --nodes 172.20.59.176,172.20.59.212,172.20.58.164

报错： 2023-02-16 15:46:27 [ERROR] [root.go:75] sealer-v0.9.1: failed to found kubeadm config from /var/lib/sealer/data/my-cluster/rootfs/etc/kubeadm.yml: failed to open configfile(/var/lib/sealer/data/my-cluster/rootfs/etc/kubeadm.yml): open /var/lib/sealer/data/my-cluster/rootfs/etc/kubeadm.yml: no such file or directory

镜像中kubeadm.yml文件： [root@test-shiyan ~]# ls /var/lib/containers/storage/overlay/84e5ad09f6a2b1c1036394472a17e60986aea6b97f3f4c59e92c7a7e3446f6e5/merged/etc 10-kubeadm.conf docker.service kubeadm.yml kubelet.service daemon.json kubeadm kubeadm.yml.tmpl registry_config.yml [root@test-shiyan ~]# grep kubernetesVersion /var/lib/containers/storage/overlay/84e5ad09f6a2b1c1036394472a17e60986aea6b97f3f4c59e92c7a7e3446f6e5/merged/etc/kubeadm.yml kubernetesVersion: v1.20.4

Relevant log output?

[root@test-shiyan ~]# sealer images
REPOSITORY                      TAG                   IMAGE ID       CREATED       SIZE
docker.io/sealerio/kubernetes   v1.20.4-sealerio-2    e4f4f174bf27   4 weeks ago   982 MB
[root@test-shiyan ~]# df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        7.8G     0  7.8G   0% /dev
tmpfs           7.8G     0  7.8G   0% /dev/shm
tmpfs           7.8G   17M  7.8G   1% /run
tmpfs           7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/vda1        80G   51G   30G  64% /
tmpfs           1.6G     0  1.6G   0% /run/user/0
[root@test-shiyan ~]# sealer version
{"gitVersion":"v0.9.1","gitCommit":"9934e11","buildDate":"2023-01-17 11:30:38","goVersion":"go1.17.13","compiler":"gc","platform":"linux/amd64"}
[root@test-shiyan ~]# cat /etc/redhat-release 
CentOS Linux release 7.8.2003 (Core)
[root@test-shiyan ~]# uname -a
Linux test-shiyan.cecdat.com 3.10.0-1127.el7.x86_64 #1 SMP Tue Mar 31 23:36:51 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
[root@test-shiyan ~]# sealer run docker.io/sealerio/kubernetes:v1.20.4-sealerio-2 --masters 172.20.57.108,172.20.58.189,172.20.58.88 --nodes 172.20.59.176,172.20.59.212,172.20.58.164
2023-02-16 15:44:08 [INFO] [pull.go:79] e4f4f174bf27fe397d579279d599fb191ab31510f4464fc37bb3f891818e8856

2023-02-16 15:44:08 [INFO] [run.go:229] start to create new cluster with image: docker.io/sealerio/kubernetes:v1.20.4-sealerio-2

2023-02-16 15:44:13 [INFO] [pull.go:79] e4f4f174bf27fe397d579279d599fb191ab31510f4464fc37bb3f891818e8856

copying files to 172.20.58.164: 31/31 
copying files to 172.20.58.88: 31/31 
copying files to 172.20.57.108: 31/31 
copying files to 172.20.58.189: 31/31 
copying files to 172.20.59.176: 31/31 
copying files to 172.20.59.212: 31/31 
+ set -e
+++ dirname /var/lib/sealer/data/my-cluster/rootfs/scripts/docker.sh
++ cd /var/lib/sealer/data/my-cluster/rootfs/scripts
++ pwd
+ scripts_path=/var/lib/sealer/data/my-cluster/rootfs/scripts
+ image_dir=/var/lib/sealer/data/my-cluster/rootfs/scripts/../images
+ DOCKER_VERSION=19.03.14-sealer
+ storage=/var/lib/docker
+ mkdir -p /var/lib/docker
+ utils_command_exists docker
+ command -v docker
++ get_distribution
++ lsb_dist=
++ '[' -r /etc/os-release ']'
+++ . /etc/os-release
++++ NAME='CentOS Linux'
++++ VERSION='7 (Core)'
++++ ID=centos
++++ ID_LIKE='rhel fedora'
++++ VERSION_ID=7
++++ PRETTY_NAME='CentOS Linux 7 (Core)'
++++ ANSI_COLOR='0;31'
++++ CPE_NAME=cpe:/o:centos:centos:7
++++ HOME_URL=https://www.centos.org/
++++ BUG_REPORT_URL=https://bugs.centos.org/
++++ CENTOS_MANTISBT_PROJECT=CentOS-7
++++ CENTOS_MANTISBT_PROJECT_VERSION=7
++++ REDHAT_SUPPORT_PRODUCT=centos
++++ REDHAT_SUPPORT_PRODUCT_VERSION=7
+++ echo centos
++ lsb_dist=centos
++ echo centos
+ lsb_dist=centos
++ echo centos
++ tr '[:upper:]' '[:lower:]'
+ lsb_dist=centos
+ echo 'current system is centos'
current system is centos
+ case "$lsb_dist" in
+ cp /var/lib/sealer/data/my-cluster/rootfs/scripts/../etc/docker.service /usr/lib/systemd/system/docker.service
+ '[' -d /etc/docker/ ']'
+ mkdir /etc/docker/ -p
+ chmod -R 755 /var/lib/sealer/data/my-cluster/rootfs/scripts/../cri
+ tar -zxvf /var/lib/sealer/data/my-cluster/rootfs/scripts/../cri/docker.tar.gz -C /usr/bin
containerd
containerd-shim
ctr
docker
docker-init
docker-proxy
dockerd
runc
+ chmod a+x /usr/bin
+ chmod a+x /usr/bin/docker
+ chmod a+x /usr/bin/dockerd
+ systemctl enable docker.service
+ systemctl restart docker.service
+ cp /var/lib/sealer/data/my-cluster/rootfs/scripts/../etc/daemon.json /etc/docker
+ [[ -n '' ]]
+ disable_selinux
+ '[' -s /etc/selinux/config ']'
+ grep SELINUX=enforcing /etc/selinux/config
+ systemctl daemon-reload
+ systemctl restart docker.service
+ check_docker_valid
+ docker info
Client:
 Debug Mode: false

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 38
 Server Version: 19.03.14-sealer
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: systemd
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ea765aba0d05254012b0b9e595e995c09186427f
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 3.10.0-1127.el7.x86_64
 Operating System: CentOS Linux 7 (Core)
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 15.51GiB
 Name: test-sealer-k8s-1.cecdat.com
 ID: ISB3:GVHU:TY63:BTNA:WNEX:YJLE:47A3:JHNO:6W6Z:2QVI:R7FU:EKUG
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true
 Insecure Registries:
  0.0.0.0/0
  ::/0
  127.0.0.0/8
 Live Restore Enabled: true

++ docker info --format '{{json .ServerVersion}}'
++ tr -d '"'
+ dockerVersion=19.03.14-sealer
+ '[' 19.03.14-sealer '!=' 19.03.14-sealer ']'
+ load_images
+ for image in '"$image_dir"/*'
+ '[' -f /var/lib/sealer/data/my-cluster/rootfs/scripts/../images/docker-amd64-registry-image.tar.gz ']'
+ docker load -q -i /var/lib/sealer/data/my-cluster/rootfs/scripts/../images/docker-amd64-registry-image.tar.gz
Loaded image: registry:2.7.1
+ bash /var/lib/sealer/data/my-cluster/rootfs/scripts/init-kube.sh
+ disable_firewalld
++ get_distribution
++ lsb_dist=
++ '[' -r /etc/os-release ']'
+++ . /etc/os-release
++++ NAME='CentOS Linux'
++++ VERSION='7 (Core)'
++++ ID=centos
++++ ID_LIKE='rhel fedora'
++++ VERSION_ID=7
++++ PRETTY_NAME='CentOS Linux 7 (Core)'
++++ ANSI_COLOR='0;31'
++++ CPE_NAME=cpe:/o:centos:centos:7
++++ HOME_URL=https://www.centos.org/
++++ BUG_REPORT_URL=https://bugs.centos.org/
++++ CENTOS_MANTISBT_PROJECT=CentOS-7
++++ CENTOS_MANTISBT_PROJECT_VERSION=7
++++ REDHAT_SUPPORT_PRODUCT=centos
++++ REDHAT_SUPPORT_PRODUCT_VERSION=7
+++ echo centos
++ lsb_dist=centos
++ echo centos
+ lsb_dist=centos
++ echo centos
++ tr '[:upper:]' '[:lower:]'
+ lsb_dist=centos
+ case "$lsb_dist" in
+ systemctl stop firewalld
Failed to stop firewalld.service: Unit firewalld.service not loaded.
+ copy_bins
+ chmod -R 755 /var/lib/sealer/data/my-cluster/rootfs/scripts/../bin/conntrack /var/lib/sealer/data/my-cluster/rootfs/scripts/../bin/containerd-rootless-setuptool.sh /var/lib/sealer/data/my-cluster/rootfs/scripts/../bin/containerd-rootless.sh /var/lib/sealer/data/my-cluster/rootfs/scripts/../bin/crictl /var/lib/sealer/data/my-cluster/rootfs/scripts/../bin/kubeadm /var/lib/sealer/data/my-cluster/rootfs/scripts/../bin/kubectl /var/lib/sealer/data/my-cluster/rootfs/scripts/../bin/kubelet /var/lib/sealer/data/my-cluster/rootfs/scripts/../bin/nerdctl /var/lib/sealer/data/my-cluster/rootfs/scripts/../bin/seautil
+ chmod 644 /var/lib/sealer/data/my-cluster/rootfs/scripts/../bin
+ cp /var/lib/sealer/data/my-cluster/rootfs/scripts/../bin/conntrack /var/lib/sealer/data/my-cluster/rootfs/scripts/../bin/containerd-rootless-setuptool.sh /var/lib/sealer/data/my-cluster/rootfs/scripts/../bin/containerd-rootless.sh /var/lib/sealer/data/my-cluster/rootfs/scripts/../bin/crictl /var/lib/sealer/data/my-cluster/rootfs/scripts/../bin/kubeadm /var/lib/sealer/data/my-cluster/rootfs/scripts/../bin/kubectl /var/lib/sealer/data/my-cluster/rootfs/scripts/../bin/kubelet /var/lib/sealer/data/my-cluster/rootfs/scripts/../bin/nerdctl /var/lib/sealer/data/my-cluster/rootfs/scripts/../bin/seautil /usr/bin
+ cp /var/lib/sealer/data/my-cluster/rootfs/scripts/../scripts/kubelet-pre-start.sh /usr/bin
+ chmod +x /usr/bin/kubelet-pre-start.sh
+ copy_kubelet_service
+ mkdir -p /etc/systemd/system
+ cp /var/lib/sealer/data/my-cluster/rootfs/scripts/../etc/kubelet.service /etc/systemd/system/
+ '[' -d /etc/systemd/system/kubelet.service.d ']'
+ mkdir /etc/systemd/system/kubelet.service.d
+ cp /var/lib/sealer/data/my-cluster/rootfs/scripts/../etc/10-kubeadm.conf /etc/systemd/system/kubelet.service.d/
+ '[' -d /var/lib/kubelet ']'
+ /usr/bin/kubelet-pre-start.sh
Disabled
# set by ack-distro
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 15
net.ipv6.conf.all.forwarding = 1
net.ipv4.conf.all.arp_filter = 0
net.ipv4.conf.all.rp_filter = 0
* Applying /usr/lib/sysctl.d/00-system.conf ...
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0
* Applying /usr/lib/sysctl.d/10-default-yama-scope.conf ...
kernel.yama.ptrace_scope = 0
* Applying /usr/lib/sysctl.d/50-default.conf ...
kernel.sysrq = 16
kernel.core_uses_pid = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.promote_secondaries = 1
net.ipv4.conf.all.promote_secondaries = 1
fs.protected_hardlinks = 1
fs.protected_symlinks = 1
* Applying /etc/sysctl.d/99-sysctl.conf ...
vm.dirty_ratio = 20
vm.dirty_background_ratio = 5
vm.dirty_writeback_centisecs = 200
vm.dirty_expire_centisecs = 100
vm.drop_caches = 2
vm.vfs_cache_pressure = 100
vm.lowmem_reserve_ratio = 32 32 8
net.ipv4.tcp_max_syn_backlog = 8192
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_mem = 178368 524288 33554432
net.ipv4.tcp_wmem = 4096 87380 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.core.netdev_max_backlog = 8912
net.ipv4.ip_local_port_range = 1024 65000
net.core.somaxconn = 8192
net.ipv4.tcp_tw_reuse = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.ipv4.tcp_fin_timeout = 15
net.ipv6.conf.all.forwarding = 1
net.ipv4.conf.all.arp_filter = 0
net.ipv4.conf.all.rp_filter = 0
* Applying /etc/sysctl.d/k8s.conf ...
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 15
net.ipv6.conf.all.forwarding = 1
net.ipv4.conf.all.arp_filter = 0
net.ipv4.conf.all.rp_filter = 0
* Applying /etc/sysctl.conf ...
vm.dirty_ratio = 20
vm.dirty_background_ratio = 5
vm.dirty_writeback_centisecs = 200
vm.dirty_expire_centisecs = 100
vm.drop_caches = 2
vm.vfs_cache_pressure = 100
vm.lowmem_reserve_ratio = 32 32 8
net.ipv4.tcp_max_syn_backlog = 8192
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_mem = 178368 524288 33554432
net.ipv4.tcp_wmem = 4096 87380 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.core.netdev_max_backlog = 8912
net.ipv4.ip_local_port_range = 1024 65000
net.core.somaxconn = 8192
net.ipv4.tcp_tw_reuse = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.ipv4.tcp_fin_timeout = 15
net.ipv6.conf.all.forwarding = 1
net.ipv4.conf.all.arp_filter = 0
net.ipv4.conf.all.rp_filter = 0
+ systemctl enable kubelet
+ set -e
+++ dirname /var/lib/sealer/data/my-cluster/rootfs/scripts/docker.sh
++ cd /var/lib/sealer/data/my-cluster/rootfs/scripts
++ pwd
+ scripts_path=/var/lib/sealer/data/my-cluster/rootfs/scripts
+ image_dir=/var/lib/sealer/data/my-cluster/rootfs/scripts/../images
+ DOCKER_VERSION=19.03.14-sealer
+ storage=/var/lib/docker
+ mkdir -p /var/lib/docker
+ utils_command_exists docker
+ command -v docker
++ get_distribution
++ lsb_dist=
++ '[' -r /etc/os-release ']'
+++ . /etc/os-release
++++ NAME='CentOS Linux'
++++ VERSION='7 (Core)'
++++ ID=centos
++++ ID_LIKE='rhel fedora'
++++ VERSION_ID=7
++++ PRETTY_NAME='CentOS Linux 7 (Core)'
++++ ANSI_COLOR='0;31'
++++ CPE_NAME=cpe:/o:centos:centos:7
++++ HOME_URL=https://www.centos.org/
++++ BUG_REPORT_URL=https://bugs.centos.org/
++++ CENTOS_MANTISBT_PROJECT=CentOS-7
++++ CENTOS_MANTISBT_PROJECT_VERSION=7
++++ REDHAT_SUPPORT_PRODUCT=centos
++++ REDHAT_SUPPORT_PRODUCT_VERSION=7
+++ echo centos
++ lsb_dist=centos
++ echo centos
+ lsb_dist=centos
++ echo centos
++ tr '[:upper:]' '[:lower:]'
+ lsb_dist=centos
+ echo 'current system is centos'
current system is centos
+ case "$lsb_dist" in
+ cp /var/lib/sealer/data/my-cluster/rootfs/scripts/../etc/docker.service /usr/lib/systemd/system/docker.service
+ '[' -d /etc/docker/ ']'
+ mkdir /etc/docker/ -p
+ chmod -R 755 /var/lib/sealer/data/my-cluster/rootfs/scripts/../cri
+ tar -zxvf /var/lib/sealer/data/my-cluster/rootfs/scripts/../cri/docker.tar.gz -C /usr/bin
containerd
containerd-shim
ctr
docker
docker-init
docker-proxy
dockerd
runc
+ chmod a+x /usr/bin
+ chmod a+x /usr/bin/docker
+ chmod a+x /usr/bin/dockerd
+ systemctl enable docker.service
+ systemctl restart docker.service
+ cp /var/lib/sealer/data/my-cluster/rootfs/scripts/../etc/daemon.json /etc/docker
+ [[ -n '' ]]
+ disable_selinux
+ '[' -s /etc/selinux/config ']'
+ grep SELINUX=enforcing /etc/selinux/config
+ systemctl daemon-reload
+ systemctl restart docker.service
+ check_docker_valid
+ docker info
Client:
 Debug Mode: false

中间几台主机的信息省略

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 75
 Server Version: 19.03.14-sealer
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: systemd
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ea765aba0d05254012b0b9e595e995c09186427f
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 3.10.0-1127.el7.x86_64
 Operating System: CentOS Linux 7 (Core)
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 62.76GiB
 Name: test-sealer-k8s-work-3.cecdat.com
 ID: GEQR:GJDP:BC2J:PJY6:MAJE:FYDK:NDVP:LAU6:T2O6:DZI5:SG7H:5SVT
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true
 Insecure Registries:
  0.0.0.0/0
  ::/0
  127.0.0.0/8
 Live Restore Enabled: true

++ docker info --format '{{json .ServerVersion}}'
++ tr -d '"'
+ dockerVersion=19.03.14-sealer
+ '[' 19.03.14-sealer '!=' 19.03.14-sealer ']'
+ load_images
+ for image in '"$image_dir"/*'
+ '[' -f /var/lib/sealer/data/my-cluster/rootfs/scripts/../images/docker-amd64-registry-image.tar.gz ']'
+ docker load -q -i /var/lib/sealer/data/my-cluster/rootfs/scripts/../images/docker-amd64-registry-image.tar.gz
Loaded image: registry:2.7.1
+ bash /var/lib/sealer/data/my-cluster/rootfs/scripts/init-kube.sh
+ disable_firewalld
++ get_distribution
++ lsb_dist=
++ '[' -r /etc/os-release ']'
+++ . /etc/os-release
++++ NAME='CentOS Linux'
++++ VERSION='7 (Core)'
++++ ID=centos
++++ ID_LIKE='rhel fedora'
++++ VERSION_ID=7
++++ PRETTY_NAME='CentOS Linux 7 (Core)'
++++ ANSI_COLOR='0;31'
++++ CPE_NAME=cpe:/o:centos:centos:7
++++ HOME_URL=https://www.centos.org/
++++ BUG_REPORT_URL=https://bugs.centos.org/
++++ CENTOS_MANTISBT_PROJECT=CentOS-7
++++ CENTOS_MANTISBT_PROJECT_VERSION=7
++++ REDHAT_SUPPORT_PRODUCT=centos
++++ REDHAT_SUPPORT_PRODUCT_VERSION=7
+++ echo centos
++ lsb_dist=centos
++ echo centos
+ lsb_dist=centos
++ echo centos
++ tr '[:upper:]' '[:lower:]'
+ lsb_dist=centos
+ case "$lsb_dist" in
+ systemctl stop firewalld
Failed to stop firewalld.service: Unit firewalld.service not loaded.
+ copy_bins
+ chmod -R 755 /var/lib/sealer/data/my-cluster/rootfs/scripts/../bin/conntrack /var/lib/sealer/data/my-cluster/rootfs/scripts/../bin/containerd-rootless-setuptool.sh /var/lib/sealer/data/my-cluster/rootfs/scripts/../bin/containerd-rootless.sh /var/lib/sealer/data/my-cluster/rootfs/scripts/../bin/crictl /var/lib/sealer/data/my-cluster/rootfs/scripts/../bin/kubeadm /var/lib/sealer/data/my-cluster/rootfs/scripts/../bin/kubectl /var/lib/sealer/data/my-cluster/rootfs/scripts/../bin/kubelet /var/lib/sealer/data/my-cluster/rootfs/scripts/../bin/nerdctl /var/lib/sealer/data/my-cluster/rootfs/scripts/../bin/seautil
+ chmod 644 /var/lib/sealer/data/my-cluster/rootfs/scripts/../bin
+ cp /var/lib/sealer/data/my-cluster/rootfs/scripts/../bin/conntrack /var/lib/sealer/data/my-cluster/rootfs/scripts/../bin/containerd-rootless-setuptool.sh /var/lib/sealer/data/my-cluster/rootfs/scripts/../bin/containerd-rootless.sh /var/lib/sealer/data/my-cluster/rootfs/scripts/../bin/crictl /var/lib/sealer/data/my-cluster/rootfs/scripts/../bin/kubeadm /var/lib/sealer/data/my-cluster/rootfs/scripts/../bin/kubectl /var/lib/sealer/data/my-cluster/rootfs/scripts/../bin/kubelet /var/lib/sealer/data/my-cluster/rootfs/scripts/../bin/nerdctl /var/lib/sealer/data/my-cluster/rootfs/scripts/../bin/seautil /usr/bin
+ cp /var/lib/sealer/data/my-cluster/rootfs/scripts/../scripts/kubelet-pre-start.sh /usr/bin
+ chmod +x /usr/bin/kubelet-pre-start.sh
+ copy_kubelet_service
+ mkdir -p /etc/systemd/system
+ cp /var/lib/sealer/data/my-cluster/rootfs/scripts/../etc/kubelet.service /etc/systemd/system/
+ '[' -d /etc/systemd/system/kubelet.service.d ']'
+ mkdir /etc/systemd/system/kubelet.service.d
+ cp /var/lib/sealer/data/my-cluster/rootfs/scripts/../etc/10-kubeadm.conf /etc/systemd/system/kubelet.service.d/
+ '[' -d /var/lib/kubelet ']'
+ /usr/bin/kubelet-pre-start.sh
Disabled
# set by ack-distro
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 15
net.ipv6.conf.all.forwarding = 1
net.ipv4.conf.all.arp_filter = 0
net.ipv4.conf.all.rp_filter = 0
* Applying /usr/lib/sysctl.d/00-system.conf ...
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0
* Applying /usr/lib/sysctl.d/10-default-yama-scope.conf ...
kernel.yama.ptrace_scope = 0
* Applying /usr/lib/sysctl.d/50-default.conf ...
kernel.sysrq = 16
kernel.core_uses_pid = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.promote_secondaries = 1
net.ipv4.conf.all.promote_secondaries = 1
fs.protected_hardlinks = 1
fs.protected_symlinks = 1
* Applying /etc/sysctl.d/99-sysctl.conf ...
vm.dirty_ratio = 20
vm.dirty_background_ratio = 5
vm.dirty_writeback_centisecs = 200
vm.dirty_expire_centisecs = 100
vm.drop_caches = 2
vm.vfs_cache_pressure = 100
vm.lowmem_reserve_ratio = 32 32 8
net.ipv4.tcp_max_syn_backlog = 8192
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_mem = 178368 524288 33554432
net.ipv4.tcp_wmem = 4096 87380 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.core.netdev_max_backlog = 8912
net.ipv4.ip_local_port_range = 1024 65000
net.core.somaxconn = 8192
net.ipv4.tcp_tw_reuse = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.ipv4.tcp_fin_timeout = 15
net.ipv6.conf.all.forwarding = 1
net.ipv4.conf.all.arp_filter = 0
net.ipv4.conf.all.rp_filter = 0
* Applying /etc/sysctl.d/k8s.conf ...
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 15
net.ipv6.conf.all.forwarding = 1
net.ipv4.conf.all.arp_filter = 0
net.ipv4.conf.all.rp_filter = 0
* Applying /etc/sysctl.conf ...
vm.dirty_ratio = 20
vm.dirty_background_ratio = 5
vm.dirty_writeback_centisecs = 200
vm.dirty_expire_centisecs = 100
vm.drop_caches = 2
vm.vfs_cache_pressure = 100
vm.lowmem_reserve_ratio = 32 32 8
net.ipv4.tcp_max_syn_backlog = 8192
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_mem = 178368 524288 33554432
net.ipv4.tcp_wmem = 4096 87380 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.core.netdev_max_backlog = 8912
net.ipv4.ip_local_port_range = 1024 65000
net.core.somaxconn = 8192
net.ipv4.tcp_tw_reuse = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.ipv4.tcp_fin_timeout = 15
net.ipv6.conf.all.forwarding = 1
net.ipv4.conf.all.arp_filter = 0
net.ipv4.conf.all.rp_filter = 0
copying files to 172.20.58.88: 283/283 
copying files to 172.20.57.108: 283/283 
copying files to 172.20.58.189: 283/283 

++ dirname init-registry.sh
+ cd .
+ REGISTRY_PORT=5000
+ VOLUME=/var/lib/sealer/data/my-cluster/rootfs/registry
+ REGISTRY_DOMAIN=sea.hub
+ container=sealer-registry
+++ pwd
++ dirname /var/lib/sealer/data/my-cluster/rootfs/scripts
+ rootfs=/var/lib/sealer/data/my-cluster/rootfs
+ config=/var/lib/sealer/data/my-cluster/rootfs/etc/registry_config.yml
+ htpasswd=/var/lib/sealer/data/my-cluster/rootfs/etc/registry_htpasswd
+ certs_dir=/var/lib/sealer/data/my-cluster/rootfs/certs
+ image_dir=/var/lib/sealer/data/my-cluster/rootfs/images
+ mkdir -p /var/lib/sealer/data/my-cluster/rootfs/registry
+ load_images
+ for image in '"$image_dir"/*'
+ '[' -f /var/lib/sealer/data/my-cluster/rootfs/images/docker-amd64-registry-image.tar.gz ']'
+ docker load -q -i /var/lib/sealer/data/my-cluster/rootfs/images/docker-amd64-registry-image.tar.gz
Loaded image: registry:2.7.1
++ docker ps -aq -f name=sealer-registry
+ '[' '' ']'
+ regArgs='-d --restart=always --net=host --name sealer-registry -v /var/lib/sealer/data/my-cluster/rootfs/certs:/certs -v /var/lib/sealer/data/my-cluster/rootfs/registry:/var/lib/registry -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/sea.hub.crt -e REGISTRY_HTTP_TLS_KEY=/certs/sea.hub.key -e REGISTRY_HTTP_DEBUG_ADDR=0.0.0.0:5002 -e REGISTRY_HTTP_DEBUG_PROMETHEUS_ENABLED=true'
+ '[' -f /var/lib/sealer/data/my-cluster/rootfs/etc/registry_config.yml ']'
+ sed -i s/5000/5000/g /var/lib/sealer/data/my-cluster/rootfs/etc/registry_config.yml
+ regArgs='-d --restart=always --net=host --name sealer-registry -v /var/lib/sealer/data/my-cluster/rootfs/certs:/certs -v /var/lib/sealer/data/my-cluster/rootfs/registry:/var/lib/registry -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/sea.hub.crt -e REGISTRY_HTTP_TLS_KEY=/certs/sea.hub.key -e REGISTRY_HTTP_DEBUG_ADDR=0.0.0.0:5002 -e REGISTRY_HTTP_DEBUG_PROMETHEUS_ENABLED=true     -v /var/lib/sealer/data/my-cluster/rootfs/etc/registry_config.yml:/etc/docker/registry/config.yml'
+ '[' -f /var/lib/sealer/data/my-cluster/rootfs/etc/registry_htpasswd ']'
+ docker run -d --restart=always --net=host --name sealer-registry -v /var/lib/sealer/data/my-cluster/rootfs/certs:/certs -v /var/lib/sealer/data/my-cluster/rootfs/registry:/var/lib/registry -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/sea.hub.crt -e REGISTRY_HTTP_TLS_KEY=/certs/sea.hub.key -e REGISTRY_HTTP_DEBUG_ADDR=0.0.0.0:5002 -e REGISTRY_HTTP_DEBUG_PROMETHEUS_ENABLED=true -v /var/lib/sealer/data/my-cluster/rootfs/etc/registry_config.yml:/etc/docker/registry/config.yml registry:2.7.1
9312e55dd619b57de405b2d51c8a0f595d1a794ab5aade4d7b6f9cf30eb28ff4
+ check_registry
+ n=1
+ ((  n <= 3  ))
++ docker inspect --format '{{json .State.Status}}' sealer-registry
+ registry_status='"running"'
+ [[ "running" == \"running\" ]]
+ break
++ dirname init-registry.sh
+ cd .
+ REGISTRY_PORT=5000
+ VOLUME=/var/lib/sealer/data/my-cluster/rootfs/registry
+ REGISTRY_DOMAIN=sea.hub
+ container=sealer-registry
+++ pwd
++ dirname /var/lib/sealer/data/my-cluster/rootfs/scripts
+ rootfs=/var/lib/sealer/data/my-cluster/rootfs
+ config=/var/lib/sealer/data/my-cluster/rootfs/etc/registry_config.yml
+ htpasswd=/var/lib/sealer/data/my-cluster/rootfs/etc/registry_htpasswd
+ certs_dir=/var/lib/sealer/data/my-cluster/rootfs/certs
+ image_dir=/var/lib/sealer/data/my-cluster/rootfs/images
+ mkdir -p /var/lib/sealer/data/my-cluster/rootfs/registry
+ load_images
+ for image in '"$image_dir"/*'
+ '[' -f /var/lib/sealer/data/my-cluster/rootfs/images/docker-amd64-registry-image.tar.gz ']'
+ docker load -q -i /var/lib/sealer/data/my-cluster/rootfs/images/docker-amd64-registry-image.tar.gz
Loaded image: registry:2.7.1
++ docker ps -aq -f name=sealer-registry
+ '[' '' ']'
+ regArgs='-d --restart=always --net=host --name sealer-registry -v /var/lib/sealer/data/my-cluster/rootfs/certs:/certs -v /var/lib/sealer/data/my-cluster/rootfs/registry:/var/lib/registry -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/sea.hub.crt -e REGISTRY_HTTP_TLS_KEY=/certs/sea.hub.key -e REGISTRY_HTTP_DEBUG_ADDR=0.0.0.0:5002 -e REGISTRY_HTTP_DEBUG_PROMETHEUS_ENABLED=true'
+ '[' -f /var/lib/sealer/data/my-cluster/rootfs/etc/registry_config.yml ']'
+ sed -i s/5000/5000/g /var/lib/sealer/data/my-cluster/rootfs/etc/registry_config.yml
+ regArgs='-d --restart=always --net=host --name sealer-registry -v /var/lib/sealer/data/my-cluster/rootfs/certs:/certs -v /var/lib/sealer/data/my-cluster/rootfs/registry:/var/lib/registry -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/sea.hub.crt -e REGISTRY_HTTP_TLS_KEY=/certs/sea.hub.key -e REGISTRY_HTTP_DEBUG_ADDR=0.0.0.0:5002 -e REGISTRY_HTTP_DEBUG_PROMETHEUS_ENABLED=true     -v /var/lib/sealer/data/my-cluster/rootfs/etc/registry_config.yml:/etc/docker/registry/config.yml'
+ '[' -f /var/lib/sealer/data/my-cluster/rootfs/etc/registry_htpasswd ']'
+ docker run -d --restart=always --net=host --name sealer-registry -v /var/lib/sealer/data/my-cluster/rootfs/certs:/certs -v /var/lib/sealer/data/my-cluster/rootfs/registry:/var/lib/registry -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/sea.hub.crt -e REGISTRY_HTTP_TLS_KEY=/certs/sea.hub.key -e REGISTRY_HTTP_DEBUG_ADDR=0.0.0.0:5002 -e REGISTRY_HTTP_DEBUG_PROMETHEUS_ENABLED=true -v /var/lib/sealer/data/my-cluster/rootfs/etc/registry_config.yml:/etc/docker/registry/config.yml registry:2.7.1
3ad0a736fb4771eb770aae46ecb090a40a5d4afd1af22b09cbffd34337791e05
+ check_registry
+ n=1
+ ((  n <= 3  ))
++ docker inspect --format '{{json .State.Status}}' sealer-registry
+ registry_status='"running"'
+ [[ "running" == \"running\" ]]
+ break
++ dirname init-registry.sh
+ cd .
+ REGISTRY_PORT=5000
+ VOLUME=/var/lib/sealer/data/my-cluster/rootfs/registry
+ REGISTRY_DOMAIN=sea.hub
+ container=sealer-registry
+++ pwd
++ dirname /var/lib/sealer/data/my-cluster/rootfs/scripts
+ rootfs=/var/lib/sealer/data/my-cluster/rootfs
+ config=/var/lib/sealer/data/my-cluster/rootfs/etc/registry_config.yml
+ htpasswd=/var/lib/sealer/data/my-cluster/rootfs/etc/registry_htpasswd
+ certs_dir=/var/lib/sealer/data/my-cluster/rootfs/certs
+ image_dir=/var/lib/sealer/data/my-cluster/rootfs/images
+ mkdir -p /var/lib/sealer/data/my-cluster/rootfs/registry
+ load_images
+ for image in '"$image_dir"/*'
+ '[' -f /var/lib/sealer/data/my-cluster/rootfs/images/docker-amd64-registry-image.tar.gz ']'
+ docker load -q -i /var/lib/sealer/data/my-cluster/rootfs/images/docker-amd64-registry-image.tar.gz
Loaded image: registry:2.7.1
++ docker ps -aq -f name=sealer-registry
+ '[' '' ']'
+ regArgs='-d --restart=always --net=host --name sealer-registry -v /var/lib/sealer/data/my-cluster/rootfs/certs:/certs -v /var/lib/sealer/data/my-cluster/rootfs/registry:/var/lib/registry -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/sea.hub.crt -e REGISTRY_HTTP_TLS_KEY=/certs/sea.hub.key -e REGISTRY_HTTP_DEBUG_ADDR=0.0.0.0:5002 -e REGISTRY_HTTP_DEBUG_PROMETHEUS_ENABLED=true'
+ '[' -f /var/lib/sealer/data/my-cluster/rootfs/etc/registry_config.yml ']'
+ sed -i s/5000/5000/g /var/lib/sealer/data/my-cluster/rootfs/etc/registry_config.yml
+ regArgs='-d --restart=always --net=host --name sealer-registry -v /var/lib/sealer/data/my-cluster/rootfs/certs:/certs -v /var/lib/sealer/data/my-cluster/rootfs/registry:/var/lib/registry -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/sea.hub.crt -e REGISTRY_HTTP_TLS_KEY=/certs/sea.hub.key -e REGISTRY_HTTP_DEBUG_ADDR=0.0.0.0:5002 -e REGISTRY_HTTP_DEBUG_PROMETHEUS_ENABLED=true     -v /var/lib/sealer/data/my-cluster/rootfs/etc/registry_config.yml:/etc/docker/registry/config.yml'
+ '[' -f /var/lib/sealer/data/my-cluster/rootfs/etc/registry_htpasswd ']'
+ docker run -d --restart=always --net=host --name sealer-registry -v /var/lib/sealer/data/my-cluster/rootfs/certs:/certs -v /var/lib/sealer/data/my-cluster/rootfs/registry:/var/lib/registry -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/sea.hub.crt -e REGISTRY_HTTP_TLS_KEY=/certs/sea.hub.key -e REGISTRY_HTTP_DEBUG_ADDR=0.0.0.0:5002 -e REGISTRY_HTTP_DEBUG_PROMETHEUS_ENABLED=true -v /var/lib/sealer/data/my-cluster/rootfs/etc/registry_config.yml:/etc/docker/registry/config.yml registry:2.7.1
76a693e84632995f1c1765cbf4598133867a200a66a14a68989f5eb221d89f5d
+ check_registry
+ n=1
+ ((  n <= 3  ))
++ docker inspect --format '{{json .State.Status}}' sealer-registry
+ registry_status='"running"'
copying files to 172.20.58.164: 32/32 
copying files to 172.20.58.88: 284/284 
copying files to 172.20.57.108: 284/284 
copying files to 172.20.58.189: 284/284 
copying files to 172.20.59.176: 32/32 
copying files to 172.20.59.212: 32/32 
2023-02-16 15:46:27 [INFO] [remove_container.go:66] b88a7b6071c9e41716cdebc555dc3cd7f0e446bf1087342228f877f569d322d1

Usage:
  sealer run [flags]

Examples:

run cluster by Clusterfile: 
  sealer run -f Clusterfile

run cluster by CLI flags:
  sealer run docker.io/sealerio/kubernetes:v1.22.15 -m 172.28.80.01 -n 172.28.80.02 -p Sealer123

run app image:
  sealer run localhost/nginx:v1

Flags:
  -f, --Clusterfile string   Clusterfile path to run a Kubernetes cluster
      --apps strings         override default AppNames of clusterimage
      --cmds strings         override default LaunchCmds of clusterimage
  -e, --env strings          set custom environment variables
  -h, --help                 help for run
  -m, --masters string       set count or IPList to masters
      --mode string          load images to the specified registry in advance (default "apply")
  -n, --nodes string         set count or IPList to nodes
  -p, --passwd string        set cloud provider or baremetal server password
      --pk string            set baremetal server private key (default "/root/.ssh/id_rsa")
      --pk-passwd string     set baremetal server private key password
      --port uint16          set the sshd service port number for the server (default port: 22) (default 22)
  -u, --user string          set baremetal server username (default "root")

Global Flags:
      --color string               set the log color mode, the possible values can be [never always] (default "always")
      --config string              config file of sealer tool (default is $HOME/.sealer.json)
  -d, --debug                      turn on debug mode
      --hide-path                  hide the log path
      --hide-time                  hide the log time
      --log-to-file                write log message to disk
  -q, --quiet                      silence the usage when fail
      --remote-logger-url string   remote logger url, if not empty, will send log to this url
      --task-name string           task name which will embedded in the remote logger header, only valid when --remote-logger-url is set

2023-02-16 15:46:27 [ERROR] [root.go:75] sealer-v0.9.1: failed to found kubeadm config from /var/lib/sealer/data/my-cluster/rootfs/etc/kubeadm.yml: failed to open configfile(/var/lib/sealer/data/my-cluster/rootfs/etc/kubeadm.yml): open /var/lib/sealer/data/my-cluster/rootfs/etc/kubeadm.yml: no such file or directory

[root@test-shiyan ~]# timed out waiting for input: auto-logout

Last login: Thu Feb 16 15:36:33 2023 from 172.16.5.2
[root@test-shiyan ~]# ls /var/lib/sealer/data/my-cluster/rootfs
certs
[root@test-shiyan ~]# df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        7.8G     0  7.8G   0% /dev
tmpfs           7.8G     0  7.8G   0% /dev/shm
tmpfs           7.8G   17M  7.8G   1% /run
tmpfs           7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/vda1        80G   51G   30G  64% /
tmpfs           1.6G     0  1.6G   0% /run/user/0
[root@test-shiyan ~]# sealer alpha mount e4f4f174bf27
2023-02-16 15:55:39 [INFO] [mount.go:111] mount cluster image e4f4f174bf27 to /var/lib/sealer/data/overlay2/e4f4f174bf27fe397d579279d599fb191ab31510f4464fc37bb3f891818e8856 successful

[root@test-shiyan ~]# df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        7.8G     0  7.8G   0% /dev
tmpfs           7.8G     0  7.8G   0% /dev/shm
tmpfs           7.8G   17M  7.8G   1% /run
tmpfs           7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/vda1        80G   51G   30G  64% /
tmpfs           1.6G     0  1.6G   0% /run/user/0
overlay          80G   51G   30G  64% /var/lib/containers/storage/overlay/84e5ad09f6a2b1c1036394472a17e60986aea6b97f3f4c59e92c7a7e3446f6e5/merged
[root@test-shiyan ~]# ls /var/lib/containers/storage/overlay/84e5ad09f6a2b1c1036394472a17e60986aea6b97f3f4c59e92c7a7e3446f6e5/merged/
application/ cri/         images/      manifests/   scripts/     
bin/         etc/         lib/         registry/    statics/     
[root@test-shiyan ~]# ls /var/lib/containers/storage/overlay/84e5ad09f6a2b1c1036394472a17e60986aea6b97f3f4c59e92c7a7e3446f6e5/merged/etc
10-kubeadm.conf  docker.service  kubeadm.yml       kubelet.service
daemon.json      kubeadm         kubeadm.yml.tmpl  registry_config.yml
[root@test-shiyan ~]# grep kubernetesVersion /var/lib/containers/storage/overlay/84e5ad09f6a2b1c1036394472a17e60986aea6b97f3f4c59e92c7a7e3446f6e5/merged/etc/kubeadm.yml
kubernetesVersion: v1.20.4

镜像信息：
[root@test-shiyan ~]# sealer inspect e4f4f174bf27
{
    "id": "e4f4f174bf27fe397d579279d599fb191ab31510f4464fc37bb3f891818e8856",
    "name": "docker.io/sealerio/kubernetes:v1.20.4-sealerio-2",
    "digest": "sha256:9abdc17f7ce44fd579664a5977c09313453ef964c531669c1c90b182d3136e85",
    "manifestv1": {
        "schemaVersion": 2,
        "mediaType": "application/vnd.oci.image.manifest.v1+json",
        "config": {
            "mediaType": "application/vnd.oci.image.config.v1+json",
            "digest": "sha256:e4f4f174bf27fe397d579279d599fb191ab31510f4464fc37bb3f891818e8856",
            "size": 2983
        },
        "layers": [
            {
                "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
                "digest": "sha256:fd49fc3d50adf51edbeb5aad972dd0a86de5e02a40d626e87cde1aea630b6e7c",
                "size": 13881192
            },
            {
                "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
                "digest": "sha256:15992a602424f5c2e8e81c7e84ecd9a1202634468660a1e5b6b5284ab7570f71",
                "size": 158048288
            },
            {
                "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
                "digest": "sha256:cb017b6e827065baf904f521e3d20615f071d4aeb4a2bedcf99dfde5b8f8bd9c",
                "size": 327
            },
            {
                "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
                "digest": "sha256:c9c659d1211026401ce03b417a66cd015dbab3c889b0ce9e06ea93206231553b",
                "size": 44656
            },
            {
                "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
                "digest": "sha256:a8ad706555812653500c715a0a1291485d10f5b7f9b245da5ae601ad07a904a6",
                "size": 578992159
            }
        ],
        "annotations": {
            "org.opencontainers.image.base.digest": "sha256:5b1a1e0139b534beef17d355ae4a961147fd3e60216da52d32314d9b7a458a3e",
            "org.opencontainers.image.base.name": "docker.io/sealerio/kubernetes:v1.20.4-sealerio-2379ed402"
        }
    },
    "ociv1": {
        "created": "2023-01-17T11:38:28.081376677Z",
        "architecture": "amd64",
        "os": "linux",
        "config": {
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ]
        },
        "rootfs": {
            "type": "layers",
            "diff_ids": [
                "sha256:4186f15f868c249ff7c341ee61267af25696c66c117860c5588438d832326833",
                "sha256:dfa941cd0523bd9641f5c06e9d0933a672d6106396e74375fa67cb74a7a5c3e9",
                "sha256:a8ffcceab9eaf268a6dc00832bb92de1f5dfc70096b3d4c6e253b8500bf6360b",
                "sha256:f55d72f346be940b42347b888a48d53654db82b07025ef05b0b0e1b1b67cb38d",
                "sha256:0cb9931698fd8e348c3f524e0aa43edd6409693f9ac3a061d99ccf7319788191"
            ]
        },
        "history": [
            {
                "created": "2023-01-17T11:37:21.863285367Z",
                "created_by": "/bin/sh -c #(nop) COPY dir:0ad33367de0843eb2709049f11c758274388a88d14b3f6c1276ab73ab0365d1b in . "
            },
            {
                "created": "2023-01-17T11:37:25.778678982Z",
                "created_by": "/bin/sh -c #(nop) COPY dir:a6785b546d5b84666289f033b437463ca1e34ef2d22df30ca52ac018bcb754cc in . ",
                "comment": "FROM d97d33dafe03"
            },
            {
                "created": "2023-01-17T11:37:27.512280786Z",
                "created_by": "/bin/sh -c #(nop) COPY file:02d2c4971ffca259c4c504b2e23dace9133cef77c27ffa4884d1c40c6aea1f51 in manifests ",
                "comment": "FROM 2f346ec23b72"
            },
            {
                "created": "2023-01-17T11:37:27.617935218Z",
                "created_by": "/bin/sh -c #(nop) LABEL \"cluster.alpha.sealer.io/cluster-runtime-version\"=\"v1.20.4\"",
                "comment": "FROM 3d5685e7745c",
                "empty_layer": true
            },
            {
                "created": "2023-01-17T11:37:27.651618271Z",
                "created_by": "/bin/sh -c #(nop) LABEL \"cluster.alpha.sealer.io/cluster-runtime-type\"=\"kubernetes\"",
                "comment": "FROM b768f9f4d54d",
                "empty_layer": true
            },
            {
                "created": "2023-01-17T11:37:27.685405773Z",
                "created_by": "/bin/sh -c #(nop) LABEL \"cluster.alpha.sealer.io/container-runtime-type\"=\"docker\"",
                "comment": "FROM ea05840a1600",
                "empty_layer": true
            },
            {
                "created": "2023-01-17T11:37:27.719864819Z",
                "created_by": "/bin/sh -c #(nop) LABEL \"cluster.alpha.sealer.io/container-runtime-version\"=\"19.03.14\"",
                "comment": "FROM 85c3584a71af",
                "empty_layer": true
            },
            {
                "created": "2023-01-17T11:37:28.011295364Z",
                "created_by": "/bin/sh -c #(nop) COPY multi:09ae96dfd662d761abe4a01ca941ef98b7e966e35996b4fea1ba6a2f0d0a9a42 in application/apps/calico/ ",
                "comment": "FROM 55b1add2239a"
            },
            {
                "created": "2023-01-17T11:37:28.12520534Z",
                "created_by": "/bin/sh -c #(nop) LABEL cluster.alpha.sealer.io/kube-cni-calico=\"true\"",
                "comment": "FROM 5abb9b082b4b",
                "empty_layer": true
            },
            {
                "created": "2023-01-17T11:37:28.160612524Z",
                "created_by": "/bin/sh -c #(nop) LABEL \"io.sealer.version\"=\"v0.9.1\"",
                "comment": "FROM 9acba8dc48e6",
                "empty_layer": true
            },
            {
                "created": "2023-01-17T11:38:30.717936894Z",
                "created_by": "/bin/sh",
                "comment": "FROM docker.io/sealerio/kubernetes:v1.20.4-sealerio-2379ed402"
            }
        ]
    },
    "buildClient": {
        "sealerVersion": "v0.9.1",
        "buildahVersion": "1.27.1"
    },
    "schemaVersion": "v1alpha1",
    "type": "kube-installer",
    "applications": [
        {
            "name": "calico",
            "type": "kube",
            "files": [
                "tigera-operator.yaml",
                "custom-resources.yaml"
            ],
            "version": "v1"
        }
    ],
    "labels": {
        "cluster.alpha.sealer.io/cluster-runtime-type": "kubernetes",
        "cluster.alpha.sealer.io/cluster-runtime-version": "v1.20.4",
        "cluster.alpha.sealer.io/container-runtime-type": "docker",
        "cluster.alpha.sealer.io/container-runtime-version": "19.03.14",
        "cluster.alpha.sealer.io/kube-cni": "[\"calico\"]",
        "io.buildah.version": "1.27.1",
        "io.sealer.version": "v0.9.1"
    },
    "launch": {
        "app_names": [
            "calico"
        ]
    },
    "containerImageList": [
        {
            "image": "docker.io/sealerio/lvscare:v1.1.3-beta.8",
            "platform": {
                "architecture": "amd64",
                "os": "linux"
            }
        },
        {
            "image": "calico/node:v3.22.1",
            "platform": {
                "architecture": "amd64",
                "os": "linux"
            }
        },
        {
            "image": "calico/pod2daemon-flexvol:v3.22.1",
            "platform": {
                "architecture": "amd64",
                "os": "linux"
            }
        },
        {
            "image": "calico/cni:v3.22.1",
            "platform": {
                "architecture": "amd64",
                "os": "linux"
            }
        },
        {
            "image": "calico/kube-controllers:v3.22.1",
            "platform": {
                "architecture": "amd64",
                "os": "linux"
            }
        },
        {
            "image": "calico/typha:v3.22.1",
            "platform": {
                "architecture": "amd64",
                "os": "linux"
            }
        },
        {
            "image": "calico/apiserver:v3.22.1",
            "platform": {
                "architecture": "amd64",
                "os": "linux"
            }
        },
        {
            "image": "k8s.gcr.io/kube-apiserver:v1.20.4",
            "platform": {
                "architecture": "amd64",
                "os": "linux"
            }
        },
        {
            "image": "k8s.gcr.io/kube-controller-manager:v1.20.4",
            "platform": {
                "architecture": "amd64",
                "os": "linux"
            }
        },
        {
            "image": "k8s.gcr.io/kube-scheduler:v1.20.4",
            "platform": {
                "architecture": "amd64",
                "os": "linux"
            }
        },
        {
            "image": "k8s.gcr.io/kube-proxy:v1.20.4",
            "platform": {
                "architecture": "amd64",
                "os": "linux"
            }
        },
        {
            "image": "k8s.gcr.io/pause:3.2",
            "platform": {
                "architecture": "amd64",
                "os": "linux"
            }
        },
        {
            "image": "k8s.gcr.io/etcd:3.4.13-0",
            "platform": {
                "architecture": "amd64",
                "os": "linux"
            }
        },
        {
            "image": "docker.io/coredns/coredns:1.7.0",
            "platform": {
                "architecture": "amd64",
                "os": "linux"
            }
        },
        {
            "image": "quay.io/tigera/operator:v1.25.3",
            "appName": "calico",
            "platform": {
                "architecture": "amd64",
                "os": "linux"
            }
        }
    ]
}

What you expected to happen?

使用sealer0.91版本能成功正常安装kubernetes v1.20.4, v1.22.15版本的集群

How to reproduce it (as minimally and precisely as possible)?

No response

Anything else we need to know?

No response

What is the version of Sealer you using?

{"gitVersion":"v0.9.1","gitCommit":"9934e11","buildDate":"2023-01-17 11:30:38","goVersion":"go1.17.13","compiler":"gc","platform":"linux/amd64"}

What is your OS environment?

CentOS Linux release 7.8.2003 (Core)

What is the Kernel version?

Linux test-shiyan.cecdat.com 3.10.0-1127.el7.x86_64 #1 SMP Tue Mar 31 23:36:51 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Other environment you want to tell us?

No response

kakaZhou719 commented 1 year ago

@Stevent-fei ,could you help us to troubleshoot this issue ?

Jackmeipx commented 1 year ago

有人在分析定位该问题吗？ 在arm架构的麒麟系统上部署，报错一样。 @kakaZhou719 @Stevent-fei