search

sealerio / sealer

Build, Share and Run Both Your Kubernetes Cluster and Distributed Applications (Project under CNCF)
http://sealer.cool
Apache License 2.0
2.06k stars 362 forks source link

can't to join nodes to cluster #2284

Closed studycloudnative closed 1 year ago

studycloudnative commented 1 year ago

What happen?

sealer version
{"gitVersion":"latest","gitCommit":"2a4d201","buildDate":"2022-04-04 01:10:39","goVersion":"go1.16.15","compiler":"gc","platform":"linux/amd64"}
apiVersion: sealer.cloud/v2
kind: Cluster
metadata:
  annotations:
    ClusterfileName: /root/rapid_init_cluster/Clusterfile
  creationTimestamp: null
  name: k8s-cluster
spec:
  env:
  - cluster_name=europe
  - harbor_address=10.246.15.168
  - time_zone=Europe/Berlin
  - docker_dir=/data1/docker
  - eth_name=eth0
  - disk_drive=nvme1n1
  - disk_mount_path=/data1
  hosts:
  - ips:
    - xxxxx
    - xxxxx
    - xxxxx
    roles:
    - master
    ssh: {}
  - ips:
    - xxxxx
    - xxxxx
    - xxxxx
    - xxxxx
    roles:
    - node
    ssh: {}
  image: kubernetes:latest
  ssh:
    passwd: xxx
    user: root
status: {}

Relevant log output?

the debug log 

2023-08-07 14:25:37 [DEBG] [scp.go:293] remote dst /var/lib/sealer/data/k8s-cluster/rootfs/bin/helm already exists and is tcopying files to 10.246.15.92: 9/14 
2023-08-07 14:25:38 [DEBG] [scp.go:293] remote dst /var/lib/sealer/data/k8s-cluster/rootfs/bin/kubeadm already exists and icopying files to 10.246.15.92: 10/14 
2023-08-07 14:25:38 [DEBG] [scp.go:293] remote dst /var/lib/sealer/data/k8s-cluster/rootfs/bin/kubectl already exists and icopying files to 10.246.15.92: 11/14 
2023-08-07 14:25:39 [DEBG] [scp.go:293] remote dst /var/lib/sealer/data/k8s-cluster/rootfs/bin/kubelet already exists and icopying files to 10.246.15.92: 12/14 
2023-08-07 14:25:40 [DEBG] [scp.go:293] remote dst /var/lib/sealer/data/k8s-cluster/rootfs/bin/nerdctl already exists and icopying files to 10.246.15.92: 13/14 
2023-08-07 14:25:40 [DEBG] [scp.go:293] remote dst /var/lib/sealer/data/k8s-cluster/rootfs/bin/seautil already exists and icopying files to 10.246.15.92: 14/14 
2023-08-07 14:25:40 [DEBG] [scp.go:200] remote copy files src /var/lib/sealer/data/k8s-cluster/mount/cri to dst /var/lib/secopying files to 10.246.15.92: 14/15 
2023-08-07 14:25:41 [DEBG] [scp.go:293] remote dst /var/lib/sealer/data/k8s-cluster/rootfs/cri/docker.tar.gz already existscopying files to 10.246.15.92: 15/15 
2023-08-07 14:25:41 [DEBG] [scp.go:200] remote copy files src /var/lib/sealer/data/k8s-cluster/mount/etc to dst /var/lib/secopying files to 10.246.15.92: 15/25 
2023-08-07 14:25:42 [DEBG] [scp.go:293] remote dst /var/lib/sealer/data/k8s-cluster/rootfs/etc/10-kubeadm.conf already exiscopying files to 10.246.15.92: 16/25 
2023-08-07 14:25:42 [DEBG] [scp.go:293] remote dst /var/lib/sealer/data/k8s-cluster/rootfs/etc/Clusterfile already exists acopying files to 10.246.15.92: 17/25 
2023-08-07 14:25:43 [DEBG] [scp.go:293] remote dst /var/lib/sealer/data/k8s-cluster/rootfs/etc/custom-resources.yaml alreadcopying files to 10.246.15.92: 18/25 
2023-08-07 14:25:43 [DEBG] [scp.go:293] remote dst /var/lib/sealer/data/k8s-cluster/rootfs/etc/daemon.json already exists acopying files to 10.246.15.92: 19/25 
2023-08-07 14:25:43 [DEBG] [scp.go:293] remote dst /var/lib/sealer/data/k8s-cluster/rootfs/etc/docker.service already existcopying files to 10.246.15.92: 20/25 
2023-08-07 14:25:44 [DEBG] [scp.go:293] remote dst /var/lib/sealer/data/k8s-cluster/rootfs/etc/kubeadm.yml already exists acopying files to 10.246.15.92: 21/25 
2023-08-07 14:25:44 [DEBG] [scp.go:293] remote dst /var/lib/sealer/data/k8s-cluster/rootfs/etc/kubelet.service already exiscopying files to 10.246.15.92: 22/25 
2023-08-07 14:25:45 [DEBG] [scp.go:293] remote dst /var/lib/sealer/data/k8s-cluster/rootfs/etc/registry.yml already exists copying files to 10.246.15.92: 23/25 
2023-08-07 14:25:45 [DEBG] [scp.go:293] remote dst /var/lib/sealer/data/k8s-cluster/rootfs/etc/registry_config.yml already copying files to 10.246.15.92: 24/25 
2023-08-07 14:25:46 [DEBG] [scp.go:293] remote dst /var/lib/sealer/data/k8s-cluster/rootfs/etc/tigera-operator.yaml alreadycopying files to 10.246.15.92: 25/25 
2023-08-07 14:25:46 [DEBG] [scp.go:200] remote copy files src /var/lib/sealer/data/k8s-cluster/mount/imageList to dst /var/copying files to 10.246.15.92: 25/26 
2023-08-07 14:25:46 [DEBG] [scp.go:293] remote dst /var/lib/sealer/data/k8s-cluster/rootfs/imageList already exists and is copying files to 10.246.15.92: 26/26 
2023-08-07 14:25:46 [DEBG] [scp.go:200] remote copy files src /var/lib/sealer/data/k8s-cluster/mount/images to dst /var/libcopying files to 10.246.15.92: 26/27 
2023-08-07 14:25:47 [DEBG] [scp.go:293] remote dst /var/lib/sealer/data/k8s-cluster/rootfs/images/registry.tar already exiscopying files to 10.246.15.92: 27/27 
2023-08-07 14:25:47 [DEBG] [scp.go:200] remote copy files src /var/lib/sealer/data/k8s-cluster/mount/manifests to dst /var/copying files to 10.246.15.92: 27/28 
2023-08-07 14:25:48 [DEBG] [scp.go:293] remote dst /var/lib/sealer/data/k8s-cluster/rootfs/manifests/imageList already exiscopying files to 10.246.15.92: 28/28 
2023-08-07 14:25:48 [DEBG] [scp.go:200] remote copy files src /var/lib/sealer/data/k8s-cluster/mount/scripts to dst /var/licopying files to 10.246.15.92: 28/57 
2023-08-07 14:25:48 [DEBG] [scp.go:293] remote dst /var/lib/sealer/data/k8s-cluster/rootfs/scripts/NewBuild_V2.0_Monitor/zacopying files to 10.246.15.92: 29/57 
2023-08-07 14:25:49 [DEBG] [scp.go:293] remote dst /var/lib/sealer/data/k8s-cluster/rootfs/scripts/NewBuild_V2.0_Monitor/zacopying files to 10.246.15.92: 30/57 
2023-08-07 14:25:49 [DEBG] [scp.go:293] remote dst /var/lib/sealer/data/k8s-cluster/rootfs/scripts/NewBuild_V2.0_Monitor/zacopying files to 10.246.15.92: 31/57 
2023-08-07 14:25:50 [DEBG] [scp.go:293] remote dst /var/lib/sealer/data/k8s-cluster/rootfs/scripts/NewBuild_V2.0_Monitor/zacopying files to 10.246.15.92: 32/57 
2023-08-07 14:25:50 [DEBG] [scp.go:293] remote dst /var/lib/sealer/data/k8s-cluster/rootfs/scripts/NewBuild_V2.0_Monitor/zacopying files to 10.246.15.92: 33/57 
2023-08-07 14:25:50 [DEBG] [scp.go:293] remote dst /var/lib/sealer/data/k8s-cluster/rootfs/scripts/NewBuild_V2.0_Monitor/zacopying files to 10.246.15.92: 34/57 
2023-08-07 14:25:51 [DEBG] [scp.go:293] remote dst /var/lib/sealer/data/k8s-cluster/rootfs/scripts/NewBuild_V2.0_Monitor/zacopying files to 10.246.15.92: 35/57 
2023-08-07 14:25:51 [DEBG] [scp.go:293] remote dst /var/lib/sealer/data/k8s-cluster/rootfs/scripts/NewBuild_V2.0_Monitor/zacopying files to 10.246.15.92: 36/57 
2023-08-07 14:25:52 [DEBG] [scp.go:293] remote dst /var/lib/sealer/data/k8s-cluster/rootfs/scripts/NewBuild_V2.0_Monitor/zacopying files to 10.246.15.92: 37/57 
2023-08-07 14:25:52 [DEBG] [scp.go:293] remote dst /var/lib/sealer/data/k8s-cluster/rootfs/scripts/NewBuild_V2.0_Monitor/zacopying files to 10.246.15.92: 38/57 
2023-08-07 14:25:53 [DEBG] [scp.go:293] remote dst /var/lib/sealer/data/k8s-cluster/rootfs/scripts/NewBuild_V2.0_Monitor/zacopying files to 10.246.15.92: 39/57 
2023-08-07 14:25:53 [DEBG] [scp.go:293] remote dst /var/lib/sealer/data/k8s-cluster/rootfs/scripts/NewBuild_V2.0_Monitor/zacopying files to 10.246.15.92: 40/57 
2023-08-07 14:25:54 [DEBG] [scp.go:293] remote dst /var/lib/sealer/data/k8s-cluster/rootfs/scripts/NewBuild_V2.0_Monitor/zacopying files to 10.246.15.92: 41/57 
2023-08-07 14:25:54 [DEBG] [scp.go:293] remote dst /var/lib/sealer/data/k8s-cluster/rootfs/scripts/NewBuild_V2.0_Monitor/zacopying files to 10.246.15.92: 42/57 
2023-08-07 14:25:55 [DEBG] [scp.go:293] remote dst /var/lib/sealer/data/k8s-cluster/rootfs/scripts/NewBuild_V2.0_Monitor/zacopying files to 10.246.15.92: 43/57 
2023-08-07 14:25:55 [DEBG] [scp.go:293] remote dst /var/lib/sealer/data/k8s-cluster/rootfs/scripts/NewBuild_V2.0_Monitor/zacopying files to 10.246.15.92: 44/57 
2023-08-07 14:25:55 [DEBG] [scp.go:293] remote dst /var/lib/sealer/data/k8s-cluster/rootfs/scripts/NewBuild_V2.0_Monitor/zacopying files to 10.246.15.92: 45/57 
2023-08-07 14:25:56 [DEBG] [scp.go:293] remote dst /var/lib/sealer/data/k8s-cluster/rootfs/scripts/NewBuild_V2.0_Monitor/zacopying files to 10.246.15.92: 46/57 
2023-08-07 14:25:56 [DEBG] [scp.go:293] remote dst /var/lib/sealer/data/k8s-cluster/rootfs/scripts/NewBuild_V2.0_Monitor/zacopying files to 10.246.15.92: 47/57 
2023-08-07 14:25:57 [DEBG] [scp.go:293] remote dst /var/lib/sealer/data/k8s-cluster/rootfs/scripts/clean.sh already exists copying files to 10.246.15.92: 48/57 
2023-08-07 14:25:57 [DEBG] [scp.go:293] remote dst /var/lib/sealer/data/k8s-cluster/rootfs/scripts/docker.sh already existscopying files to 10.246.15.92: 49/57 
2023-08-07 14:25:58 [DEBG] [scp.go:293] remote dst /var/lib/sealer/data/k8s-cluster/rootfs/scripts/func/log.sh already exiscopying files to 10.246.15.92: 50/57 
2023-08-07 14:26:00 [DEBG] [scp.go:293] remote dst /var/lib/sealer/data/k8s-cluster/rootfs/scripts/harbor/harbor-offline-incopying files to 10.246.15.92: 51/57 
2023-08-07 14:26:00 [DEBG] [scp.go:293] remote dst /var/lib/sealer/data/k8s-cluster/rootfs/scripts/init-kube.sh already exicopying files to 10.246.15.92: 52/57 
2023-08-07 14:26:01 [DEBG] [scp.go:293] remote dst /var/lib/sealer/data/k8s-cluster/rootfs/scripts/init-registry.sh alreadycopying files to 10.246.15.92: 53/57 
2023-08-07 14:26:01 [DEBG] [scp.go:293] remote dst /var/lib/sealer/data/k8s-cluster/rootfs/scripts/init.sh already exists acopying files to 10.246.15.92: 54/57 
2023-08-07 14:26:01 [DEBG] [scp.go:293] remote dst /var/lib/sealer/data/k8s-cluster/rootfs/scripts/init_node.sh already exicopying files to 10.246.15.92: 55/57 
2023-08-07 14:26:02 [DEBG] [scp.go:293] remote dst /var/lib/sealer/data/k8s-cluster/rootfs/scripts/kubelet-pre-start.sh alrcopying files to 10.246.15.92: 56/57 
2023-08-07 14:26:02 [DEBG] [scp.go:293] remote dst /var/lib/sealer/data/k8s-cluster/rootfs/scripts/post-install.sh already copying files to 10.246.15.92: 57/57 
2023-08-07 14:26:02 [DEBG] [scp.go:200] remote copy files src /var/lib/sealer/data/k8s-cluster/mount/statics to dst /var/licopying files to 10.246.15.92: 57/58 
2023-08-07 14:26:03 [DEBG] [scp.go:293] remote dst /var/lib/sealer/data/k8s-cluster/rootfs/statics/audit-policy.yml alreadycopying files to 10.246.15.92: 58/58 
export disk_drive=nvme1n1 disk_mount_path=/data1 cluster_name=europe harbor_address=10.246.15.168 time_zone=Europe/Berlin docker_dir=/data1/docker eth_name=eth0 && cd /var/lib/sealer/data/k8s-cluster/rootfs  && chmod +x scripts/* && cd scripts && bash init.sh /var/lib/docker sea.hub 5000
2023-08-07 14:28:10 [WARN] [local.go:130] failed to umount image , failed to unmount dir /var/lib/sealer/data/k8s-cluster/mount,err: retry action timeout: device or resource busy
Error: mount rootfs failed copy rootfs failed failed to copy sub files new sftp client failed ssh: handshake failed: ssh: unable to authenticate, attempted methods [none password], no supported methods remain

### What you expected to happen?

_No response_

### How to reproduce it (as minimally and precisely as possible)?

```YAML
Kubefile

FROM kubernetes:v1.19.8
COPY rootfs/bin/* bin
COPY rootfs/etc/* etc
COPY rootfs/scripts/* scripts
#COPY rootfs/manifests/* manifests
#COPY rootfs/plugins/* plugins

use plugin to init node,and then to excute ``sealer join --nodes xxxx,xxxx



### Anything else we need to know?

_No response_

### What is the version of Sealer you using?

{"gitVersion":"latest","gitCommit":"2a4d201","buildDate":"2022-04-04 01:10:39","goVersion":"go1.16.15","compiler":"gc","platform":"linux/amd64"}

### What is your OS environment?

Amazon Linux 2

### What is the Kernel version?

4.14.311-233.529.amzn2.x86_64

### Other environment you want to tell us?

_No response_
starnop commented 1 year ago

You can describe the specific changes before and after the operation

studycloudnative commented 1 year ago

You can describe the specific changes before and after the operation

what do you mean the specific changes before and after the operation?

in a few days ago, it's will to join node. but not work today. I have checked the password and ssh configuration multiple times.

studycloudnative commented 1 year ago

问题已经解决。 出现这种情况是因为有人在 k8s 层 delete 一个 node，但是在 sealer 的状态文件中依然存在这个节点的信息，导致在 scp的Fetch函数异常