Found multiple CRI endpoints on the host

willzhang commented 1 year ago

Your current KubeKey version

v3.0.1

Describe this feature

support config criSocket

:22:44 CST [KubernetesStatusModule] Get kubernetes cluster status
00:22:44 CST stdout: [deploy]
v1.25.3
00:22:44 CST stdout: [deploy]
deploy   v1.25.3   [map[address:172.17.10.195 type:InternalIP] map[address:deploy type:Hostname]]
00:22:44 CST stdout: [deploy]
Found multiple CRI endpoints on the host. Please define which one do you wish to use by setting the 'criSocket' field in the kubeadm configuration file: unix:///var/run/containerd/containerd.sock, unix:///var/run/crio/crio.sock
To see the stack trace of this error execute with --v=5 or higher
00:22:44 CST message: [deploy]
Failed to upload kubeadm certs: Failed to exec command: sudo -E /bin/bash -c "/usr/local/bin/kubeadm init phase upload-certs --upload-certs" 
Found multiple CRI endpoints on the host. Please define which one do you wish to use by setting the 'criSocket' field in the kubeadm configuration file: unix:///var/run/containerd/containerd.sock, unix:///var/run/crio/crio.sock
To see the stack trace of this error execute with --v=5 or higher: Process exited with status 1
00:22:44 CST retry: [deploy]
00:22:49 CST stdout: [deploy]
v1.25.3
00:22:49 CST stdout: [deploy]
deploy   v1.25.3   [map[address:172.17.10.195 type:InternalIP] map[address:deploy type:Hostname]]
00:22:49 CST stdout: [deploy]
Found multiple CRI endpoints on the host. Please define which one do you wish to use by setting the 'criSocket' field in the kubeadm configuration file: unix:///var/run/containerd/containerd.sock, unix:///var/run/crio/crio.sock
To see the stack trace of this error execute with --v=5 or higher
00:22:49 CST message: [deploy]
Failed to upload kubeadm certs: Failed to exec command: sudo -E /bin/bash -c "/usr/local/bin/kubeadm init phase upload-certs --upload-certs" 
Found multiple CRI endpoints on the host. Please define which one do you wish to use by setting the 'criSocket' field in the kubeadm configuration file: unix:///var/run/containerd/containerd.sock, unix:///var/run/crio/crio.sock
To see the stack trace of this error execute with --v=5 or higher: Process exited with status 1
00:22:49 CST retry: [deploy]
00:22:54 CST stdout: [deploy]
v1.25.3
00:22:55 CST stdout: [deploy]
deploy   v1.25.3   [map[address:172.17.10.195 type:InternalIP] map[address:deploy type:Hostname]]
00:22:55 CST stdout: [deploy]
Found multiple CRI endpoints on the host. Please define which one do you wish to use by setting the 'criSocket' field in the kubeadm configuration file: unix:///var/run/containerd/containerd.sock, unix:///var/run/crio/crio.sock
To see the stack trace of this error execute with --v=5 or higher
00:22:55 CST message: [deploy]
Failed to upload kubeadm certs: Failed to exec command: sudo -E /bin/bash -c "/usr/local/bin/kubeadm init phase upload-certs --upload-certs" 
Found multiple CRI endpoints on the host. Please define which one do you wish to use by setting the 'criSocket' field in the kubeadm configuration file: unix:///var/run/containerd/containerd.sock, unix:///var/run/crio/crio.sock
To see the stack trace of this error execute with --v=5 or higher: Process exited with status 1
00:22:55 CST failed: [deploy]
error: Pipeline[CreateClusterPipeline] execute failed: Module[KubernetesStatusModule] exec failed: 
failed: [deploy] [GetClusterStatus] exec failed after 3 retires: Failed to upload kubeadm certs: Failed to exec command: sudo -E /bin/bash -c "/usr/local/bin/kubeadm init phase upload-certs --upload-certs" 
Found multiple CRI endpoints on the host. Please define which one do you wish to use by setting the 'criSocket' field in the kubeadm configuration file: unix:///var/run/containerd/containerd.sock, unix:///var/run/crio/crio.sock
To see the stack trace of this error execute with --v=5 or higher: Process exited with status 1
root@deploy:/data/kubernetes/k8s_v1.25.3#

Describe the solution you'd like

config-sample.yaml , this not effect

  kubernetes:
    version: v1.25.3
    clusterName: cluster.local
    autoRenewCerts: true
    containerManager: crio
    InitConfiguration:
      nodeRegistration:
        criSocket: /var/run/crio/crio.sock

Additional information

No response

24sama commented 1 year ago

Can you paste the file /etc/kubernetes/kubeadm-config.yaml?

willzhang commented 1 year ago

root@ubuntu:/data/kubernetes# cat /etc/kubernetes/kubeadm-config.yaml
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
etcd:
  external:
    endpoints:
    - https://172.17.10.197:2379
    caFile: /etc/ssl/etcd/ssl/ca.pem
    certFile: /etc/ssl/etcd/ssl/node-ubuntu.pem
    keyFile: /etc/ssl/etcd/ssl/node-ubuntu-key.pem
dns:
  type: CoreDNS
  imageRepository: repository.cucc.cn/kubesphereio
  imageTag: 1.9.3
imageRepository: repository.cucc.cn/kubesphereio
kubernetesVersion: v1.25.3
certificatesDir: /etc/kubernetes/pki
clusterName: cluster.local
controlPlaneEndpoint: 172.17.10.197:6443
networking:
  dnsDomain: cluster.local
  podSubnet: 10.233.64.0/18
  serviceSubnet: 10.233.0.0/18
apiServer:
  extraArgs:
    audit-log-maxage: "30"
    audit-log-maxbackup: "10"
    audit-log-maxsize: "100"
    bind-address: 0.0.0.0
    feature-gates: RotateKubeletServerCertificate=true,ExpandCSIVolumes=true,CSIStorageCapacity=true
  certSANs:
    - kubernetes
    - kubernetes.default
    - kubernetes.default.svc
    - kubernetes.default.svc.cluster.local
    - localhost
    - 127.0.0.1
    - 172.17.10.197
    - 172.17.10.197
    - ubuntu
    - ubuntu.cluster.local
    - 10.233.0.1
controllerManager:
  extraArgs:
    node-cidr-mask-size: "24"
    bind-address: 0.0.0.0
    cluster-signing-duration: 87600h
    feature-gates: RotateKubeletServerCertificate=true,ExpandCSIVolumes=true,CSIStorageCapacity=true
  extraVolumes:
  - name: host-time
    hostPath: /etc/localtime
    mountPath: /etc/localtime
    readOnly: true
scheduler:
  extraArgs:
    bind-address: 0.0.0.0
    feature-gates: RotateKubeletServerCertificate=true,ExpandCSIVolumes=true,CSIStorageCapacity=true

---
apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 172.17.10.197
  bindPort: 6443
nodeRegistration:
  criSocket: unix:///var/run/crio/crio.sock
  kubeletExtraArgs:
    cgroup-driver: systemd
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
clusterCIDR: 10.233.64.0/18
iptables:
  masqueradeAll: false
  masqueradeBit: 14
  minSyncPeriod: 0s
  syncPeriod: 30s
mode: ipvs
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
clusterDNS:
- 169.254.25.10
clusterDomain: cluster.local
containerLogMaxFiles: 3
containerLogMaxSize: 5Mi
evictionHard:
  memory.available: 5%
  pid.available: 10%
evictionMaxPodGracePeriod: 120
evictionPressureTransitionPeriod: 30s
evictionSoft:
  memory.available: 10%
evictionSoftGracePeriod:
  memory.available: 2m
featureGates:
  CSIStorageCapacity: true
  ExpandCSIVolumes: true
  RotateKubeletServerCertificate: true
kubeReserved:
  cpu: 200m
  memory: 250Mi
maxPods: 110
podPidsLimit: 10000
rotateCertificates: true
systemReserved:
  cpu: 200m
  memory: 250Mi

willzhang commented 1 year ago

it not use /etc/kubernetes/kubeadm-config.yaml ?

kubeadm init phase upload-certs --upload-certs --config /etc/kubernetes/kubeadm-config.yaml

willzhang commented 1 year ago

InitConfiguration: nodeRegistration: criSocket: /var/run/crio/crio.sock

not support InitConfiguration field ？ https://github.com/kubesphere/kubekey/blob/ec903fe13dfed73ffd3f72f4beec3123675ce4d0/cmd/kk/apis/kubekey/v1alpha2/kubernetes_types.go#L32

should i use ?

  kubernetes:
    version: v1.25.3
    clusterName: cluster.local
    autoRenewCerts: true
    containerManager: crio
    ContainerRuntimeEndpoint="/var/run/crio/crio.sock"

xiaods commented 1 year ago

PR welcome

kubesphere / kubekey