fenlin88l commented 1 year ago

What is version of KubeKey has the issue?

./kk version kk version: &version.Info{Major:"3", Minor:"0", GitVersion:"v3.0.2", GitCommit:"1c395d22e75528d0a7d07c40e1af4830de265a23", GitTreeState:"clean", BuildDate:"2022-11-22T02:04:26Z", GoVersion:"go1.19.2", Compiler:"gc", Platform:"linux/amd64"}

What is your os environment?

centos7 内核3.10.0-1160.an7.x86_64

KubeKey config file

apiVersion: kubekey.kubesphere.io/v1alpha2
kind: Cluster
metadata:
  name: sample
spec:
  hosts:
  - {name: master, address: 100.x.x.x, internalAddress: 100.x.x.x, user: xxx, password: "xxxxxx"}
  - {name: node01, address: 100.x.x.x, internalAddress: 100.x.x.x, user: xxx, password: "xxxxxx"}
  - {name: node02, address: 100.x.x.x, internalAddress: 100.x.x.x, user: xxx, password: "xxxxxx"}
  roleGroups:
    etcd:
    - master
    control-plane: 
    - master
    worker:
    - node01
    - node02
  controlPlaneEndpoint:
    ## Internal loadbalancer for apiservers 
    # internalLoadbalancer: haproxy

    domain: lb.kubesphere.local
    address: ""
    port: 6443
  kubernetes:
    version: v1.25.3
    clusterName: cluster.local
    autoRenewCerts: true
    containerManager: containerd
  etcd:
    type: kubekey
  network:
    plugin: calico
    kubePodsCIDR: 10.233.64.0/18
    kubeServiceCIDR: 10.233.0.0/18
    ## multus support. https://github.com/k8snetworkplumbingwg/multus-cni
    multusCNI:
      enabled: false
  registry:
    #type: harbor
    #auths:
    #   host: 
    #    port: 8086
    #    username: admin
    #    password: Harbor12345
    #    certsPath: "/etc/docker/certs.d/dockerhub.kubekey.local"
    privateRegistry: ""
    namespaceOverride: ""
    registryMirrors: []
    insecureRegistries: []
  addons: []
  #- name: nfs-client
  #  namespace: kube-system
  #  sources:
  #    chart:
  #      name: nfs-client-provisioner
  #      repo: https://charts.kubesphere.io/main
  #      values:
  #      - nfs.server=
  #      - nfs.path=/nfs/data
  #      - storageClass.defaultClass=true

---
apiVersion: installer.kubesphere.io/v1alpha1
kind: ClusterConfiguration
metadata:
  name: ks-installer
  namespace: kubesphere-system
  labels:
    version: v3.3.0
spec:
  persistence:
    storageClass: ""
  authentication:
    jwtSecret: ""
  zone: ""
  local_registry: ""
  namespace_override: ""
  # dev_tag: ""
  etcd:
    monitoring: false
    endpointIps: localhost
    port: 2379
    tlsEnable: true
  common:
    core:
      console:
        enableMultiLogin: true
        port: 30880
        type: NodePort
    # apiserver:
    #  resources: {}
    # controllerManager:
    #  resources: {}
    redis:
      enabled: false
      volumeSize: 2Gi
    openldap:
      enabled: false
      volumeSize: 2Gi
    minio:
      volumeSize: 20Gi
    monitoring:
      # type: external
      endpoint: http://prometheus-operated.kubesphere-monitoring-system.svc:9090
      GPUMonitoring:
        enabled: false
    gpu:
      kinds:
      - resourceName: "nvidia.com/gpu"
        resourceType: "GPU"
        default: true
    es:
      # master:
      #   volumeSize: 4Gi
      #   replicas: 1
      #   resources: {}
      # data:
      #   volumeSize: 20Gi
      #   replicas: 1
      #   resources: {}
      logMaxAge: 7
      elkPrefix: logstash
      basicAuth:
        enabled: false
        username: ""
        password: ""
      externalElasticsearchHost: ""
      externalElasticsearchPort: ""
  alerting:
    enabled: false
    # thanosruler:
    #   replicas: 1
    #   resources: {}
  auditing:
    enabled: false
    # operator:
    #   resources: {}
    # webhook:
    #   resources: {}
  devops:
    enabled: false
    # resources: {}
    jenkinsMemoryLim: 2Gi
    jenkinsMemoryReq: 1500Mi
    jenkinsVolumeSize: 8Gi
    jenkinsJavaOpts_Xms: 1200m
    jenkinsJavaOpts_Xmx: 1600m
    jenkinsJavaOpts_MaxRAM: 2g
  events:
    enabled: false
    # operator:
    #   resources: {}
    # exporter:
    #   resources: {}
    # ruler:
    #   enabled: true
    #   replicas: 2
    #   resources: {}
  logging:
    enabled: false
    logsidecar:
      enabled: true
      replicas: 2
      # resources: {}
  metrics_server:
    enabled: false
  monitoring:
    storageClass: ""
    node_exporter:
      port: 9100
      # resources: {}
    # kube_rbac_proxy:
    #   resources: {}
    # kube_state_metrics:
    #   resources: {}
    # prometheus:
    #   replicas: 1
    #   volumeSize: 20Gi
    #   resources: {}
    #   operator:
    #     resources: {}
    # alertmanager:
    #   replicas: 1
    #   resources: {}
    # notification_manager:
    #   resources: {}
    #   operator:
    #     resources: {}
    #   proxy:
    #     resources: {}
    gpu:
      nvidia_dcgm_exporter:
        enabled: false
        # resources: {}
  multicluster:
    clusterRole: none
  network:
    networkpolicy:
      enabled: false
    ippool:
      type: none
    topology:
      type: none
  openpitrix:
    store:
      enabled: false
  servicemesh:
    enabled: false
    istio:
      components:
        ingressGateways:
        - name: istio-ingressgateway
          enabled: false
        cni:
          enabled: false
  edgeruntime:
    enabled: false
    kubeedge:
      enabled: false
      cloudCore:
        cloudHub:
          advertiseAddress:
            - ""
        service:
          cloudhubNodePort: "30000"
          cloudhubQuicNodePort: "30001"
          cloudhubHttpsNodePort: "30002"
          cloudstreamNodePort: "30003"
          tunnelNodePort: "30004"
        # resources: {}
        # hostNetWork: false
      iptables-manager:
        enabled: true
        mode: "external"
        # resources: {}
      # edgeService:
      #   resources: {}
  terminal:
    timeout: 600

A clear and concise description of what happend.

相关配置： apiVersion: installer.kubesphere.io/v1alpha1 kind: ClusterConfiguration metadata: name: ks-installer namespace: kubesphere-system labels: version: v3.3.0 spec: persistence: storageClass: "" authentication: jwtSecret: "" zone: "" local_registry: "" namespace_override: ""

dev_tag: ""

etcd: monitoring: false endpointIps: localhost port: 2379

Relevant log output


./kk create config --with-kubernetes v1.25.3  --with-kubesphere v3.3.0

./kk create cluster -f config-sample.yaml --container-manager containerd 
安装到这一步后  Please wait for the installation to complete:  >>---> 
查日志提示如下：
16:11:37 CST failed: [master]
error: Pipeline[CreateClusterPipeline] execute failed: Module[CheckResultModule] exec failed:failed: [master] execute task timeout, Timeout=2h

Error: failed to fetch group version resources batch/v1beta1: the server could not find the requested resource
2023/07/05 16:31:19 failed to fetch group version resources batch/v1beta1: the server could not find the requested resource

11:55:49 CST success: [master]
11:55:49 CST [CheckResultModule] Check ks-installer result
13:55:49 CST failed: [master]

登录KubeSphere页面
登录控制台提示：request to http://ks-apiserver/oauth/token failed, reason: connect ECONNREFUSED 10.233.8.38:80
![image](https://github.com/kubesphere/kubekey/assets/34697390/e700054e-25cc-46a4-a58b-0a8df7fd4704)

### Additional information

_No response_

redscholar commented 1 year ago

From your logs, it seems that the installation of KubeSphere has failed. KubeSphere is installed using ks-installer, and the ks-installer pod is in the kubesphere-system namespace. Can you provide the logs of the ks-installer pod?

fenlin88l commented 1 year ago

从您的日志来看，KubeSphere 安装失败。KubeSphere 使用进行安装ks-installer，并且ks-installerpod 位于kubesphere-system命名空间中。能提供一下pod的日志ks-installer吗？

[root@# kubectl 日志 -f pod/ks-installer-555b855cdc-gtcjx -n kubesphere-system 2023-07-06T17:13:59+08:00 INFO : shell-operator latest 2023-07-06T17:13:59+08:00 INFO : Use temporary dir: /tmp/shell-operator 2023-07-06T17:13:59+08:00 INFO : Initialize hooks manager ... 2023-07-06T17:13:59+08:00 INFO : Search and load hooks ... 2023-07-06T17:13:59+08:00 INFO : HTTP SERVER Listening on 0.0.0.0:9115 2023-07-06T17:13:59+08:00 INFO : Load hook config from '/hooks/kubesphere/installRunner.py' 2023-07-06T17:14:00+08:00 INFO : Load hook config from '/hooks/kubesphere/schedule.sh' 2023-07-06T17:14:00+08:00 INFO : Initializing schedule manager ... 2023-07-06T17:14:00+08:00 INFO : KUBE Init Kubernetes client 2023-07-06T17:14:00+08:00 INFO : KUBE-INIT Kubernetes client is configured successfully 2023-07-06T17:14:00+08:00 INFO : MAIN: run main loop 2023-07-06T17:14:00+08:00 INFO : MAIN: add onStartup tasks 2023-07-06T17:14:00+08:00 INFO : Running schedule manager ... 2023-07-06T17:14:00+08:00 INFO : QUEUE add all HookRun@OnStartup 2023-07-06T17:14:00+08:00 INFO : MSTOR Create new metric shell_operator_live_ticks 2023-07-06T17:14:00+08:00 INFO : MSTOR Create new metric shell_operator_tasks_queue_length 2023-07-06T17:14:00+08:00 INFO : GVR for kind 'ClusterConfiguration' is installer.kubesphere.io/v1alpha1, Resource=clusterconfigurations 2023-07-06T17:14:00+08:00 INFO : EVENT Kube event '768ceec5-268d-4eb5-b38c-8e6c1c52d020' 2023-07-06T17:14:00+08:00 INFO : QUEUE add TASK_HOOK_RUN@KUBE_EVENTS kubesphere/installRunner.py 2023-07-06T17:14:03+08:00 INFO : TASK_RUN HookRun@KUBE_EVENTS kubesphere/installRunner.py 2023-07-06T17:14:03+08:00 INFO : Running hook 'kubesphere/installRunner.py' binding 'KUBE_EVENTS' ... [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'

PLAY [localhost] ***

TASK [download : Generating images list] *** skipping: [localhost]

TASK [download : Synchronizing images] *****

TASK [kubesphere-defaults : KubeSphere | Setting images' namespace override] *** ok: [localhost]

TASK [kubesphere-defaults : KubeSphere | Configuring defaults] ***** ok: [localhost] => { "msg": "Check roles/kubesphere-defaults/defaults/main.yml" }

TASK [preinstall : KubeSphere | Stopping if Kubernetes version is nonsupport] *** ok: [localhost] => { "changed": false, "msg": "All assertions passed" }

TASK [preinstall : KubeSphere | Checking StorageClass] ***** changed: [localhost]

TASK [preinstall : KubeSphere | Stopping if StorageClass was not found] **** skipping: [localhost]

TASK [preinstall : KubeSphere | Checking default StorageClass] ***** changed: [localhost]

TASK [preinstall : KubeSphere | Stopping if default StorageClass was not found] *** ok: [localhost] => { "changed": false, "msg": "All assertions passed" }

TASK [preinstall : KubeSphere | Checking KubeSphere component] ***** changed: [localhost]

TASK [preinstall : KubeSphere | Getting KubeSphere component version] ** skipping: [localhost]

TASK [preinstall : KubeSphere | Getting KubeSphere component version] ** skipping: [localhost] => (item=ks-openldap) skipping: [localhost] => (item=ks-redis) skipping: [localhost] => (item=ks-minio) skipping: [localhost] => (item=ks-openpitrix) skipping: [localhost] => (item=elasticsearch-logging) skipping: [localhost] => (item=elasticsearch-logging-curator) skipping: [localhost] => (item=istio) skipping: [localhost] => (item=istio-init) skipping: [localhost] => (item=jaeger-operator) skipping: [localhost] => (item=ks-jenkins) skipping: [localhost] => (item=ks-sonarqube) skipping: [localhost] => (item=logging-fluentbit-operator) skipping: [localhost] => (item=uc) skipping: [localhost] => (item=metrics-server)

PLAY RECAP ***** localhost : ok=7 changed=3 unreachable=0 failed=0 skipped=5 rescued=0 ignored=0 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'

PLAY [localhost] ***

TASK [download : Generating images list] *** skipping: [localhost]

TASK [download : Synchronizing images] *****

TASK [kubesphere-defaults : KubeSphere | Setting images' namespace override] *** ok: [localhost]

TASK [kubesphere-defaults : KubeSphere | Configuring defaults] ***** ok: [localhost] => { "msg": "Check roles/kubesphere-defaults/defaults/main.yml" }

TASK [Metrics-Server | Getting metrics-server installation files] ** skipping: [localhost]

TASK [metrics-server : Metrics-Server | Creating manifests] **** skipping: [localhost] => (item={'file': 'metrics-server.yaml'})

TASK [metrics-server : Metrics-Server | Checking Metrics-Server] *** skipping: [localhost]

TASK [Metrics-Server | Uninstalling old metrics-server] **** skipping: [localhost]

TASK [Metrics-Server | Installing new metrics-server] ** skipping: [localhost]

TASK [metrics-server : Metrics-Server | Waitting for metrics.k8s.io ready] ***** skipping: [localhost]

TASK [Metrics-Server | Importing metrics-server status] **** skipping: [localhost]

PLAY RECAP ***** localhost : ok=2 changed=0 unreachable=0 failed=0 skipped=9 rescued=0 ignored=0 [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'

PLAY [localhost] ***

TASK [download : Generating images list] *** skipping: [localhost]

TASK [download : Synchronizing images] *****

TASK [kubesphere-defaults : KubeSphere | Setting images' namespace override] *** ok: [localhost]

TASK [kubesphere-defaults : KubeSphere | Configuring defaults] ***** ok: [localhost] => { "msg": "Check roles/kubesphere-defaults/defaults/main.yml" }

TASK [common : KubeSphere | Checking kube-node-lease namespace] **** changed: [localhost]

TASK [common : KubeSphere | Getting system namespaces] ***** ok: [localhost]

TASK [common : set_fact] *** ok: [localhost]

TASK [common : debug] ** ok: [localhost] => { "msg": [ "kubesphere-system", "kubesphere-controls-system", "kubesphere-monitoring-system", "kubesphere-monitoring-federated", "kube-node-lease" ] }

TASK [common : KubeSphere | Creating KubeSphere namespace] ***** changed: [localhost] => (item=kubesphere-system) changed: [localhost] => (item=kubesphere-controls-system) changed: [localhost] => (item=kubesphere-monitoring-system) changed: [localhost] => (item=kubesphere-monitoring-federated) changed: [localhost] => (item=kube-node-lease)

TASK [common : KubeSphere | Labeling system-workspace] ***** changed: [localhost] => (item=default) changed: [localhost] => (item=kube-public) changed: [localhost] => (item=kube-system) changed: [localhost] => (item=kubesphere-system) changed: [localhost] => (item=kubesphere-controls-system) changed: [localhost] => (item=kubesphere-monitoring-system) changed: [localhost] => (item=kubesphere-monitoring-federated) changed: [localhost] => (item=kube-node-lease)

TASK [common : KubeSphere | Labeling namespace for network policy] ***** changed: [localhost]

TASK [common : KubeSphere | Getting Kubernetes master num] ***** changed: [localhost]

TASK [common : KubeSphere | Setting master num] **** ok: [localhost]

TASK [KubeSphere | Getting common component installation files] **** changed: [localhost] => (item=common)

TASK [common : KubeSphere | Checking Kubernetes version] *** changed: [localhost]

TASK [KubeSphere | Getting common component installation files] **** changed: [localhost] => (item=snapshot-controller)

TASK [common : KubeSphere | Creating snapshot controller values] *** changed: [localhost] => (item={'name': 'custom-values-snapshot-controller', 'file': 'custom-values-snapshot-controller.yaml'})

TASK [common : KubeSphere | Updating snapshot crd] ***** changed: [localhost]

TASK [common : KubeSphere | Deploying snapshot controller] ***** changed: [localhost]

TASK [KubeSphere | Checking openpitrix common component] *** changed: [localhost]

TASK [common : include_tasks] ** skipping: [localhost] => (item={'op': 'openpitrix-db', 'ks': 'mysql-pvc'}) skipping: [localhost] => (item={'op': 'openpitrix-etcd', 'ks': 'etcd-pvc'})

TASK [common : Getting PersistentVolumeName (mysql)] *** skipping: [localhost]

TASK [common : Getting PersistentVolumeSize (mysql)] *** skipping: [localhost]

TASK [common : Setting PersistentVolumeName (mysql)] *** skipping: [localhost]

TASK [common : Setting PersistentVolumeSize (mysql)] *** skipping: [localhost]

TASK [common : Getting PersistentVolumeName (etcd)] **** skipping: [localhost]

TASK [common : Getting PersistentVolumeSize (etcd)] **** skipping: [localhost]

TASK [common : Setting PersistentVolumeName (etcd)] **** skipping: [localhost]

TASK [common : Setting PersistentVolumeSize (etcd)] **** skipping: [localhost]

TASK [common : KubeSphere | Checking mysql PersistentVolumeClaim] ** changed: [localhost]

TASK [common : KubeSphere | Setting mysql db pv size] ** skipping: [localhost]

TASK [common : KubeSphere | Checking redis PersistentVolumeClaim] ** changed: [localhost]

TASK [common : KubeSphere | Setting redis db pv size] ** skipping: [localhost]

TASK [common : KubeSphere | Checking minio PersistentVolumeClaim] ** changed: [localhost]

TASK [common : KubeSphere | Setting minio pv size] ***** skipping: [localhost]

TASK [common : KubeSphere | Checking openldap PersistentVolumeClaim] *** changed: [localhost]

TASK [common : KubeSphere | Setting openldap pv size] ** skipping: [localhost]

TASK [common : KubeSphere | Checking etcd db PersistentVolumeClaim] **** changed: [localhost]

TASK [common : KubeSphere | Setting etcd pv size] ** skipping: [localhost]

TASK [common : KubeSphere | Checking redis ha PersistentVolumeClaim] *** changed: [localhost]

TASK [common : KubeSphere | Setting redis ha pv size] ** skipping: [localhost]

TASK [common : KubeSphere | Checking es-master PersistentVolumeClaim] ** changed: [localhost]

TASK [common : KubeSphere | Setting es master pv size] ***** skipping: [localhost]

TASK [common : KubeSphere | Checking es data PersistentVolumeClaim] **** changed: [localhost]

TASK [common : KubeSphere | Setting es data pv size] *** skipping: [localhost]

TASK [KubeSphere | Creating common component manifests] **** changed: [localhost] => (item={'path': 'redis', 'file': 'redis.yaml'})

TASK [common : KubeSphere | Deploying etcd and mysql] ** skipping: [localhost] => (item=etcd.yaml) skipping: [localhost] => (item=mysql.yaml)

TASK [common : KubeSphere | Getting minio installation files] ** skipping: [localhost] => (item=minio-ha)

TASK [common : KubeSphere | Creating manifests] **** skipping: [localhost] => (item={'name': 'custom-values-minio', 'file': 'custom-values-minio.yaml'})

TASK [common : KubeSphere | Checking minio] **** skipping: [localhost]

TASK [common : KubeSphere | Deploying minio] *** skipping: [localhost]

TASK [common : debug] ** skipping: [localhost]

TASK [common : fail] *** skipping: [localhost]

TASK [common : KubeSphere | Importing minio status] **** skipping: [localhost]

TASK [common : KubeSphere | Generet Random password] *** skipping: [localhost]

TASK [common : KubeSphere | Creating Redis Password Secret] **** skipping: [localhost]

TASK [common : KubeSphere | Getting redis installation files] ** skipping: [localhost] => (item=redis-ha)

TASK [common : KubeSphere | Creating manifests] **** skipping: [localhost] => (item={'name': 'custom-values-redis', 'file': 'custom-values-redis.yaml'})

TASK [common : KubeSphere | Checking old redis status] ***** skipping: [localhost]

TASK [common : KubeSphere | Deleting and backup old redis svc] ***** skipping: [localhost]

TASK [common : KubeSphere | Deploying redis] *** skipping: [localhost]

TASK [common : KubeSphere | Deploying redis] *** skipping: [localhost] => (item=redis.yaml)

TASK [common : KubeSphere | Importing redis status] **** skipping: [localhost]

TASK [common : KubeSphere | Getting openldap installation files] *** skipping: [localhost] => (item=openldap-ha)

TASK [common : KubeSphere | Creating manifests] **** skipping: [localhost] => (item={'name': 'custom-values-openldap', 'file': 'custom-values-openldap.yaml'})

TASK [common : KubeSphere | Checking old openldap status] ** skipping: [localhost]

TASK [common : KubeSphere | Shutdown ks-account] *** skipping: [localhost]

TASK [common : KubeSphere | Deleting and backup old openldap svc] ** skipping: [localhost]

TASK [common : KubeSphere | Checking openldap] ***** skipping: [localhost]

TASK [common : KubeSphere | Deploying openldap] **** skipping: [localhost]

TASK [common : KubeSphere | Loading old openldap data] ***** skipping: [localhost]

TASK [common : KubeSphere | Checking openldap-ha status] *** skipping: [localhost]

TASK [common : KubeSphere | Getting openldap-ha pod list] ** skipping: [localhost]

TASK [common : KubeSphere | Getting old openldap data] ***** skipping: [localhost]

TASK [common : KubeSphere | Migrating openldap data] *** skipping: [localhost]

TASK [common : KubeSphere | Disabling old openldap] **** skipping: [localhost]

TASK [common : KubeSphere | Restarting openldap] *** skipping: [localhost]

TASK [common : KubeSphere | Restarting ks-account] ***** skipping: [localhost]

TASK [common : KubeSphere | Importing openldap status] ***** skipping: [localhost]

TASK [common : KubeSphere | Checking KubeSphere Config is Exists] ** changed: [localhost]