kubesphere在线kubernetes部署失败

[AkideLiu]

1.system：

cat /etc/redhat-release

CentOS Linux release 7.7.1908 (Core)

2.kubernetes version：

kubectl version | grep Server

Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T11:05:50Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}

3.helm version：

helm version

Client: &version.Version{SemVer:"v2.15.1", GitCommit:"cf1de4f8ba70eded310918a8af3a96bfe8e7683b", GitTreeState:"clean"}

Server: &version.Version{SemVer:"v2.15.1", GitCommit:"cf1de4f8ba70eded310918a8af3a96bfe8e7683b", GitTreeState:"clean"}

4.storageClass：

kubectl get sc

NAME PROVISIONER AGE

thin-disk kubernetes.io/vsphere-volume 10h

5.error logs：

kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l job-name=kubesphere-installer -o jsonpath='{.items[0].metadata.name}') -f

[WARNING]: provided hosts list is empty, only localhost is available. Note

that the implicit localhost does not match 'all'

PLAY [localhost] ***

TASK [download : include_tasks] ****

skipping: [localhost]

TASK [download : Download items] ***

skipping: [localhost]

TASK [download : Sync container] ***

skipping: [localhost]

TASK [kubesphere-defaults : Configure defaults] ****

ok: [localhost] => {

"msg": "Check roles/kubespray-defaults/defaults/main.yml" 

}

TASK [ks-devops/sonarqube : ks-devops | Creating ks-jenkins webhook(external)] ***

skipping: [localhost]

TASK [ks-devops/sonarqube : ks-devops | Getting sonarqube installation files] ***

changed: [localhost]

TASK [ks-devops/sonarqube : Sonarqube | Creating manifests] ****

changed: [localhost] => (item={u'name': u'custom-values-sonarqube', u'file': u'custom-values-sonarqube.yaml'})

TASK [ks-devops/sonarqube : Sonarqube | Deploy Sonarqube] **

fatal: [localhost]: FAILED! => {"changed": true, "cmd": "/usr/local/bin/helm upgrade --install ks-sonarqube /etc/kubesphere/sonarqube/sonarqube-0.13.5.tgz -f /etc/kubesphere/sonarqube/custom-values-sonarqube.yaml --namespace kubesphere-devops-system\n", "delta": "0:00:01.665860", "end": "2019-10-25 18:20:48.887010", "msg": "non-zero return code", "rc": 1, "start": "2019-10-25 18:20:47.221150", "stderr": "Error: UPGRADE FAILED: configmaps is forbidden: User \"system:serviceaccount:kube-system:default\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kube-system\"", "stderr_lines": ["Error: UPGRADE FAILED: configmaps is forbidden: User \"system:serviceaccount:kube-system:default\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kube-system\""], "stdout": "", "stdout_lines": []}

PLAY RECAP *****

localhost : ok=3 changed=2 unreachable=0 failed=1 skipped=4 rescued=0 ignored=0

[AkideLiu]

Helm issus resolution:

https://medium.com/@pczarkowski/easily-install-uninstall-helm-on-rbac-kubernetes-8c3c0e22d0d7

https://github.com/helm/helm/issues/3130#issuecomment-345506262

[AkideLiu]

第二次出现问题： TASK [metrics-server : Metrics-Server | Getting metrics-server installation files] ***

changed: [localhost]

TASK [metrics-server : Metrics-Server | Creating manifests] ****

changed: [localhost] => (item={u'type': u'config', u'name': u'values', u'file': u'values.yaml'})

TASK [metrics-server : Metrics-Server | Installing metrics-server] *****

fatal: [localhost]: FAILED! => {"changed": true, "cmd": "/usr/local/bin/helm upgrade --install metrics-server/etc/kubesphere/metrics-server --namespace kube-system\n", "delta": "0:00:00.975578", "end": "2019-10-25 19:19:29.722614", "msg": "non-zero return code", "rc": 1, "start": "2019-10-25 19:19:28.747036", "stderr": "Error: Could not get apiVersions from Kubernetes: unable to retrieve the complete list of server APIs: authentication.istio.io/v1alpha1: the server could not find the requested resource, config.istio.io/v1alpha2: the server could not find the requested resource, networking.istio.io/v1alpha3: the server could not find the requested resource, rbac.istio.io/v1alpha1: the server could not find the requested resource", "stderr_lines": ["Error: Could not get apiVersions from Kubernetes: unable to retrieve the complete list of server APIs: authentication.istio.io/v1alpha1: the server could not find the requested resource, config.istio.io/v1alpha2: the server couldnot find the requested resource, networking.istio.io/v1alpha3: the server could not find the requested resource, rbac.istio.io/v1alpha1: the server could not find the requested resource"], "stdout": "Release \"metrics-server\" does not exist. Installing it now.", "stdout_lines": ["Release \"metrics-server\" does not exist. Installing it now."]}

PLAY RECAP *****

localhost : ok=35 changed=27 unreachable=0 failed=1 skipped=7 rescued=0 ignored=0

[AkideLiu]

第三次出现问题： FAILED - RETRYING: ks-monitor | Installing prometheus-operator (5 retries left).

FAILED - RETRYING: ks-monitor | Installing prometheus-operator (4 retries left).

FAILED - RETRYING: ks-monitor | Installing prometheus-operator (3 retries left).

FAILED - RETRYING: ks-monitor | Installing prometheus-operator (2 retries left).

FAILED - RETRYING: ks-monitor | Installing prometheus-operator (1 retries left).

failed: [localhost] (item=sources) => {"ansible_loop_var": "item", "attempts": 5, "changed": true, "cmd": "/usr/local/bin/kubectl apply -f /etc/kubesphere/prometheus/sources", "delta": "0:00:01.786886", "end": "2019-10-25 19:22:49.231470", "failed_when_result": true, "item": "sources", "msg": "non-zero return code", "rc": 1, "start": "2019-10-25 19:22:47.444584", "stderr": "Error from server (Invalid): error when creating \"/etc/kubesphere/prometheus/sources/prometheus-prometheus.yaml\": Prometheus.monitoring.coreos.com \"k8s\" is invalid: []:Invalid value: map[string]interface {}{\"apiVersion\":\"monitoring.coreos.com/v1\", \"kind\":\"Prometheus\", \"metadata\":map[string]interface {}{\"annotations\":map[string]interface {}{\"kubectl.kubernetes.io/last-applied-configuration\":\"{\\"apiVersion\\":\\"monitoring.coreos.com/v1\\",\\"kind\\":\\"Prometheus\\",\\"metadata\\":{\\"annotations\\":{},\\"labels\\":{\\"prometheus\\":\\"k8s\\"},\\"name\\":\\"k8s\\",\\"namespace\\":\\"kubesphere-monitoring-system\\"},\\"spec\\":{\\"affinity\\":{\\"podAntiAffinity\\":{\\"preferredDuringSchedulingIgnoredDuringExecution\\":[{\\"podAffinityTerm\\":{\\"labelSelector\\":{\\"matchExpressions\\":[{\\"key\\":\\"prometheus\\",\\"operator\\":\\"In\\",\\"values\\":[\\"k8s\\"]}]},\\"namespaces\\":[\\"kubesphere-monitoring-system\\"],\\"topologyKey\\":\\"kubernetes.io/hostname\\"},\\"weight\\":100}]}},\\"baseImage\\":\\"dockerhub.qingcloud.com/prometheus/prometheus\\",\\"imagePullSecrets\\":[{\\"name\\":\\"qingcloud\\"}],\\"nodeSelector\\":{\\"beta.kubernetes.io/os\\":\\"linux\\"},\\"query\\":{\\"maxConcurrency\\":1000},\\"replicas\\":2,\\"resources\\":{\\"limits\\":{\\"cpu\\":\\"2.5\\",\\"memory\\":\\"2Gi\\"},\\"requests\\":{\\"cpu\\":\\"80m\\",\\"memory\\":\\"400Mi\\"}},\\"retention\\":\\"7d\\",\\"ruleSelector\\":{\\"matchLabels\\":{\\"prometheus\\":\\"k8s\\",\\"role\\":\\"alert-rules\\"}},\\"scrapeInterval\\":\\"1m\\",\\"securityContext\\":{\\"fsGroup\\":0,\\"runAsNonRoot\\":false,\\"runAsUser\\":0},\\"serviceAccountName\\":\\"prometheus-k8s\\",\\"serviceMonitorNamespaceSelector\\":{},\\"serviceMonitorSelector\\":{\\"matchExpressions\\":[{\\"key\\":\\"k8s-app\\",\\"operator\\":\\"In\\",\\"values\\":[\\"kube-state-metrics\\",\\"node-exporter\\",\\"kubelet\\",\\"prometheus-system\\"]}]},\\"storage\\":{\\"volumeClaimTemplate\\":{\\"spec\\":{\\"resources\\":{\\"requests\\":{\\"storage\\":\\"20Gi\\"}},\\"storageClassName\\":null}}},\\"tolerations\\":[{\\"effect\\":\\"NoSchedule\\",\\"key\\":\\"dedicated\\",\\"operator\\":\\"Equal\\",\\"value\\":\\"monitoring\\"}],\\"version\\":\\"v2.5.0\\"}}\n\"}, \"creationTimestamp\":\"2019-10-25T19:22:48Z\", \"generation\":1, \"labels\":map[string]interface {}{\"prometheus\":\"k8s\"}, \"name\":\"k8s\", \"namespace\":\"kubesphere-monitoring-system\", \"uid\":\"f41b2c48-f370-46be-b923-e2478db45e10\"}, \"spec\":map[string]interface {}{\"affinity\":map[string]interface {}{\"podAntiAffinity\":map[string]interface {}{\"preferredDuringSchedulingIgnoredDuringExecution\":[]interface {}{map[string]interface {}{\"podAffinityTerm\":map[string]interface {}{\"labelSelector\":map[string]interface {}{\"matchExpressions\":[]interface {}{map[string]interface {}{\"key\":\"prometheus\", \"operator\":\"In\", \"values\":[]interface {}{\"k8s\"}}}}, \"namespaces\":[]interface {}{\"kubesphere-monitoring-system\"}, \"topologyKey\":\"kubernetes.io/hostname\"}, \"weight\":100}}}}, \"baseImage\":\"dockerhub.qingcloud.com/prometheus/prometheus\", \"imagePullSecrets\":[]interface {}{map[string]interface {}{\"name\":\"qingcloud\"}}, \"nodeSelector\":map[string]interface {}{\"beta.kubernetes.io/os\":\"linux\"}, \"query\":map[string]interface {}{\"maxConcurrency\":1000}, \"replicas\":2, \"resources\":map[string]interface {}{\"limits\":map[string]interface {}{\"cpu\":\"2.5\", \"memory\":\"2Gi\"}, \"requests\":map[string]interface {}{\"cpu\":\"80m\", \"memory\":\"400Mi\"}}, \"retention\":\"7d\", \"ruleSelector\":map[string]interface{}{\"matchLabels\":map[string]interface {}{\"prometheus\":\"k8s\", \"role\":\"alert-rules\"}}, \"scrapeInterval\":\"1m\", \"securityContext\":map[string]interface {}{\"fsGroup\":0, \"runAsNonRoot\":false, \"runAsUser\":0}, \"serviceAccountName\":\"prometheus-k8s\", \"serviceMonitorNamespaceSelector\":map[string]interface {}{}, \"serviceMonitorSelector\":map[string]interface {}{\"matchExpressions\":[]interface {}{map[string]interface {}{\"key\":\"k8s-app\", \"operator\":\"In\", \"values\":[]interface {}{\"kube-state-metrics\", \"node-exporter\", \"kubelet\", \"prometheus-system\"}}}}, \"storage\":map[string]interface {}{\"volumeClaimTemplate\":map[string]interface {}{\"spec\":map[string]interface {}{\"resources\":map[string]interface {}{\"requests\":map[string]interface {}{\"storage\":\"20Gi\"}}, \"storageClassName\":interface {}(nil)}}}, \"tolerations\":[]interface {}{map[string]interface {}{\"effect\":\"NoSchedule\", \"key\":\"dedicated\", \"operator\":\"Equal\", \"value\":\"monitoring\"}}, \"version\":\"v2.5.0\"}}: validation failure list:\nspec.storage.volumeClaimTemplate.spec.storageClassName in body must be of type string: \"null\"\nError from server (Invalid): error when creating \"/etc/kubesphere/prometheus/sources/prometheus-prometheusSystem.yaml\": Prometheus.monitoring.coreos.com \"k8s-system\" is invalid: []: Invalid value: map[string]interface {}{\"apiVersion\":\"monitoring.coreos.com/v1\", \"kind\":\"Prometheus\", \"metadata\":map[string]interface {}{\"annotations\":map[string]interface {}{\"kubectl.kubernetes.io/last-applied-configuration\":\"{\\"apiVersion\\":\\"monitoring.coreos.com/v1\\",\\"kind\\":\\"Prometheus\\",\\"metadata\\":{\\"annotations\\":{},\\"labels\\":{\\"prometheus\\":\\"k8s-system\\"},\\"name\\":\\"k8s-system\\",\\"namespace\\":\\"kubesphere-monitoring-system\\"},\\"spec\\":{\\"additionalScrapeConfigs\\":{\\"key\\":\\"prometheus-additional.yaml\\",\\"name\\":\\"additional-scrape-configs\\"},\\"affinity\\":{\\"podAntiAffinity\\":{\\"preferredDuringSchedulingIgnoredDuringExecution\\":[{\\"podAffinityTerm\\":{\\"labelSelector\\":{\\"matchExpressions\\":[{\\"key\\":\\"prometheus\\",\\"operator\\":\\"In\\",\\"values\\":[\\"k8s-system\\"]}]},\\"namespaces\\":[\\"kubesphere-monitoring-system\\"],\\"topologyKey\\":\\"kubernetes.io/hostname\\"},\\"weight\\":100}]}},\\"baseImage\\":\\"dockerhub.qingcloud.com/prometheus/prometheus\\",\\"imagePullSecrets\\":[{\\"name\\":\\"qingcloud\\"}],\\"nodeSelector\\":{\\"beta.kubernetes.io/os\\":\\"linux\\"},\\"query\\":{\\"maxConcurrency\\":1000},\\"replicas\\":2,\\"resources\\":{\\"limits\\":{\\"cpu\\":\\"2.5\\",\\"memory\\":\\"2Gi\\"},\\"requests\\":{\\"cpu\\":\\"60m\\",\\"memory\\":\\"400Mi\\"}},\\"retention\\":\\"7d\\",\\"ruleSelector\\":{\\"matchLabels\\":{\\"prometheus\\":\\"k8s\\",\\"role\\":\\"alert-rules\\"}},\\"scrapeInterval\\":\\"1m\\",\\"secrets\\":[\\"kube-etcd-client-certs\\"],\\"securityContext\\":{\\"fsGroup\\":0,\\"runAsNonRoot\\":false,\\"runAsUser\\":0},\\"serviceAccountName\\":\\"prometheus-k8s\\",\\"serviceMonitorNamespaceSelector\\":{},\\"serviceMonitorSelector\\":{\\"matchExpressions\\":[{\\"key\\":\\"k8s-app\\",\\"operator\\":\\"In\\",\\"values\\":[\\"etcd\\",\\"coredns\\",\\"apiserver\\",\\"prometheus\\",\\"kube-scheduler\\",\\"kube-controller-manager\\"]}]},\\"storage\\":{\\"volumeClaimTemplate\\":{\\"spec\\":{\\"resources\\":{\\"requests\\":{\\"storage\\":\\"20Gi\\"}},\\"storageClassName\\":null}}},\\"tolerations\\":[{\\"effect\\":\\"NoSchedule\\",\\"key\\":\\"dedicated\\",\\"operator\\":\\"Equal\\",\\"value\\":\\"monitoring\\"}],\\"version\\":\\"v2.5.0\\"}}\n\"}, \"creationTimestamp\":\"2019-10-25T19:22:48Z\", \"generation\":1, \"labels\":map[string]interface {}{\"prometheus\":\"k8s-system\"}, \"name\":\"k8s-system\", \"namespace\":\"kubesphere-monitoring-system\", \"uid\":\"868782a4-2536-4662-967d-84aa8139d21d\"}, \"spec\":map[string]interface {}{\"additionalScrapeConfigs\":map[string]interface {}{\"key\":\"prometheus-additional.yaml\", \"name\":\"additional-scrape-configs\"}, \"affinity\":map[string]interface {}{\"podAntiAffinity\":map[string]interface {}{\"preferredDuringSchedulingIgnoredDuringExecution\":[]interface {}{map[string]interface {}{\"podAffinityTerm\":map[string]interface {}{\"labelSelector\":map[string]interface {}{\"matchExpressions\":[]interface {}{map[string]interface {}{\"key\":\"prometheus\", \"operator\":\"In\", \"values\":[]interface {}{\"k8s-system\"}}}}, \"namespaces\":[]interface {}{\"kubesphere-monitoring-system\"}, \"topologyKey\":\"kubernetes.io/hostname\"}, \"weight\":100}}}}, \"baseImage\":\"dockerhub.qingcloud.com/prometheus/prometheus\", \"imagePullSecrets\":[]interface {}{map[string]interface {}{\"name\":\"qingcloud\"}},\"nodeSelector\":map[string]interface {}{\"beta.kubernetes.io/os\":\"linux\"}, \"query\":map[string]interface{}{\"maxConcurrency\":1000}, \"replicas\":2, \"resources\":map[string]interface {}{\"limits\":map[string]interface {}{\"cpu\":\"2.5\", \"memory\":\"2Gi\"}, \"requests\":map[string]interface {}{\"cpu\":\"60m\", \"memory\":\"400Mi\"}}, \"retention\":\"7d\", \"ruleSelector\":map[string]interface {}{\"matchLabels\":map[string]interface {}{\"prometheus\":\"k8s\", \"role\":\"alert-rules\"}}, \"scrapeInterval\":\"1m\", \"secrets\":[]interface{}{\"kube-etcd-client-certs\"}, \"securityContext\":map[string]interface {}{\"fsGroup\":0, \"runAsNonRoot\":false, \"runAsUser\":0}, \"serviceAccountName\":\"prometheus-k8s\", \"serviceMonitorNamespaceSelector\":map[string]interface {}{}, \"serviceMonitorSelector\":map[string]interface {}{\"matchExpressions\":[]interface {}{map[string]interface {}{\"key\":\"k8s-app\", \"operator\":\"In\", \"values\":[]interface {}{\"etcd\", \"coredns\", \"apiserver\", \"prometheus\", \"kube-scheduler\", \"kube-controller-manager\"}}}}, \"storage\":map[string]interface {}{\"volumeClaimTemplate\":map[string]interface {}{\"spec\":map[string]interface {}{\"resources\":map[string]interface {}{\"requests\":map[string]interface {}{\"storage\":\"20Gi\"}}, \"storageClassName\":interface {}(nil)}}}, \"tolerations\":[]interface {}{map[string]interface {}{\"effect\":\"NoSchedule\", \"key\":\"dedicated\", \"operator\":\"Equal\", \"value\":\"monitoring\"}}, \"version\":\"v2.5.0\"}}: validation failure list:\nspec.storage.volumeClaimTemplate.spec.storageClassName in body must be of type string: \"null\"", "stderr_lines": ["Error from server (Invalid): error when creating \"/etc/kubesphere/prometheus/sources/prometheus-prometheus.yaml\": Prometheus.monitoring.coreos.com \"k8s\" is invalid: []: Invalid value: map[string]interface {}{\"apiVersion\":\"monitoring.coreos.com/v1\", \"kind\":\"Prometheus\", \"metadata\":map[string]interface {}{\"annotations\":map[string]interface {}{\"kubectl.kubernetes.io/last-applied-configuration\":\"{\\"apiVersion\\":\\"monitoring.coreos.com/v1\\",\\"kind\\":\\"Prometheus\\",\\"metadata\\":{\\"annotations\\":{},\\"labels\\":{\\"prometheus\\":\\"k8s\\"},\\"name\\":\\"k8s\\",\\"namespace\\":\\"kubesphere-monitoring-system\\"},\\"spec\\":{\\"affinity\\":{\\"podAntiAffinity\\":{\\"preferredDuringSchedulingIgnoredDuringExecution\\":[{\\"podAffinityTerm\\":{\\"labelSelector\\":{\\"matchExpressions\\":[{\\"key\\":\\"prometheus\\",\\"operator\\":\\"In\\",\\"values\\":[\\"k8s\\"]}]},\\"namespaces\\":[\\"kubesphere-monitoring-system\\"],\\"topologyKey\\":\\"kubernetes.io/hostname\\"},\\"weight\\":100}]}},\\"baseImage\\":\\"dockerhub.qingcloud.com/prometheus/prometheus\\",\\"imagePullSecrets\\":[{\\"name\\":\\"qingcloud\\"}],\\"nodeSelector\\":{\\"beta.kubernetes.io/os\\":\\"linux\\"},\\"query\\":{\\"maxConcurrency\\":1000},\\"replicas\\":2,\\"resources\\":{\\"limits\\":{\\"cpu\\":\\"2.5\\",\\"memory\\":\\"2Gi\\"},\\"requests\\":{\\"cpu\\":\\"80m\\",\\"memory\\":\\"400Mi\\"}},\\"retention\\":\\"7d\\",\\"ruleSelector\\":{\\"matchLabels\\":{\\"prometheus\\":\\"k8s\\",\\"role\\":\\"alert-rules\\"}},\\"scrapeInterval\\":\\"1m\\",\\"securityContext\\":{\\"fsGroup\\":0,\\"runAsNonRoot\\":false,\\"runAsUser\\":0},\\"serviceAccountName\\":\\"prometheus-k8s\\",\\"serviceMonitorNamespaceSelector\\":{},\\"serviceMonitorSelector\\":{\\"matchExpressions\\":[{\\"key\\":\\"k8s-app\\",\\"operator\\":\\"In\\",\\"values\\":[\\"kube-state-metrics\\",\\"node-exporter\\",\\"kubelet\\",\\"prometheus-system\\"]}]},\\"storage\\":{\\"volumeClaimTemplate\\":{\\"spec\\":{\\"resources\\":{\\"requests\\":{\\"storage\\":\\"20Gi\\"}},\\"storageClassName\\":null}}},\\"tolerations\\":[{\\"effect\\":\\"NoSchedule\\",\\"key\\":\\"dedicated\\",\\"operator\\":\\"Equal\\",\\"value\\":\\"monitoring\\"}],\\"version\\":\\"v2.5.0\\"}}\n\"}, \"creationTimestamp\":\"2019-10-25T19:22:48Z\", \"generation\":1, \"labels\":map[string]interface {}{\"prometheus\":\"k8s\"}, \"name\":\"k8s\", \"namespace\":\"kubesphere-monitoring-system\", \"uid\":\"f41b2c48-f370-46be-b923-e2478db45e10\"}, \"spec\":map[string]interface {}{\"affinity\":map[string]interface {}{\"podAntiAffinity\":map[string]interface {}{\"preferredDuringSchedulingIgnoredDuringExecution\":[]interface {}{map[string]interface {}{\"podAffinityTerm\":map[string]interface {}{\"labelSelector\":map[string]interface {}{\"matchExpressions\":[]interface {}{map[string]interface {}{\"key\":\"prometheus\", \"operator\":\"In\", \"values\":[]interface {}{\"k8s\"}}}}, \"namespaces\":[]interface {}{\"kubesphere-monitoring-system\"}, \"topologyKey\":\"kubernetes.io/hostname\"}, \"weight\":100}}}}, \"baseImage\":\"dockerhub.qingcloud.com/prometheus/prometheus\", \"imagePullSecrets\":[]interface {}{map[string]interface {}{\"name\":\"qingcloud\"}}, \"nodeSelector\":map[string]interface {}{\"beta.kubernetes.io/os\":\"linux\"}, \"query\":map[string]interface {}{\"maxConcurrency\":1000}, \"replicas\":2, \"resources\":map[string]interface {}{\"limits\":map[string]interface {}{\"cpu\":\"2.5\", \"memory\":\"2Gi\"}, \"requests\":map[string]interface {}{\"cpu\":\"80m\", \"memory\":\"400Mi\"}}, \"retention\":\"7d\", \"ruleSelector\":map[string]interface {}{\"matchLabels\":map[string]interface {}{\"prometheus\":\"k8s\", \"role\":\"alert-rules\"}}, \"scrapeInterval\":\"1m\", \"securityContext\":map[string]interface {}{\"fsGroup\":0, \"runAsNonRoot\":false, \"runAsUser\":0}, \"serviceAccountName\":\"prometheus-k8s\", \"serviceMonitorNamespaceSelector\":map[string]interface {}{}, \"serviceMonitorSelector\":map[string]interface {}{\"matchExpressions\":[]interface {}{map[string]interface {}{\"key\":\"k8s-app\", \"operator\":\"In\", \"values\":[]interface {}{\"kube-state-metrics\", \"node-exporter\", \"kubelet\", \"prometheus-system\"}}}}, \"storage\":map[string]interface {}{\"volumeClaimTemplate\":map[string]interface {}{\"spec\":map[string]interface {}{\"resources\":map[string]interface {}{\"requests\":map[string]interface {}{\"storage\":\"20Gi\"}}, \"storageClassName\":interface {}(nil)}}}, \"tolerations\":[]interface {}{map[string]interface {}{\"effect\":\"NoSchedule\", \"key\":\"dedicated\", \"operator\":\"Equal\", \"value\":\"monitoring\"}}, \"version\":\"v2.5.0\"}}: validation failure list:", "spec.storage.volumeClaimTemplate.spec.storageClassName in body must be of type string: \"null\"", "Error from server (Invalid): error when creating \"/etc/kubesphere/prometheus/sources/prometheus-prometheusSystem.yaml\": Prometheus.monitoring.coreos.com \"k8s-system\" is invalid: []: Invalid value: map[string]interface {}{\"apiVersion\":\"monitoring.coreos.com/v1\", \"kind\":\"Prometheus\", \"metadata\":map[string]interface {}{\"annotations\":map[string]interface {}{\"kubectl.kubernetes.io/last-applied-configuration\":\"{\\"apiVersion\\":\\"monitoring.coreos.com/v1\\",\\"kind\\":\\"Prometheus\\",\\"metadata\\":{\\"annotations\\":{},\\"labels\\":{\\"prometheus\\":\\"k8s-system\\"},\\"name\\":\\"k8s-system\\",\\"namespace\\":\\"kubesphere-monitoring-system\\"},\\"spec\\":{\\"additionalScrapeConfigs\\":{\\"key\\":\\"prometheus-additional.yaml\\",\\"name\\":\\"additional-scrape-configs\\"},\\"affinity\\":{\\"podAntiAffinity\\":{\\"preferredDuringSchedulingIgnoredDuringExecution\\":[{\\"podAffinityTerm\\":{\\"labelSelector\\":{\\"matchExpressions\\":[{\\"key\\":\\"prometheus\\",\\"operator\\":\\"In\\",\\"values\\":[\\"k8s-system\\"]}]},\\"namespaces\\":[\\"kubesphere-monitoring-system\\"],\\"topologyKey\\":\\"kubernetes.io/hostname\\"},\\"weight\\":100}]}},\\"baseImage\\":\\"dockerhub.qingcloud.com/prometheus/prometheus\\",\\"imagePullSecrets\\":[{\\"name\\":\\"qingcloud\\"}],\\"nodeSelector\\":{\\"beta.kubernetes.io/os\\":\\"linux\\"},\\"query\\":{\\"maxConcurrency\\":1000},\\"replicas\\":2,\\"resources\\":{\\"limits\\":{\\"cpu\\":\\"2.5\\",\\"memory\\":\\"2Gi\\"},\\"requests\\":{\\"cpu\\":\\"60m\\",\\"memory\\":\\"400Mi\\"}},\\"retention\\":\\"7d\\",\\"ruleSelector\\":{\\"matchLabels\\":{\\"prometheus\\":\\"k8s\\",\\"role\\":\\"alert-rules\\"}},\\"scrapeInterval\\":\\"1m\\",\\"secrets\\":[\\"kube-etcd-client-certs\\"],\\"securityContext\\":{\\"fsGroup\\":0,\\"runAsNonRoot\\":false,\\"runAsUser\\":0},\\"serviceAccountName\\":\\"prometheus-k8s\\",\\"serviceMonitorNamespaceSelector\\":{},\\"serviceMonitorSelector\\":{\\"matchExpressions\\":[{\\"key\\":\\"k8s-app\\",\\"operator\\":\\"In\\",\\"values\\":[\\"etcd\\",\\"coredns\\",\\"apiserver\\",\\"prometheus\\",\\"kube-scheduler\\",\\"kube-controller-manager\\"]}]},\\"storage\\":{\\"volumeClaimTemplate\\":{\\"spec\\":{\\"resources\\":{\\"requests\\":{\\"storage\\":\\"20Gi\\"}},\\"storageClassName\\":null}}},\\"tolerations\\":[{\\"effect\\":\\"NoSchedule\\",\\"key\\":\\"dedicated\\",\\"operator\\":\\"Equal\\",\\"value\\":\\"monitoring\\"}],\\"version\\":\\"v2.5.0\\"}}\n\"}, \"creationTimestamp\":\"2019-10-25T19:22:48Z\", \"generation\":1, \"labels\":map[string]interface {}{\"prometheus\":\"k8s-system\"}, \"name\":\"k8s-system\", \"namespace\":\"kubesphere-monitoring-system\", \"uid\":\"868782a4-2536-4662-967d-84aa8139d21d\"}, \"spec\":map[string]interface {}{\"additionalScrapeConfigs\":map[string]interface {}{\"key\":\"prometheus-additional.yaml\", \"name\":\"additional-scrape-configs\"}, \"affinity\":map[string]interface {}{\"podAntiAffinity\":map[string]interface {}{\"preferredDuringSchedulingIgnoredDuringExecution\":[]interface {}{map[string]interface {}{\"podAffinityTerm\":map[string]interface {}{\"labelSelector\":map[string]interface {}{\"matchExpressions\":[]interface {}{map[string]interface {}{\"key\":\"prometheus\", \"operator\":\"In\", \"values\":[]interface {}{\"k8s-system\"}}}}, \"namespaces\":[]interface {}{\"kubesphere-monitoring-system\"}, \"topologyKey\":\"kubernetes.io/hostname\"}, \"weight\":100}}}}, \"baseImage\":\"dockerhub.qingcloud.com/prometheus/prometheus\", \"imagePullSecrets\":[]interface {}{map[string]interface {}{\"name\":\"qingcloud\"}}, \"nodeSelector\":map[string]interface {}{\"beta.kubernetes.io/os\":\"linux\"}, \"query\":map[string]interface {}{\"maxConcurrency\":1000}, \"replicas\":2, \"resources\":map[string]interface {}{\"limits\":map[string]interface {}{\"cpu\":\"2.5\", \"memory\":\"2Gi\"}, \"requests\":map[string]interface {}{\"cpu\":\"60m\", \"memory\":\"400Mi\"}}, \"retention\":\"7d\", \"ruleSelector\":map[string]interface {}{\"matchLabels\":map[string]interface {}{\"prometheus\":\"k8s\", \"role\":\"alert-rules\"}}, \"scrapeInterval\":\"1m\", \"secrets\":[]interface {}{\"kube-etcd-client-certs\"}, \"securityContext\":map[string]interface {}{\"fsGroup\":0, \"runAsNonRoot\":false, \"runAsUser\":0}, \"serviceAccountName\":\"prometheus-k8s\", \"serviceMonitorNamespaceSelector\":map[string]interface {}{}, \"serviceMonitorSelector\":map[string]interface {}{\"matchExpressions\":[]interface {}{map[string]interface {}{\"key\":\"k8s-app\", \"operator\":\"In\", \"values\":[]interface {}{\"etcd\", \"coredns\", \"apiserver\", \"prometheus\", \"kube-scheduler\", \"kube-controller-manager\"}}}}, \"storage\":map[string]interface {}{\"volumeClaimTemplate\":map[string]interface {}{\"spec\":map[string]interface {}{\"resources\":map[string]interface {}{\"requests\":map[string]interface {}{\"storage\":\"20Gi\"}}, \"storageClassName\":interface {}(nil)}}}, \"tolerations\":[]interface {}{map[string]interface {}{\"effect\":\"NoSchedule\", \"key\":\"dedicated\", \"operator\":\"Equal\", \"value\":\"monitoring\"}}, \"version\":\"v2.5.0\"}}: validation failure list:", "spec.storage.volumeClaimTemplate.spec.storageClassName in body must be of type string: \"null\""], "stdout": "secret/additional-scrape-configs unchanged\nclusterrole.rbac.authorization.k8s.io/kube-state-metrics unchanged\nclusterrolebinding.rbac.authorization.k8s.io/kube-state-metrics unchanged\ndeployment.apps/kube-state-metrics unchanged\nrole.rbac.authorization.k8s.io/kube-state-metrics unchanged\nrolebinding.rbac.authorization.k8s.io/kube-state-metrics unchanged\nservice/kube-state-metrics unchanged\nserviceaccount/kube-state-metrics unchanged\nservicemonitor.monitoring.coreos.com/kube-state-metrics unchanged\nclusterrole.rbac.authorization.k8s.io/node-exporter unchanged\nclusterrolebinding.rbac.authorization.k8s.io/node-exporter unchanged\ndaemonset.apps/node-exporter configured\nservice/node-exporter unchanged\nserviceaccount/node-exporter unchanged\nservicemonitor.monitoring.coreos.com/node-exporter unchanged\nclusterrole.rbac.authorization.k8s.io/prometheus-k8s unchanged\nclusterrolebinding.rbac.authorization.k8s.io/prometheus-k8s unchanged\nendpoints/etcd unchanged\nrolebinding.rbac.authorization.k8s.io/prometheus-k8s-config unchanged\nrolebinding.rbac.authorization.k8s.io/prometheus-k8s unchanged\nrolebinding.rbac.authorization.k8s.io/prometheus-k8s unchanged\nrolebinding.rbac.authorization.k8s.io/prometheus-k8s unchanged\nrolebinding.rbac.authorization.k8s.io/prometheus-k8s unchanged\nrole.rbac.authorization.k8s.io/prometheus-k8s-config unchanged\nrole.rbac.authorization.k8s.io/prometheus-k8s unchanged\nrole.rbac.authorization.k8s.io/prometheus-k8s unchanged\nrole.rbac.authorization.k8s.io/prometheus-k8s unchanged\nrole.rbac.authorization.k8s.io/prometheus-k8s unchanged\nprometheusrule.monitoring.coreos.com/prometheus-k8s-rules unchanged\nservice/prometheus-k8s unchanged\nserviceaccount/prometheus-k8s unchanged\nservice/etcd configured\nservice/kube-controller-manager unchanged\nservice/kube-scheduler unchanged\nservicemonitor.monitoring.coreos.com/prometheus unchanged\nservicemonitor.monitoring.coreos.com/kube-apiserver unchanged\nservicemonitor.monitoring.coreos.com/coredns unchanged\nservicemonitor.monitoring.coreos.com/etcd unchanged\nservicemonitor.monitoring.coreos.com/kube-controller-manager unchanged\nservicemonitor.monitoring.coreos.com/kube-scheduler unchanged\nservicemonitor.monitoring.coreos.com/kubelet unchanged\nservicemonitor.monitoring.coreos.com/prometheus-system unchanged\nservice/prometheus-k8s-system unchanged", "stdout_lines": ["secret/additional-scrape-configs unchanged", "clusterrole.rbac.authorization.k8s.io/kube-state-metrics unchanged", "clusterrolebinding.rbac.authorization.k8s.io/kube-state-metrics unchanged", "deployment.apps/kube-state-metrics unchanged", "role.rbac.authorization.k8s.io/kube-state-metrics unchanged", "rolebinding.rbac.authorization.k8s.io/kube-state-metrics unchanged", "service/kube-state-metrics unchanged", "serviceaccount/kube-state-metrics unchanged", "servicemonitor.monitoring.coreos.com/kube-state-metrics unchanged", "clusterrole.rbac.authorization.k8s.io/node-exporter unchanged", "clusterrolebinding.rbac.authorization.k8s.io/node-exporter unchanged", "daemonset.apps/node-exporter configured", "service/node-exporter unchanged", "serviceaccount/node-exporter unchanged", "servicemonitor.monitoring.coreos.com/node-exporter unchanged", "clusterrole.rbac.authorization.k8s.io/prometheus-k8s unchanged", "clusterrolebinding.rbac.authorization.k8s.io/prometheus-k8s unchanged", "endpoints/etcd unchanged", "rolebinding.rbac.authorization.k8s.io/prometheus-k8s-config unchanged", "rolebinding.rbac.authorization.k8s.io/prometheus-k8s unchanged", "rolebinding.rbac.authorization.k8s.io/prometheus-k8s unchanged", "rolebinding.rbac.authorization.k8s.io/prometheus-k8s unchanged", "rolebinding.rbac.authorization.k8s.io/prometheus-k8s unchanged", "role.rbac.authorization.k8s.io/prometheus-k8s-config unchanged", "role.rbac.authorization.k8s.io/prometheus-k8s unchanged", "role.rbac.authorization.k8s.io/prometheus-k8s unchanged", "role.rbac.authorization.k8s.io/prometheus-k8s unchanged", "role.rbac.authorization.k8s.io/prometheus-k8s unchanged", "prometheusrule.monitoring.coreos.com/prometheus-k8s-rules unchanged", "service/prometheus-k8s unchanged", "serviceaccount/prometheus-k8s unchanged", "service/etcd configured", "service/kube-controller-manager unchanged", "service/kube-scheduler unchanged", "servicemonitor.monitoring.coreos.com/prometheus unchanged", "servicemonitor.monitoring.coreos.com/kube-apiserver unchanged", "servicemonitor.monitoring.coreos.com/coredns unchanged", "servicemonitor.monitoring.coreos.com/etcd unchanged", "servicemonitor.monitoring.coreos.com/kube-controller-manager unchanged", "servicemonitor.monitoring.coreos.com/kube-scheduler unchanged", "servicemonitor.monitoring.coreos.com/kubelet unchanged", "servicemonitor.monitoring.coreos.com/prometheus-system unchanged", "service/prometheus-k8s-system unchanged"]}

PLAY RECAP *****

localhost : ok=61 changed=52 unreachable=0 failed=1 skipped=9 rescued=0 ignored=0

[pixiake]

config 中是否配置了storageclass

[ghost]

@pixiake when set up, i met the same issue with error as shown up by Akdie, the problem resolved by following this discussion;

https://github.com/helm/helm/issues/3130#issuecomment-345506262

why not update the install README.md by adding the following scripts:

# upgrade helm
helm init --upgrade --service-account tiller

# helm_rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: tiller
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: tiller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: tiller
    namespace: kube-system

# apply helm_rbac.yaml
kubectl apply -f helm_rabc.yaml

you can edit and make some adjustment when somewhere wrong i expressed; i hope this will make help for new commer to k8s and kubesphere;

[AkideLiu]

@hsdtsyl Thanks for your resolution given.

I have reviewd your resolution that related to helm rabc. But it may not just related to rabc role management and authentication due there are several errors occur.

After that errors disappeared when i use the same version helm client and server while the kubernetes cluster downgraded.

In last few days, 4 different deployment mode tests was done about kubesphere and figured out that kubesphere 2.0.2 currently can support kubesphere kubernetes v1.13.5 properly on my end.

You may can see the test results in following link: https://github.com/kubesphere/ks-installer/issues/23#issuecomment-546528093

Supported kubernetes version on my end was found from all-in-one deployment mode.

[AkideLiu]

@pixiake

4.storageClass：

kubectl get sc

NAME PROVISIONER AGE

thin-disk kubernetes.io/vsphere-volume 10h

configure 中没有定义特别的storageclass，这个SC是default 也是惟一的

[pixiake]

config 中要填下storageclass name

[kissingtiger]

我也遇到同样的问题，请问楼主解决了吗？

[AkideLiu]

我也遇到同样的问题，请问楼主解决了吗？

https://github.com/kubesphere/ks-installer/issues/334#issuecomment-546861289

我的办法是使用低版本的k8s

[kissingtiger]

请问你成功的k8s版本是哪个版本？ 我的环境是： k8s 1.13.10 helm 2.13.1

会出现下面的报错 FAILED - RETRYING: ks-monitor | Installing prometheus-operator (5 retries left).

FAILED - RETRYING: ks-monitor | Installing prometheus-operator (4 retries left).

FAILED - RETRYING: ks-monitor | Installing prometheus-operator (3 retries left).

FAILED - RETRYING: ks-monitor | Installing prometheus-operator (2 retries left).

FAILED - RETRYING: ks-monitor | Installing prometheus-operator (1 retries left).

failed: [localhost] (item=sources) => {"ansible_loop_var": "item", "attempts": 5, "changed": true, "cmd": "/usr/local/bin/kubectl apply -f /etc/kubesphere/prometheus/sources", "delta": "0:00:01.786886", "end": "2019-10-25 19:22:49.231470", "failed_when_result": true, "item": "sources", "msg": "non-zero return code", "rc": 1, "start": "2019-10-25 19:22:47.444584", "stderr": "Error from server (Invalid): error when creating "/etc/kubesphere/prometheus/sources/prometheus-prometheus.yaml": Prometheus.monitoring.coreos.com "k8s" is invalid: []:Invalid value: map[string]interface {}{"apiVersion":"monitoring.coreos.com/v1", "kind":"Prometheus", "metadata":map[string]interface {}{"annotations":map[string]interface {}{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"monitoring.coreos.com/v1\",\"kind\":\"Prometheus\",\"metadata\":{\"annotations\":{},\"labels\":{\"prometheus\":\"k8s\"},\"name\":\"k8s\",\"namespace\":\"kubesphere-monitoring-system\"},\"spec\":{\"affinity\":{\"podAntiAffinity\":{\"preferredDuringSchedulingIgnoredDuringExecution\":[{\"podAffinityTerm\":{\"labelSelector\":{\"matchExpressions\":[{\"key\":\"prometheus\",\"operator\":\"In\",\"values\":[\"k8s\"]}]},\"namespaces\":[\"kubesphere-monitoring-system\"],\"topologyKey\":\"kubernetes.io/hostname\"},\"weight\":100}]}},\"baseImage\":\"dockerhub.qingcloud.com/prometheus/prometheus\",\"imagePullSecrets\":[{\"name\":\"qingcloud\"}],\"nodeSelector\":{\"beta.kubernetes.io/os\":\"linux\"},\"query\":{\"maxConcurrency\":1000},\"replicas\":2,\"resources\":{\"limits\":{\"cpu\":\"2.5\",\"memory\":\"2Gi\"},\"requests\":{\"cpu\":\"80m\",\"memory\":\"400Mi\"}},\"retention\":\"7d\",\"ruleSelector\":{\"matchLabels\":{\"prometheus\":\"k8s\",\"role\":\"alert-rules\"}},\"scrapeInterval\":\"1m\",\"securityContext\":{\"fsGroup\":0,\"runAsNonRoot\":false,\"runAsUser\":0},\"serviceAccountName\":\"prometheus-k8s\",\"serviceMonitorNamespaceSelector\":{},\"serviceMonitorSelector\":{\"matchExpressions\":[{\"key\":\"k8s-app\",\"operator\":\"In\",\"values\":[\"kube-state-metrics\",\"node-exporter\",\"kubelet\",\"prometheus-system\"]}]},\"storage\":{\"volumeClaimTemplate\":{\"spec\":{\"resources\":{\"requests\":{\"storage\":\"20Gi\"}},\"storageClassName\":null}}},\"tolerations\":[{\"effect\":\"NoSchedule\",\"key\":\"dedicated\",\"operator\":\"Equal\",\"value\":\"monitoring\"}],\"version\":\"v2.5.0\"}}\n"}, "creationTimestamp":"2019-10-25T19:22:48Z", "generation":1, "labels":map[string]interface {}{"prometheus":"k8s"}, "name":"k8s", "namespace":"kubesphere-monitoring-system", "uid":"f41b2c48-f370-46be-b923-e2478db45e10"}, "spec":map[string]interface {}{"affinity":map[string]interface {}{"podAntiAffinity":map[string]interface {}{"preferredDuringSchedulingIgnoredDuringExecution":[]interface {}{map[string]interface {}{"podAffinityTerm":map[string]interface {}{"labelSelector":map[string]interface {}{"matchExpressions":[]interface {}{map[string]interface {}{"key":"prometheus", "operator":"In", "values":[]interface {}{"k8s"}}}}, "namespaces":[]interface {}{"kubesphere-monitoring-system"}, "topologyKey":"kubernetes.io/hostname"}, "weight":100}}}}, "baseImage":"dockerhub.qingcloud.com/prometheus/prometheus", "imagePullSecrets":[]interface {}{map[string]interface {}{"name":"qingcloud"}}, "nodeSelector":map[string]interface {}{"beta.kubernetes.io/os":"linux"}, "query":map[string]interface {}{"maxConcurrency":1000}, "replicas":2, "resources":map[string]interface {}{"limits":map[string]interface {}{"cpu":"2.5", "memory":"2Gi"}, "requests":map[string]interface {}{"cpu":"80m", "memory":"400Mi"}}, "retention":"7d", "ruleSelector":map[string]interface{}{"matchLabels":map[string]interface {}{"prometheus":"k8s", "role":"alert-rules"}}, "scrapeInterval":"1m", "securityContext":map[string]interface {}{"fsGroup":0, "runAsNonRoot":false, "runAsUser":0}, "serviceAccountName":"prometheus-k8s", "serviceMonitorNamespaceSelector":map[string]interface {}{}, "serviceMonitorSelector":map[string]interface {}{"matchExpressions":[]interface {}{map[string]interface {}{"key":"k8s-app", "operator":"In", "values":[]interface {}{"kube-state-metrics", "node-exporter", "kubelet", "prometheus-system"}}}}, "storage":map[string]interface {}{"volumeClaimTemplate":map[string]interface {}{"spec":map[string]interface {}{"resources":map[string]interface {}{"requests":map[string]interface {}{"storage":"20Gi"}}, "storageClassName":interface {}(nil)}}}, "tolerations":[]interface {}{map[string]interface {}{"effect":"NoSchedule", "key":"dedicated", "operator":"Equal", "value":"monitoring"}}, "version":"v2.5.0"}}: validation failure list:\nspec.storage.volumeClaimTemplate.spec.storageClassName in body must be of type string: "null"\nError from server (Invalid): error when creating "/etc/kubesphere/prometheus/sources/prometheus-prometheusSystem.yaml": Prometheus.monitoring.coreos.com "k8s-system" is invalid: []: Invalid value: map[string]interface {}{"apiVersion":"monitoring.coreos.com/v1", "kind":"Prometheus", "metadata":map[string]interface {}{"annotations":map[string]interface {}{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"monitoring.coreos.com/v1\",\"kind\":\"Prometheus\",\"metadata\":{\"annotations\":{},\"labels\":{\"prometheus\":\"k8s-system\"},\"name\":\"k8s-system\",\"namespace\":\"kubesphere-monitoring-system\"},\"spec\":{\"additionalScrapeConfigs\":{\"key\":\"prometheus-additional.yaml\",\"name\":\"additional-scrape-configs\"},\"affinity\":{\"podAntiAffinity\":{\"preferredDuringSchedulingIgnoredDuringExecution\":[{\"podAffinityTerm\":{\"labelSelector\":{\"matchExpressions\":[{\"key\":\"prometheus\",\"operator\":\"In\",\"values\":[\"k8s-system\"]}]},\"namespaces\":[\"kubesphere-monitoring-system\"],\"topologyKey\":\"kubernetes.io/hostname\"},\"weight\":100}]}},\"baseImage\":\"dockerhub.qingcloud.com/prometheus/prometheus\",\"imagePullSecrets\":[{\"name\":\"qingcloud\"}],\"nodeSelector\":{\"beta.kubernetes.io/os\":\"linux\"},\"query\":{\"maxConcurrency\":1000},\"replicas\":2,\"resources\":{\"limits\":{\"cpu\":\"2.5\",\"memory\":\"2Gi\"},\"requests\":{\"cpu\":\"60m\",\"memory\":\"400Mi\"}},\"retention\":\"7d\",\"ruleSelector\":{\"matchLabels\":{\"prometheus\":\"k8s\",\"role\":\"alert-rules\"}},\"scrapeInterval\":\"1m\",\"secrets\":[\"kube-etcd-client-certs\"],\"securityContext\":{\"fsGroup\":0,\"runAsNonRoot\":false,\"runAsUser\":0},\"serviceAccountName\":\"prometheus-k8s\",\"serviceMonitorNamespaceSelector\":{},\"serviceMonitorSelector\":{\"matchExpressions\":[{\"key\":\"k8s-app\",\"operator\":\"In\",\"values\":[\"etcd\",\"coredns\",\"apiserver\",\"prometheus\",\"kube-scheduler\",\"kube-controller-manager\"]}]},\"storage\":{\"volumeClaimTemplate\":{\"spec\":{\"resources\":{\"requests\":{\"storage\":\"20Gi\"}},\"storageClassName\":null}}},\"tolerations\":[{\"effect\":\"NoSchedule\",\"key\":\"dedicated\",\"operator\":\"Equal\",\"value\":\"monitoring\"}],\"version\":\"v2.5.0\"}}\n"}, "creationTimestamp":"2019-10-25T19:22:48Z", "generation":1, "labels":map[string]interface {}{"prometheus":"k8s-system"}, "name":"k8s-system", "namespace":"kubesphere-monitoring-system", "uid":"868782a4-2536-4662-967d-84aa8139d21d"}, "spec":map[string]interface {}{"additionalScrapeConfigs":map[string]interface {}{"key":"prometheus-additional.yaml", "name":"additional-scrape-configs"}, "affinity":map[string]interface {}{"podAntiAffinity":map[string]interface {}{"preferredDuringSchedulingIgnoredDuringExecution":[]interface {}{map[string]interface {}{"podAffinityTerm":map[string]interface {}{"labelSelector":map[string]interface {}{"matchExpressions":[]interface {}{map[string]interface {}{"key":"prometheus", "operator":"In", "values":[]interface {}{"k8s-system"}}}}, "namespaces":[]interface {}{"kubesphere-monitoring-system"}, "topologyKey":"kubernetes.io/hostname"}, "weight":100}}}}, "baseImage":"dockerhub.qingcloud.com/prometheus/prometheus", "imagePullSecrets":[]interface {}{map[string]interface {}{"name":"qingcloud"}},"nodeSelector":map[string]interface {}{"beta.kubernetes.io/os":"linux"}, "query":map[string]interface{}{"maxConcurrency":1000}, "replicas":2, "resources":map[string]interface {}{"limits":map[string]interface {}{"cpu":"2.5", "memory":"2Gi"}, "requests":map[string]interface {}{"cpu":"60m", "memory":"400Mi"}}, "retention":"7d", "ruleSelector":map[string]interface {}{"matchLabels":map[string]interface {}{"prometheus":"k8s", "role":"alert-rules"}}, "scrapeInterval":"1m", "secrets":[]interface{}{"kube-etcd-client-certs"}, "securityContext":map[string]interface {}{"fsGroup":0, "runAsNonRoot":false, "runAsUser":0}, "serviceAccountName":"prometheus-k8s", "serviceMonitorNamespaceSelector":map[string]interface {}{}, "serviceMonitorSelector":map[string]interface {}{"matchExpressions":[]interface {}{map[string]interface {}{"key":"k8s-app", "operator":"In", "values":[]interface {}{"etcd", "coredns", "apiserver", "prometheus", "kube-scheduler", "kube-controller-manager"}}}}, "storage":map[string]interface {}{"volumeClaimTemplate":map[string]interface {}{"spec":map[string]interface {}{"resources":map[string]interface {}{"requests":map[string]interface {}{"storage":"20Gi"}}, "storageClassName":interface {}(nil)}}}, "tolerations":[]interface {}{map[string]interface {}{"effect":"NoSchedule", "key":"dedicated", "operator":"Equal", "value":"monitoring"}}, "version":"v2.5.0"}}: validation failure list:\nspec.storage.volumeClaimTemplate.spec.storageClassName in body must be of type string: "null"", "stderr_lines": ["Error from server (Invalid): error when creating "/etc/kubesphere/prometheus/sources/prometheus-prometheus.yaml": Prometheus.monitoring.coreos.com "k8s" is invalid: []: Invalid value: map[string]interface {}{"apiVersion":"monitoring.coreos.com/v1", "kind":"Prometheus", "metadata":map[string]interface {}{"annotations":map[string]interface {}{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"monitoring.coreos.com/v1\",\"kind\":\"Prometheus\",\"metadata\":{\"annotations\":{},\"labels\":{\"prometheus\":\"k8s\"},\"name\":\"k8s\",\"namespace\":\"kubesphere-monitoring-system\"},\"spec\":{\"affinity\":{\"podAntiAffinity\":{\"preferredDuringSchedulingIgnoredDuringExecution\":[{\"podAffinityTerm\":{\"labelSelector\":{\"matchExpressions\":[{\"key\":\"prometheus\",\"operator\":\"In\",\"values\":[\"k8s\"]}]},\"namespaces\":[\"kubesphere-monitoring-system\"],\"topologyKey\":\"kubernetes.io/hostname\"},\"weight\":100}]}},\"baseImage\":\"dockerhub.qingcloud.com/prometheus/prometheus\",\"imagePullSecrets\":[{\"name\":\"qingcloud\"}],\"nodeSelector\":{\"beta.kubernetes.io/os\":\"linux\"},\"query\":{\"maxConcurrency\":1000},\"replicas\":2,\"resources\":{\"limits\":{\"cpu\":\"2.5\",\"memory\":\"2Gi\"},\"requests\":{\"cpu\":\"80m\",\"memory\":\"400Mi\"}},\"retention\":\"7d\",\"ruleSelector\":{\"matchLabels\":{\"prometheus\":\"k8s\",\"role\":\"alert-rules\"}},\"scrapeInterval\":\"1m\",\"securityContext\":{\"fsGroup\":0,\"runAsNonRoot\":false,\"runAsUser\":0},\"serviceAccountName\":\"prometheus-k8s\",\"serviceMonitorNamespaceSelector\":{},\"serviceMonitorSelector\":{\"matchExpressions\":[{\"key\":\"k8s-app\",\"operator\":\"In\",\"values\":[\"kube-state-metrics\",\"node-exporter\",\"kubelet\",\"prometheus-system\"]}]},\"storage\":{\"volumeClaimTemplate\":{\"spec\":{\"resources\":{\"requests\":{\"storage\":\"20Gi\"}},\"storageClassName\":null}}},\"tolerations\":[{\"effect\":\"NoSchedule\",\"key\":\"dedicated\",\"operator\":\"Equal\",\"value\":\"monitoring\"}],\"version\":\"v2.5.0\"}}\n"}, "creationTimestamp":"2019-10-25T19:22:48Z", "generation":1, "labels":map[string]interface {}{"prometheus":"k8s"}, "name":"k8s", "namespace":"kubesphere-monitoring-system", "uid":"f41b2c48-f370-46be-b923-e2478db45e10"}, "spec":map[string]interface {}{"affinity":map[string]interface {}{"podAntiAffinity":map[string]interface {}{"preferredDuringSchedulingIgnoredDuringExecution":[]interface {}{map[string]interface {}{"podAffinityTerm":map[string]interface {}{"labelSelector":map[string]interface {}{"matchExpressions":[]interface {}{map[string]interface {}{"key":"prometheus", "operator":"In", "values":[]interface {}{"k8s"}}}}, "namespaces":[]interface {}{"kubesphere-monitoring-system"}, "topologyKey":"kubernetes.io/hostname"}, "weight":100}}}}, "baseImage":"dockerhub.qingcloud.com/prometheus/prometheus", "imagePullSecrets":[]interface {}{map[string]interface {}{"name":"qingcloud"}}, "nodeSelector":map[string]interface {}{"beta.kubernetes.io/os":"linux"}, "query":map[string]interface {}{"maxConcurrency":1000}, "replicas":2, "resources":map[string]interface {}{"limits":map[string]interface {}{"cpu":"2.5", "memory":"2Gi"}, "requests":map[string]interface {}{"cpu":"80m", "memory":"400Mi"}}, "retention":"7d", "ruleSelector":map[string]interface {}{"matchLabels":map[string]interface {}{"prometheus":"k8s", "role":"alert-rules"}}, "scrapeInterval":"1m", "securityContext":map[string]interface {}{"fsGroup":0, "runAsNonRoot":false, "runAsUser":0}, "serviceAccountName":"prometheus-k8s", "serviceMonitorNamespaceSelector":map[string]interface {}{}, "serviceMonitorSelector":map[string]interface {}{"matchExpressions":[]interface {}{map[string]interface {}{"key":"k8s-app", "operator":"In", "values":[]interface {}{"kube-state-metrics", "node-exporter", "kubelet", "prometheus-system"}}}}, "storage":map[string]interface {}{"volumeClaimTemplate":map[string]interface {}{"spec":map[string]interface {}{"resources":map[string]interface {}{"requests":map[string]interface {}{"storage":"20Gi"}}, "storageClassName":interface {}(nil)}}}, "tolerations":[]interface {}{map[string]interface {}{"effect":"NoSchedule", "key":"dedicated", "operator":"Equal", "value":"monitoring"}}, "version":"v2.5.0"}}: validation failure list:", "spec.storage.volumeClaimTemplate.spec.storageClassName in body must be of type string: "null"", "Error from server (Invalid): error when creating "/etc/kubesphere/prometheus/sources/prometheus-prometheusSystem.yaml": Prometheus.monitoring.coreos.com "k8s-system" is invalid: []: Invalid value: map[string]interface {}{"apiVersion":"monitoring.coreos.com/v1", "kind":"Prometheus", "metadata":map[string]interface {}{"annotations":map[string]interface {}{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"monitoring.coreos.com/v1\",\"kind\":\"Prometheus\",\"metadata\":{\"annotations\":{},\"labels\":{\"prometheus\":\"k8s-system\"},\"name\":\"k8s-system\",\"namespace\":\"kubesphere-monitoring-system\"},\"spec\":{\"additionalScrapeConfigs\":{\"key\":\"prometheus-additional.yaml\",\"name\":\"additional-scrape-configs\"},\"affinity\":{\"podAntiAffinity\":{\"preferredDuringSchedulingIgnoredDuringExecution\":[{\"podAffinityTerm\":{\"labelSelector\":{\"matchExpressions\":[{\"key\":\"prometheus\",\"operator\":\"In\",\"values\":[\"k8s-system\"]}]},\"namespaces\":[\"kubesphere-monitoring-system\"],\"topologyKey\":\"kubernetes.io/hostname\"},\"weight\":100}]}},\"baseImage\":\"dockerhub.qingcloud.com/prometheus/prometheus\",\"imagePullSecrets\":[{\"name\":\"qingcloud\"}],\"nodeSelector\":{\"beta.kubernetes.io/os\":\"linux\"},\"query\":{\"maxConcurrency\":1000},\"replicas\":2,\"resources\":{\"limits\":{\"cpu\":\"2.5\",\"memory\":\"2Gi\"},\"requests\":{\"cpu\":\"60m\",\"memory\":\"400Mi\"}},\"retention\":\"7d\",\"ruleSelector\":{\"matchLabels\":{\"prometheus\":\"k8s\",\"role\":\"alert-rules\"}},\"scrapeInterval\":\"1m\",\"secrets\":[\"kube-etcd-client-certs\"],\"securityContext\":{\"fsGroup\":0,\"runAsNonRoot\":false,\"runAsUser\":0},\"serviceAccountName\":\"prometheus-k8s\",\"serviceMonitorNamespaceSelector\":{},\"serviceMonitorSelector\":{\"matchExpressions\":[{\"key\":\"k8s-app\",\"operator\":\"In\",\"values\":[\"etcd\",\"coredns\",\"apiserver\",\"prometheus\",\"kube-scheduler\",\"kube-controller-manager\"]}]},\"storage\":{\"volumeClaimTemplate\":{\"spec\":{\"resources\":{\"requests\":{\"storage\":\"20Gi\"}},\"storageClassName\":null}}},\"tolerations\":[{\"effect\":\"NoSchedule\",\"key\":\"dedicated\",\"operator\":\"Equal\",\"value\":\"monitoring\"}],\"version\":\"v2.5.0\"}}\n"}, "creationTimestamp":"2019-10-25T19:22:48Z", "generation":1, "labels":map[string]interface {}{"prometheus":"k8s-system"}, "name":"k8s-system", "namespace":"kubesphere-monitoring-system", "uid":"868782a4-2536-4662-967d-84aa8139d21d"}, "spec":map[string]interface {}{"additionalScrapeConfigs":map[string]interface {}{"key":"prometheus-additional.yaml", "name":"additional-scrape-configs"}, "affinity":map[string]interface {}{"podAntiAffinity":map[string]interface {}{"preferredDuringSchedulingIgnoredDuringExecution":[]interface {}{map[string]interface {}{"podAffinityTerm":map[string]interface {}{"labelSelector":map[string]interface {}{"matchExpressions":[]interface {}{map[string]interface {}{"key":"prometheus", "operator":"In", "values":[]interface {}{"k8s-system"}}}}, "namespaces":[]interface {}{"kubesphere-monitoring-system"}, "topologyKey":"kubernetes.io/hostname"}, "weight":100}}}}, "baseImage":"dockerhub.qingcloud.com/prometheus/prometheus", "imagePullSecrets":[]interface {}{map[string]interface {}{"name":"qingcloud"}}, "nodeSelector":map[string]interface {}{"beta.kubernetes.io/os":"linux"}, "query":map[string]interface {}{"maxConcurrency":1000}, "replicas":2, "resources":map[string]interface {}{"limits":map[string]interface {}{"cpu":"2.5", "memory":"2Gi"}, "requests":map[string]interface {}{"cpu":"60m", "memory":"400Mi"}}, "retention":"7d", "ruleSelector":map[string]interface {}{"matchLabels":map[string]interface {}{"prometheus":"k8s", "role":"alert-rules"}}, "scrapeInterval":"1m", "secrets":[]interface {}{"kube-etcd-client-certs"}, "securityContext":map[string]interface {}{"fsGroup":0, "runAsNonRoot":false, "runAsUser":0}, "serviceAccountName":"prometheus-k8s", "serviceMonitorNamespaceSelector":map[string]interface {}{}, "serviceMonitorSelector":map[string]interface {}{"matchExpressions":[]interface {}{map[string]interface {}{"key":"k8s-app", "operator":"In", "values":[]interface {}{"etcd", "coredns", "apiserver", "prometheus", "kube-scheduler", "kube-controller-manager"}}}}, "storage":map[string]interface {}{"volumeClaimTemplate":map[string]interface {}{"spec":map[string]interface {}{"resources":map[string]interface {}{"requests":map[string]interface {}{"storage":"20Gi"}}, "storageClassName":interface {}(nil)}}}, "tolerations":[]interface {}{map[string]interface {}{"effect":"NoSchedule", "key":"dedicated", "operator":"Equal", "value":"monitoring"}}, "version":"v2.5.0"}}: validation failure list:", "spec.storage.volumeClaimTemplate.spec.storageClassName in body must be of type string: "null""], "stdout": "secret/additional-scrape-configs unchanged\nclusterrole.rbac.authorization.k8s.io/kube-state-metrics unchanged\nclusterrolebinding.rbac.authorization.k8s.io/kube-state-metrics unchanged\ndeployment.apps/kube-state-metrics unchanged\nrole.rbac.authorization.k8s.io/kube-state-metrics unchanged\nrolebinding.rbac.authorization.k8s.io/kube-state-metrics unchanged\nservice/kube-state-metrics unchanged\nserviceaccount/kube-state-metrics unchanged\nservicemonitor.monitoring.coreos.com/kube-state-metrics unchanged\nclusterrole.rbac.authorization.k8s.io/node-exporter unchanged\nclusterrolebinding.rbac.authorization.k8s.io/node-exporter unchanged\ndaemonset.apps/node-exporter configured\nservice/node-exporter unchanged\nserviceaccount/node-exporter unchanged\nservicemonitor.monitoring.coreos.com/node-exporter unchanged\nclusterrole.rbac.authorization.k8s.io/prometheus-k8s unchanged\nclusterrolebinding.rbac.authorization.k8s.io/prometheus-k8s unchanged\nendpoints/etcd unchanged\nrolebinding.rbac.authorization.k8s.io/prometheus-k8s-config unchanged\nrolebinding.rbac.authorization.k8s.io/prometheus-k8s unchanged\nrolebinding.rbac.authorization.k8s.io/prometheus-k8s unchanged\nrolebinding.rbac.authorization.k8s.io/prometheus-k8s unchanged\nrolebinding.rbac.authorization.k8s.io/prometheus-k8s unchanged\nrole.rbac.authorization.k8s.io/prometheus-k8s-config unchanged\nrole.rbac.authorization.k8s.io/prometheus-k8s unchanged\nrole.rbac.authorization.k8s.io/prometheus-k8s unchanged\nrole.rbac.authorization.k8s.io/prometheus-k8s unchanged\nrole.rbac.authorization.k8s.io/prometheus-k8s unchanged\nprometheusrule.monitoring.coreos.com/prometheus-k8s-rules unchanged\nservice/prometheus-k8s unchanged\nserviceaccount/prometheus-k8s unchanged\nservice/etcd configured\nservice/kube-controller-manager unchanged\nservice/kube-scheduler unchanged\nservicemonitor.monitoring.coreos.com/prometheus unchanged\nservicemonitor.monitoring.coreos.com/kube-apiserver unchanged\nservicemonitor.monitoring.coreos.com/coredns unchanged\nservicemonitor.monitoring.coreos.com/etcd unchanged\nservicemonitor.monitoring.coreos.com/kube-controller-manager unchanged\nservicemonitor.monitoring.coreos.com/kube-scheduler unchanged\nservicemonitor.monitoring.coreos.com/kubelet unchanged\nservicemonitor.monitoring.coreos.com/prometheus-system unchanged\nservice/prometheus-k8s-system unchanged", "stdout_lines": ["secret/additional-scrape-configs unchanged", "clusterrole.rbac.authorization.k8s.io/kube-state-metrics unchanged", "clusterrolebinding.rbac.authorization.k8s.io/kube-state-metrics unchanged", "deployment.apps/kube-state-metrics unchanged", "role.rbac.authorization.k8s.io/kube-state-metrics unchanged", "rolebinding.rbac.authorization.k8s.io/kube-state-metrics unchanged", "service/kube-state-metrics unchanged", "serviceaccount/kube-state-metrics unchanged", "servicemonitor.monitoring.coreos.com/kube-state-metrics unchanged", "clusterrole.rbac.authorization.k8s.io/node-exporter unchanged", "clusterrolebinding.rbac.authorization.k8s.io/node-exporter unchanged", "daemonset.apps/node-exporter configured", "service/node-exporter unchanged", "serviceaccount/node-exporter unchanged", "servicemonitor.monitoring.coreos.com/node-exporter unchanged", "clusterrole.rbac.authorization.k8s.io/prometheus-k8s unchanged", "clusterrolebinding.rbac.authorization.k8s.io/prometheus-k8s unchanged", "endpoints/etcd unchanged", "rolebinding.rbac.authorization.k8s.io/prometheus-k8s-config unchanged", "rolebinding.rbac.authorization.k8s.io/prometheus-k8s unchanged", "rolebinding.rbac.authorization.k8s.io/prometheus-k8s unchanged", "rolebinding.rbac.authorization.k8s.io/prometheus-k8s unchanged", "rolebinding.rbac.authorization.k8s.io/prometheus-k8s unchanged", "role.rbac.authorization.k8s.io/prometheus-k8s-config unchanged", "role.rbac.authorization.k8s.io/prometheus-k8s unchanged", "role.rbac.authorization.k8s.io/prometheus-k8s unchanged", "role.rbac.authorization.k8s.io/prometheus-k8s unchanged", "role.rbac.authorization.k8s.io/prometheus-k8s unchanged", "prometheusrule.monitoring.coreos.com/prometheus-k8s-rules unchanged", "service/prometheus-k8s unchanged", "serviceaccount/prometheus-k8s unchanged", "service/etcd configured", "service/kube-controller-manager unchanged", "service/kube-scheduler unchanged", "servicemonitor.monitoring.coreos.com/prometheus unchanged", "servicemonitor.monitoring.coreos.com/kube-apiserver unchanged", "servicemonitor.monitoring.coreos.com/coredns unchanged", "servicemonitor.monitoring.coreos.com/etcd unchanged", "servicemonitor.monitoring.coreos.com/kube-controller-manager unchanged", "servicemonitor.monitoring.coreos.com/kube-scheduler unchanged", "servicemonitor.monitoring.coreos.com/kubelet unchanged", "servicemonitor.monitoring.coreos.com/prometheus-system unchanged", "service/prometheus-k8s-system unchanged"]}

PLAY RECAP *****

localhost : ok=61 changed=52 unreachable=0 failed=1 skipped=9 rescued=0 ignored=0

[AkideLiu]

@kissingtiger 我是用的kubesphere 2.0.2，k8s v1.13.5 helm我忘记了，应该是2.1x版本