Invalid value: []byte{0xa}: unable to load root certificates: unable to parse bytes as PEM block

slieer commented 2 weeks ago

Describe the bug

Rocky Linux release 8.10 (Green Obsidian) minikube version: v1.34.0 kubectl version: Client Version: v1.31.2 Kustomize Version: v5.4.2 Server Version: v1.31.0 chectl version: chectl/7.94.0 linux-x64 node-v18.18.0 or chectl/7.93.0 linux-x64 node-v18.18.0

chectl server:deploy --platform minikube › Current Kubernetes context: 'minikube' ✔ Verify Kubernetes API...[1.31] ✔ Minikube preflight checklist ✔ Verify if kubectl is installed...[OK] ✔ Verify if minikube is installed...[OK] ✔ Verify if minikube is running...[OK] ✔ Enable minikube ingress addon...[Enabled] ✔ Retrieving minikube IP and domain for ingress URLs...[192.168.49.2.nip.io] ✔ Checking minikube version...[1.34.0] ✔ Create Namespace eclipse-che...[Exists] ✔ Install Cert Manager v1.8.2 ✔ Apply resources...[Exists] ✔ Wait for Cert Manager pods ready...[OK] ✔ Install Dex ✔ Create Namespace dex...[Exists] ✔ Create Certificates...[Exists: /tmp/dex-ca.crt] ✔ Create ConfigMap dex-ca...[Updated] ✔ Create ServiceAccount dex...[Exists] ✔ Create ClusterRole dex...[Exists] ✔ Create ClusterRoleBinding dex...[Exists] ✔ Create Service dex...[Exists] ✔ Create Ingress dex...[Exists] ✔ Generate Dex username and password...[Exists] ✔ Create ConfigMap dex...[Exists] ✔ Create Deployment dex...[Exists] ✔ Configure API server ✔ Create /etc/ca-certificates directory...[Created] ✔ Copy Dex certificate into Minikube...[OK] ✔ Configure Minikube API server...[OK] ✔ Wait for Minikube API server...[OK] ✔ Start following Eclipse Che installation logs...[OK] ❯ Deploy Eclipse Che operator ❯ Install Dev Workspace operator ✔ Create Namespace devworkspace-controller...[Exists] ✖ Create Dev Workspace operator resources → issuer.cert-manager.io/devworkspace-controller-selfsigned-issuer unchanged Wait for Dev Workspace operator ready Create ServiceAccount che-operator Create RBAC Wait for Cert Manager pods ready Create Certificate che-operator-serving-cert Create Issuer che-operator-selfsigned-issuer Create Service che-operator-service Create CRD checlusters.org.eclipse.che Waiting Create Deployment che-operator Eclipse Che Operator pod bootstrap Create ValidatingWebhookConfiguration org.eclipse.che Create MutatingWebhookConfiguration org.eclipse.che Create CheCluster Custom Resource Error: Command server:deploy failed with the error: Command failed with exit code 1: kubectl apply -f /usr/local/lib/chectl/templates/devworkspace-operator/kubernetes/combined.yaml Error from server (Invalid): error when applying patch: {"spec":{"conversion":{"webhook":{"clientConfig":{"caBundle":"Cg=="}}}},"status":{"acceptedNames":{"kind":"","plural":""},"conditions":[],"storedVersions":[]}} to: Resource: "apiextensions.k8s.io/v1, Resource=customresourcedefinitions", GroupVersionKind: "apiextensions.k8s.io/v1, Kind=CustomResourceDefinition" Name: "devworkspaces.workspace.devfile.io", Namespace: "" for: "/usr/local/lib/chectl/templates/devworkspace-operator/kubernetes/combined.yaml": error when patching "/usr/local/lib/chectl/templates/devworkspace-operator/kubernetes/combined.yaml": CustomResourceDefinition.apiextensions.k8s.io "devworkspaces.workspace.devfile.io" is invalid: spec.conversion.webhookClientConfig.caBundle: Invalid value: []byte{0xa}: unable to load root certificates: unable to parse bytes as PEM block Error from server (Invalid): error when applying patch: {"spec":{"conversion":{"webhook":{"clientConfig":{"caBundle":"Cg=="}}}},"status":{"acceptedNames":{"kind":"","plural":""},"conditions":[],"storedVersions":[]}} to: Resource: "apiextensions.k8s.io/v1, Resource=customresourcedefinitions", GroupVersionKind: "apiextensions.k8s.io/v1, Kind=CustomResourceDefinition" Name: "devworkspacetemplates.workspace.devfile.io", Namespace: "" for: "/usr/local/lib/chectl/templates/devworkspace-operator/kubernetes/combined.yaml": error when patching "/usr/local/lib/chectl/templates/devworkspace-operator/kubernetes/combined.yaml": CustomResourceDefinition.apiextensions.k8s.io "devworkspacetemplates.workspace.devfile.io" is invalid: spec.conversion.webhookClientConfig.caBundle: Invalid value: []byte{0xa}: unable to load root certificates: unable to parse bytes as PEM block customresourcedefinition.apiextensions.k8s.io/devworkspaceoperatorconfigs.controller.devfile.io configured customresourcedefinition.apiextensions.k8s.io/devworkspaceroutings.controller.devfile.io configured serviceaccount/devworkspace-controller-serviceaccount unchanged role.rbac.authorization.k8s.io/devworkspace-controller-leader-election-role unchanged clusterrole.rbac.authorization.k8s.io/devworkspace-controller-edit-workspaces unchanged clusterrole.rbac.authorization.k8s.io/devworkspace-controller-metrics-reader unchanged clusterrole.rbac.authorization.k8s.io/devworkspace-controller-proxy-role unchanged clusterrole.rbac.authorization.k8s.io/devworkspace-controller-role configured clusterrole.rbac.authorization.k8s.io/devworkspace-controller-view-workspaces unchanged rolebinding.rbac.authorization.k8s.io/devworkspace-controller-leader-election-rolebinding unchanged clusterrolebinding.rbac.authorization.k8s.io/devworkspace-controller-proxy-rolebinding unchanged clusterrolebinding.rbac.authorization.k8s.io/devworkspace-controller-rolebinding unchanged service/devworkspace-controller-manager-service unchanged service/devworkspace-controller-metrics unchanged deployment.apps/devworkspace-controller-manager configured certificate.cert-manager.io/devworkspace-controller-serving-cert unchanged issuer.cert-manager.io/devworkspace-controller-selfsigned-issuer unchanged See details: /home/skyworth/.cache/chectl/error.log. Eclipse Che logs: /tmp/chectl-logs/1730961011828. at newError (/usr/local/lib/chectl/lib/utils/utls.js:39:19) at wrapCommandError (/usr/local/lib/chectl/lib/utils/command-utils.js:54:32) at Deploy. (/usr/local/lib/chectl/lib/commands/server/deploy.js:82:65) at Generator.throw () at rejected (/usr/local/lib/chectl/node_modules/tslib/tslib.js:167:69) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) Cause: Error: Command failed with exit code 1: kubectl apply -f /usr/local/lib/chectl/templates/devworkspace-operator/kubernetes/combined.yaml Error from server (Invalid): error when applying patch: {"spec":{"conversion":{"webhook":{"clientConfig":{"caBundle":"Cg=="}}}},"status":{"acceptedNames":{"kind":"","plural":""},"conditions":[],"storedVersions":[]}} to: Resource: "apiextensions.k8s.io/v1, Resource=customresourcedefinitions", GroupVersionKind: "apiextensions.k8s.io/v1, Kind=CustomResourceDefinition" Name: "devworkspaces.workspace.devfile.io", Namespace: "" for: "/usr/local/lib/chectl/templates/devworkspace-operator/kubernetes/combined.yaml": error when patching "/usr/local/lib/chectl/templates/devworkspace-operator/kubernetes/combined.yaml": CustomResourceDefinition.apiextensions.k8s.io "devworkspaces.workspace.devfile.io" is invalid: spec.conversion.webhookClientConfig.caBundle: Invalid value: []byte{0xa}: unable to load root certificates: unable to parse bytes as PEM block Error from server (Invalid): error when applying patch: {"spec":{"conversion":{"webhook":{"clientConfig":{"caBundle":"Cg=="}}}},"status":{"acceptedNames":{"kind":"","plural":""},"conditions":[],"storedVersions":[]}} to: Resource: "apiextensions.k8s.io/v1, Resource=customresourcedefinitions", GroupVersionKind: "apiextensions.k8s.io/v1, Kind=CustomResourceDefinition" Name: "devworkspacetemplates.workspace.devfile.io", Namespace: "" for: "/usr/local/lib/chectl/templates/devworkspace-operator/kubernetes/combined.yaml": error when patching "/usr/local/lib/chectl/templates/devworkspace-operator/kubernetes/combined.yaml": CustomResourceDefinition.apiextensions.k8s.io "devworkspacetemplates.workspace.devfile.io" is invalid: spec.conversion.webhookClientConfig.caBundle: Invalid value: []byte{0xa}: unable to load root certificates: unable to parse bytes as PEM block customresourcedefinition.apiextensions.k8s.io/devworkspaceoperatorconfigs.controller.devfile.io configured customresourcedefinition.apiextensions.k8s.io/devworkspaceroutings.controller.devfile.io configured serviceaccount/devworkspace-controller-serviceaccount unchanged role.rbac.authorization.k8s.io/devworkspace-controller-leader-election-role unchanged clusterrole.rbac.authorization.k8s.io/devworkspace-controller-edit-workspaces unchanged clusterrole.rbac.authorization.k8s.io/devworkspace-controller-metrics-reader unchanged clusterrole.rbac.authorization.k8s.io/devworkspace-controller-proxy-role unchanged clusterrole.rbac.authorization.k8s.io/devworkspace-controller-role configured clusterrole.rbac.authorization.k8s.io/devworkspace-controller-view-workspaces unchanged rolebinding.rbac.authorization.k8s.io/devworkspace-controller-leader-election-rolebinding unchanged clusterrolebinding.rbac.authorization.k8s.io/devworkspace-controller-proxy-rolebinding unchanged clusterrolebinding.rbac.authorization.k8s.io/devworkspace-controller-rolebinding unchanged service/devworkspace-controller-manager-service unchanged service/devworkspace-controller-metrics unchanged deployment.apps/devworkspace-controller-manager configured certificate.cert-manager.io/devworkspace-controller-serving-cert unchanged issuer.cert-manager.io/devworkspace-controller-selfsigned-issuer unchanged at makeError (/usr/local/lib/chectl/node_modules/execa/lib/error.js:60:11) at handlePromise (/usr/local/lib/chectl/node_modules/execa/index.js:118:26) at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

minikube kubectl -- get po -A NAMESPACE NAME READY STATUS RESTARTS AGE cert-manager cert-manager-54f9d599b-mn52s 1/1 Running 4 (4m28s ago) 44h cert-manager cert-manager-cainjector-648f59958c-ws8nk 1/1 Running 6 (4m29s ago) 44h cert-manager cert-manager-webhook-7b845b56cb-k9gdj 1/1 Running 5 (4m29s ago) 44h devworkspace-controller devworkspace-controller-manager-f54dbb6f6-vs55l 2/2 Running 0 3m12s devworkspace-controller devworkspace-webhook-server-7c4b65bdb9-2t9nt 2/2 Running 0 118s devworkspace-controller devworkspace-webhook-server-7c4b65bdb9-kt9lf 2/2 Running 0 2m18s dex dex-7687bb6d68-k68gc 1/1 Running 7 (3m57s ago) 44h ingress-nginx ingress-nginx-admission-create-btgxv 0/1 Completed 0 45h ingress-nginx ingress-nginx-admission-patch-46wbc 0/1 Completed 0 45h ingress-nginx ingress-nginx-controller-857f8876df-dn89f 1/1 Running 4 (4m19s ago) 45h kube-system coredns-d4ddbc888-72f9c 1/1 Running 5 (4m24s ago) 46h kube-system coredns-d4ddbc888-xn45q 1/1 Running 4 (4m24s ago) 46h kube-system etcd-minikube 1/1 Running 6 (4m28s ago) 46h kube-system kube-apiserver-minikube 1/1 Running 3 (4m18s ago) 44h kube-system kube-controller-manager-minikube 1/1 Running 5 (4m29s ago) 46h kube-system kube-proxy-xqvwd 1/1 Running 5 (4m29s ago) 46h kube-system kube-scheduler-minikube 1/1 Running 5 (4m28s ago) 46h kube-system metrics-server-686dff4775-j2dhq 1/1 Running 8 (3m57s ago) 45h kube-system storage-provisioner 1/1 Running 6 (4m29s ago) 46h kubernetes-dashboard dashboard-metrics-scraper-c5db448b4-jdmwx 1/1 Running 4 (4m29s ago) 45h kubernetes-dashboard kubernetes-dashboard-695b96c756-qfrdq 1/1 Running 5 (4m28s ago) 45h

Che version

7.93/ 7.94

Steps to reproduce

chectl server:deploy --platform minikube

Expected behavior

che install success.

Runtime

minikube

Screenshots

No response

Installation method

chectl/latest

Environment

Rocky Linux release 8.10 (Green Obsidian)

Eclipse Che Logs

No response

Additional context

No response

tolusha commented 2 weeks ago

It seems it is impossible to deploy DWO/Che operator on the latest Kubernetes version

tolusha commented 2 weeks ago

@slieer Could reinstall minikube and deploy che one more time? Currently I can't reproduce the issue. Sometimes I have storage is (re)initializing problem. Maybe the latest minikube is not stable.

slieer commented 2 weeks ago

@slieer Could reinstall minikube and deploy che one more time? Currently I can't reproduce the issue. Sometimes I have storage is (re)initializing problem. Maybe the latest minikube is not stable.

Thank you for your response. I'll try again.

slieer commented 4 days ago

kubectl apply -f /usr/local/lib/chectl/templates/devworkspace-operator/kubernetes/combined.yaml

customresourcedefinition.apiextensions.k8s.io/devworkspaceoperatorconfigs.controller.devfile.io configured customresourcedefinition.apiextensions.k8s.io/devworkspaceroutings.controller.devfile.io configured serviceaccount/devworkspace-controller-serviceaccount unchanged role.rbac.authorization.k8s.io/devworkspace-controller-leader-election-role unchanged clusterrole.rbac.authorization.k8s.io/devworkspace-controller-edit-workspaces unchanged clusterrole.rbac.authorization.k8s.io/devworkspace-controller-metrics-reader unchanged clusterrole.rbac.authorization.k8s.io/devworkspace-controller-proxy-role unchanged clusterrole.rbac.authorization.k8s.io/devworkspace-controller-role configured clusterrole.rbac.authorization.k8s.io/devworkspace-controller-view-workspaces unchanged rolebinding.rbac.authorization.k8s.io/devworkspace-controller-leader-election-rolebinding unchanged clusterrolebinding.rbac.authorization.k8s.io/devworkspace-controller-proxy-rolebinding unchanged clusterrolebinding.rbac.authorization.k8s.io/devworkspace-controller-rolebinding unchanged service/devworkspace-controller-manager-service unchanged service/devworkspace-controller-metrics unchanged deployment.apps/devworkspace-controller-manager configured certificate.cert-manager.io/devworkspace-controller-serving-cert unchanged issuer.cert-manager.io/devworkspace-controller-selfsigned-issuer unchanged Error from server (Invalid): error when applying patch: {"spec":{"conversion":{"webhook":{"clientConfig":{"caBundle":"Cg=="}}}},"status":{"acceptedNames":{"kind":"","plural":""},"conditions":[],"storedVersions":[]}} to: Resource: "apiextensions.k8s.io/v1, Resource=customresourcedefinitions", GroupVersionKind: "apiextensions.k8s.io/v1, Kind=CustomResourceDefinition" Name: "devworkspaces.workspace.devfile.io", Namespace: "" for: "/usr/local/lib/chectl/templates/devworkspace-operator/kubernetes/combined.yaml": error when patching "/usr/local/lib/chectl/templates/devworkspace-operator/kubernetes/combined.yaml": CustomResourceDefinition.apiextensions.k8s.io "devworkspaces.workspace.devfile.io" is invalid: spec.conversion.webhookClientConfig.caBundle: Invalid value: []byte{0xa}: unable to load root certificates: unable to parse bytes as PEM block Error from server (Invalid): error when applying patch: {"spec":{"conversion":{"webhook":{"clientConfig":{"caBundle":"Cg=="}}}},"status":{"acceptedNames":{"kind":"","plural":""},"conditions":[],"storedVersions":[]}} to: Resource: "apiextensions.k8s.io/v1, Resource=customresourcedefinitions", GroupVersionKind: "apiextensions.k8s.io/v1, Kind=CustomResourceDefinition" Name: "devworkspacetemplates.workspace.devfile.io", Namespace: "" for: "/usr/local/lib/chectl/templates/devworkspace-operator/kubernetes/combined.yaml": error when patching "/usr/local/lib/chectl/templates/devworkspace-operator/kubernetes/combined.yaml": CustomResourceDefinition.apiextensions.k8s.io "devworkspacetemplates.workspace.devfile.io" is invalid: spec.conversion.webhookClientConfig.caBundle: Invalid value: []byte{0xa}: unable to load root certificates: unable to parse bytes as PEM block

It's still like this. It looks like it's a cert-manager related issue. The current version of cert-manager in CHE is too low. Are there any plans to upgrade to the latest version from cert-manager.io?

tolusha commented 4 days ago

@slieer There is no problem to udpate certmanager to a newer version. Could you try the following on the clean minikube:

oc apply -f  https://github.com/cert-manager/cert-manager/releases/download/v1.8.2/cert-manager.yaml
oc apply -f https://raw.githubusercontent.com/devfile/devworkspace-operator/refs/tags/v0.31.2/deploy/deployment/kubernetes/combined.yaml

slieer commented 3 days ago

@slieer Could reinstall minikube and deploy che one more time? Currently I can't reproduce the issue. Sometimes I have storage is (re)initializing problem. Maybe the latest minikube is not stable.

Thank you for your response. I'll try again.

OK， Thanks. Very thank you for your attention and response.

To address this issue, I think the first step should be to handle the self-signed certificate. However, the optional steps described in the documentation may not be accurate. I will try this method.

https://eclipse.dev/che/docs/stable/administration-guide/configuring-che-with-self-signed-certificate/

tolusha commented 2 days ago

@slieer Please let me know if you need any help

eclipse-che / che