search

eclipse-che / che

Kubernetes based Cloud Development Environments for Enterprise Teams
http://eclipse.org/che
Eclipse Public License 2.0
6.96k stars 1.19k forks source link

Chectl fails for "Eclipse Che pod bootstrap: Downloading Images -> "helm:latest" #19078

Closed frankae closed 3 years ago

frankae commented 3 years ago

Describe the bug

ISSUE in all of the below scenarios is the same. The "helm:latest" can't be pulled. Before Friday, 12th Feb 2021 that had worked. That was reported in different flavors at different times. Here something changed

While doing some tests to explore the opportunity to use che in our company I had made some installations on different systems with individuell setups for

both with --classic approach. After that the procedure was the same for all tests.

  1. Download chectl as described in the installation sections.

  2. Update chectl to the latest version.

  3. Deploy che-server via chectl server:deploy in the following variants ai. docker-desktop: chectl server:deploy -p docker-desktop -i helm aii. docker-desktop: chectl server:deploy -p docker-desktop -i helm -m bi. UBUNTU 20.04.1 LTS (snap microk8s): chectl server:deploy -p microk8s -i helm bii. UBUNTU 20.04.1 LTS (snap microk8s): chectl server:deploy -p microk8s -i helm -m ci. UBUNTU 20.04.1 LTS (snap juju, lxd-based charmed kubernetes): chectl server:deploy -p k8s -i helm cii. UBUNTU 20.04.1 LTS (snap juju, lxd-based charmed kubernetes): chectl server:deploy -p k8s -i helm -m

  4. Output from chectl server:deploy (microk8s, which is the identical for all tests: chectl server:deploy -p microk8s -i helm › Current Kubernetes context: 'loaded-context' › Installer type is set to: 'operator' Enable CLI usage data to be sent to Red Hat online services. [y/n]: y ✔ Verify Kubernetes API...OK ✔ 👀 Looking for an already existing Eclipse Che instance ✔ Verify if Eclipse Che is deployed into namespace "eclipse-che"...it is not ✔ ✈️ MicroK8s preflight checklist ✔ Verify if kubectl is installed ✔ Verify if microk8s is installed ✔ Verify if microk8s is running ↓ Start microk8s [skipped] → MicroK8s is already running. ✔ Check Kubernetes version: Found v1.20.1-34+e7db93d188d0d1. ✔ Verify if microk8s ingress and storage addons is enabled ✔ Enable microk8s ingress addon ↓ Enable microk8s storage addon [skipped] → Storage addon is already enabled. ✔ Retrieving microk8s IP and domain for ingress URLs...172.31.3.62.nip.io. ↓ Check if cluster accessible [skipped] ✔ Following Eclipse Che logs ✔ Start following Operator logs...done ✔ Start following Eclipse Che server logs...done ✔ Start following Postgres logs...done ✔ Start following Keycloak logs...done ✔ Start following Plugin registry logs...done ✔ Start following Devfile registry logs...done ✔ Start following namespace events...done › Warning: Consider using the more reliable 'OLM' installer when deploying a stable › release of Eclipse Che (--installer=olm). ✔ 🏃 Running the Eclipse Che operator ✔ Create Namespace (eclipse-che)...Done. ✔ Create ServiceAccount che-operator in namespace eclipse-che...done. ✔ Read Roles and Bindings...done. ✔ Creating Roles and Bindings...done. ✔ Create CRD checlusters.org.eclipse.che...done. ✔ Waiting 5 seconds for the new Kubernetes resources to get flushed...done. ✔ Create deployment che-operator in namespace eclipse-che...done. ✔ Operator pod bootstrap ✔ Scheduling...done ✔ Downloading images...done ✔ Starting...done ✔ Prepare Eclipse Che cluster CR...Done. ✔ Create the Custom Resource of type checlusters.org.eclipse.che in the namespace ec lipse-che...done. ❯ ✅ Post installation checklist ✔ PostgreSQL pod bootstrap ✔ Scheduling...done ✔ Downloading images...done ✔ Starting...done ✔ Keycloak pod bootstrap ✔ Scheduling...done ✔ Downloading images...done ✔ Starting...done ✔ Devfile registry pod bootstrap ✔ Scheduling...done ✔ Downloading images...done ✔ Starting...done ✔ Plugin registry pod bootstrap ✔ Scheduling...done ✔ Downloading images...done ✔ Starting...done ❯ Eclipse Che pod bootstrap ✔ Scheduling...done ✖ Downloading images...failed → Failed to download image, reason: ImagePullBackOff, message: Back-off pulling … Starting Eclipse Che status check Retrieving Keycloak admin credentials Retrieving Che self-signed CA certificate Prepare post installation output › Error: Error: Failed to download image, reason: ImagePullBackOff, message: › Back-off pulling image "helm:latest". › Command server:deploy failed. Error log: /home/devfrank/.cache/chectl/error.log

  5. Output from evfrank@ubu2004-che-mk:~$ kubectl describe -n eclipse-che pod che-8678f8f794-rs9v9 > /tmp/runbooks_describe_pod.txt

evfrank@ubu2004-che-mk:~$ more /tmp/runbooks_describe_pod.txt**
Name: che-8678f8f794-rs9v9 Namespace: eclipse-che Priority: 0 Node: ubu2004-che-mk/172.31.3.62 Start Time: Mon, 15 Feb 2021 13:53:48 +0000 Labels: app=che app.kubernetes.io/component=che app.kubernetes.io/instance=che app.kubernetes.io/managed-by=che-operator app.kubernetes.io/name=che component=che pod-template-hash=8678f8f794 Annotations: cni.projectcalico.org/podIP: 10.1.25.87/32 cni.projectcalico.org/podIPs: 10.1.25.87/32 Status: Pending IP: 10.1.25.87 IPs: IP: 10.1.25.87 Controlled By: ReplicaSet/che-8678f8f794 Containers: che: Container ID:
Image: helm:latest Image ID:
Ports: 8080/TCP, 8000/TCP, 8888/TCP Host Ports: 0/TCP, 0/TCP, 0/TCP State: Waiting Reason: ImagePullBackOff Ready: False Restart Count: 0 Limits: cpu: 1 memory: 1Gi Requests: cpu: 100m memory: 512Mi Liveness: http-get http://:8080/api/system/state delay=400s timeout=3s period=10s #success=1 #failure=3 Readiness: http-get http://:8080/api/system/state delay=25s timeout=5s period=10s #success=1 #failure=18 Environment Variables from: che ConfigMap Optional: false Environment: CHE_SELFSIGNEDCERT: <set to the key 'ca.crt' in secret 'self-signed-certificate'> Optional: true CHE_GIT_SELFSIGNEDCERT:
CHE_GIT_SELFSIGNEDCERTHOST:
CHE_KEYCLOAK_ADMINPASSWORD: <set to the key 'password' in secret 'che-identity-secret'> Optional: false CHE_KEYCLOAK_ADMIN__USERNAME: <set to the key 'user' in secret 'che-identity-secret'> Optional: false CM_REVISION: 8570,7998 KUBERNETES_NAMESPACE: eclipse-che (v1:metadata.namespace) CHE_JDBC_USERNAME: <set to the key 'user' in secret 'che-postgres-secret'> Optional: false CHE_JDBC_PASSWORD: <set to the key 'password' in secret 'che-postgres-secret'> Optional: false Mounts: /public-certs from che-public-certs (rw) /var/run/secrets/kubernetes.io/serviceaccount from che-token-xzc8m (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: che-public-certs: Type: ConfigMap (a volume populated by a ConfigMap) Name: ca-certs-merged Optional: false che-token-xzc8m: Type: Secret (a volume populated by a Secret) SecretName: che-token-xzc8m Optional: false QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message

Normal Scheduled 60m default-scheduler Successfully assigned eclipse-che/che-8678f8f794-rs9v9 to ubu2004-c he-mk Normal Pulling 59m (x4 over 60m) kubelet Pulling image "helm:latest" Warning Failed 59m (x4 over 60m) kubelet Error: ErrImagePull Warning Failed 59m (x4 over 60m) kubelet Failed to pull image "helm:latest": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/helm:latest": failed to resolve reference "docker.io/library/helm:latest": pul l access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed Warning Failed 25m (x151 over 60m) kubelet Error: ImagePullBackOff Normal BackOff 4m38s (x215 over 60m) kubelet Back-off pulling image "helm:latest"

ISSUE in all scenarios is the same. The "helm:latest" can't be pulled. Before Friday, 12th Feb 2021 that had worked.

Che version

Steps to reproduce

Expected behavior

Runtime

Screenshots

Installation method

Environment

Eclipse Che Logs

Additional context

sparkoo commented 3 years ago

thank you for reporting. Can you please try it with stable version? Just update chectl to latest stable version with chectl update stable should be enough.

frankae commented 3 years ago

Hi, had done a quick test on the Windows10 docker-desktop:

  1. chectl server:delete , and yes
  2. chectl update stable
  3. chectl server:deploy -p docker-desktop -i helm -m

Result: as before. In "Eclipse Che pod bootstrap", "Downloading images ...failed"

sparkoo commented 3 years ago

@frankae oh I see it now. You're using -i helm, but -i is parameter for che-server image. You should use -a helm or --installer=helm. See chectl --help for further details.