Workspace Error - Failed to Download Plugin

[Les-Linux]

Hello,

I deployed a clean Openshift 4.9.11 instance (single node) with AD LDAP integration. Following that, I deployed the Che Operator and Cluster Instance v.7.41.2. Once I login to che with oAuth and create a workspace, I am presented with the following error(s)

failed to download plugin from https://download.jboss.org/jbosstools/static/jdt.ls/stable/java-0.63.0-2222.vsix: Get https://download.jboss.org/jbosstools/static/jdt.ls/stable/java-0.63.0-2222.vsix: x509: certificate is valid for *.apps.k8s.mydomain, not download.jboss.org

failed to download plugin from https://open-vsx.org/api/vscjava/vscode-java-test/0.28.1/file/vscjava.vscode-java-test-0.28.1.vsix: Get https://open-vsx.org/api/vscjava/vscode-java-test/0.28.1/file/vscjava.vscode-java-test-0.28.1.vsix: x509: certificate is valid for *.apps.k8s.mydomain not open-vsx.org

Back-off restarting failed container

Back-off restarting failed container

Pulling from quay.io is working as expected, based on the log output.

This is a new installation, both for Openshift and Che. I did not make any changes to the che instance in any form and not sure why this error is being thrown. The only other error I could find beside the workspace log output, is attached

.

[azatsarynnyy]

@ibuziuk if I remember correctly, plugin-broker is your team's area

[Les-Linux]

I am not using a Proxy either, in case you need to know.

[Les-Linux]

If you require further details (setup, network etc) do not hesitate to reach out to me.

[Les-Linux]

Would anybody be available to troubleshoot this matter?

[tolusha]

failed to download plugin from

You can see that plubin broker is configured with Self signed certificate (/tmp/che/secret/ca.crt) and CA bundle certificate path (/public-certs). These public certs are mounted from workspace<...>.ca-certs config map which must contain DigiCert Global Root CA which is used to access https://download.jboss.org. Could you check it?

[tolusha]

You can find workspace<...>.ca-certs in a user's namespace when workspace is starting.

[tolusha]

If you have some issues finding that configmap, you can create any configmap to get those certificates. After that OpenShift injects certificates into this configmap.

kind: ConfigMap
apiVersion: v1
metadata:
  name: ca-certs
  namespace: default
  labels:
    config.openshift.io/inject-trusted-cabundle: 'true'

[Les-Linux]

Hi Anatolii, thanks very much for the info. I will check it out immediately and let you know the outcome.

[Les-Linux]

Anatolli, this is the output when I look at my workspaceqsp9g5zpy4dikhew.ca-certs configmap

Name: workspaceqsp9g5zpy4dikhew.ca-certs Namespace: lfleming-che Labels: che.original_name=ca-certs che.workspace_id=workspaceqsp9g5zpy4dikhew config.openshift.io/inject-trusted-cabundle=true Annotations: che.eclipse.org/included-configmaps:

-- LIsted Certs in ConfigMap -- DigiCert Assured ID Root CA DigiCert Assured ID Root G2 DigiCert Assured ID Root G3 DigiCert Global Root CA DigiCert Global Root G2 DigiCert Global Root G3 DigiCert High Assurance EV Root CA DigiCert Trusted Root G4

[Les-Linux]

Digicert Global Root CA seems fine when I decode Certificate Information: Common Name: DigiCert Global Root CA Organization: DigiCert Inc Organization Unit: www.digicert.com Country: US Valid From: November 9, 2006 Valid To: November 9, 2031 Issuer: DigiCert Global Root CA, DigiCert Inc Serial Number: 083be056904246b1a1756ac95991c74a

[tolusha]

ok. Certificates exists and it correct. Could you check if they are mounted in /public-certs folder of plugin folder then?

[Les-Linux]

output of the che pod

the only difference I noticed is the group id being 1000670000 and not root like the rest. Group Details: 1000670000:x:1000670000:0:1000670000 user:/:/sbin/nologin

dr-xr-xr-x. 13 root root 0 Jan 11 15:29 sys drwxrwsrwx. 3 root 1000670000 59 Jan 11 15:52 public-certs drwxr-xr-x. 1 root root 18 Jan 11 15:53 etc

Pod: che-75d78cbf8b-4564g Directory: /public-certs lrwxrwxrwx. 1 root 1000670000 31 Jan 11 15:52 ..data -> ..2022_01_11_15_52_49.475254648 drwxr-sr-x. 2 root 1000670000 6 Jan 11 15:52 ..2022_01_11_15_52_49.475254648 drwxrwsrwx. 3 root 1000670000 59 Jan 11 15:52 . dr-xr-xr-x. 1 root root 83 Jan 11 15:53 ..

Mounts: overlay on / type overlay (rw,relatime,context="system_u:object_r:container_file_t:s0:c10,c26",lowerdir=/var/lib/containers/storage/overlay/l/JDXVXJJEEZJVBVHLNE45WO7SJT:/var/lib/containers/storage/overlay/l/UYGWWBPU5S525EJMO5T7TKTGIT:/var/lib/containers/storage/overlay/l/XTEJZPI5ZZKWYLS5RCYZRJHIWI:/var/lib/containers/storage/overlay/l/CRUGLGIDPMIY6LCSV54AJ4FDM2:/var/lib/containers/storage/overlay/l/3FGRMZ4SLABTDPMYW27TXERQ43:/var/lib/containers/storage/overlay/l/UECGMNBBGRS22HXK6VH7VFUF73:/var/lib/containers/storage/overlay/l/3YDVZBKSUK6PRGVMH6A3UH3LGJ:/var/lib/containers/storage/overlay/l/JWRCFT3OMDRJIDVBUE3U5SSBUQ,upperdir=/var/lib/containers/storage/overlay/1146d82685151a995b48cb14520bb232d1dd3c2b85c4601b5be91568902e2867/diff,workdir=/var/lib/containers/storage/overlay/1146d82685151a995b48cb14520bb232d1dd3c2b85c4601b5be91568902e2867/work) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev type tmpfs (rw,nosuid,context="system_u:object_r:container_file_t:s0:c10,c26",size=65536k,mode=755) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,context="system_u:object_r:container_file_t:s0:c10,c26",gid=5,mode=620,ptmxmode=666) mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime,seclabel) sysfs on /sys type sysfs (ro,nosuid,nodev,noexec,relatime,seclabel) tmpfs on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,relatime,context="system_u:object_r:container_file_t:s0:c10,c26",mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (ro,nosuid,nodev,noexec,relatime,seclabel,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) cgroup on /sys/fs/cgroup/hugetlb type cgroup (ro,nosuid,nodev,noexec,relatime,seclabel,hugetlb) cgroup on /sys/fs/cgroup/freezer type cgroup (ro,nosuid,nodev,noexec,relatime,seclabel,freezer) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (ro,nosuid,nodev,noexec,relatime,seclabel,net_cls,net_prio) cgroup on /sys/fs/cgroup/cpuset type cgroup (ro,nosuid,nodev,noexec,relatime,seclabel,cpuset) cgroup on /sys/fs/cgroup/rdma type cgroup (ro,nosuid,nodev,noexec,relatime,seclabel,rdma) cgroup on /sys/fs/cgroup/pids type cgroup (ro,nosuid,nodev,noexec,relatime,seclabel,pids) cgroup on /sys/fs/cgroup/memory type cgroup (ro,nosuid,nodev,noexec,relatime,seclabel,memory) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (ro,nosuid,nodev,noexec,relatime,seclabel,cpu,cpuacct) cgroup on /sys/fs/cgroup/devices type cgroup (ro,nosuid,nodev,noexec,relatime,seclabel,devices) cgroup on /sys/fs/cgroup/blkio type cgroup (ro,nosuid,nodev,noexec,relatime,seclabel,blkio) cgroup on /sys/fs/cgroup/perf_event type cgroup (ro,nosuid,nodev,noexec,relatime,seclabel,perf_event) shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,context="system_u:object_r:container_file_t:s0:c10,c26",size=65536k) tmpfs on /etc/resolv.conf type tmpfs (rw,nosuid,nodev,noexec,seclabel,mode=755) tmpfs on /etc/hostname type tmpfs (rw,nosuid,nodev,seclabel,mode=755) tmpfs on /etc/passwd type tmpfs (rw,nosuid,nodev,noexec,seclabel,mode=755) /dev/sda4 on /public-certs type xfs (ro,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota) /dev/sda4 on /etc/hosts type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota) /dev/sda4 on /dev/termination-log type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota) tmpfs on /run/secrets type tmpfs (rw,nosuid,nodev,seclabel,mode=755) tmpfs on /run/secrets/kubernetes.io/serviceaccount type tmpfs (ro,relatime,seclabel,size=1048576k) proc on /proc/bus type proc (ro,nosuid,nodev,noexec,relatime) proc on /proc/fs type proc (ro,nosuid,nodev,noexec,relatime) proc on /proc/irq type proc (ro,nosuid,nodev,noexec,relatime) proc on /proc/sys type proc (ro,nosuid,nodev,noexec,relatime) proc on /proc/sysrq-trigger type proc (ro,nosuid,nodev,noexec,relatime) tmpfs on /proc/acpi type tmpfs (ro,relatime,context="system_u:object_r:container_file_t:s0:c10,c26") tmpfs on /proc/kcore type tmpfs (rw,nosuid,context="system_u:object_r:container_file_t:s0:c10,c26",size=65536k,mode=755) tmpfs on /proc/keys type tmpfs (rw,nosuid,context="system_u:object_r:container_file_t:s0:c10,c26",size=65536k,mode=755) tmpfs on /proc/timer_list type tmpfs (rw,nosuid,context="system_u:object_r:container_file_t:s0:c10,c26",size=65536k,mode=755) tmpfs on /proc/sched_debug type tmpfs (rw,nosuid,context="system_u:object_r:container_file_t:s0:c10,c26",size=65536k,mode=755) tmpfs on /proc/scsi type tmpfs (ro,relatime,context="system_u:object_r:container_file_t:s0:c10,c26") tmpfs on /sys/firmware type tmpfs (ro,relatime,context="system_u:object_r:container_file_t:s0:c10,c26")

[tolusha]

Does /public-certs is empty?

[Les-Linux]

Hello @tolusha ,

Yes it would seem so. See image for greater detail .

[Les-Linux]

I connected to the Che Pod, just want to ensure that was the correct Pod to query?

[Les-Linux]

@tolusha if you need me to pull further logs or clarify anything for you, don't hesitate to ping me.

[tolusha]

Unfortunately not che pod log. che-plugin-broker pod appears for a couple of seconds in user's namespace to download plugins

[Les-Linux]

the che pod was the correct source for the certificate folder?

[tolusha]

yes, /public-certs in che pod can be empty

[Les-Linux]

do you need me to spin up the workspace again, connect to that pod and see if there is a certs folder mounted in that pod?

[tolusha]

yes, that's would be nice

[Les-Linux]

will try to grab that info

[tolusha]

@Les-Linux Could you try a new Eclipse Che version 7.42.0 ? It has a different architecture [1] and does not contain plugin broker

[1] https://www.eclipse.org/che/docs/next/administration-guide/architecture-overview-with-devworkspace/

[Les-Linux]

Hi @tolusha , I had noticed that there was an update for Openshift 4.9.13 and Che 7.42.0 and applied both this morning. The workspace now bootstraps and the pet-clinic project is pulled from GIT :). Unfortunately however, when I try to run/debug the app, it throws the error 'the debug session type "java" is not supported', furthermore the launch.json schema has an issue with the property hostName - see attached image.

[tolusha]

Could you have a look pls? @svor

[Les-Linux]

@svor if you require logs etc. just let me know what you need and I will pull the info. This is a vanilla deployment so no changes made by me :).

[svor]

it seems java-plugin wasn't installed, do you see some errors in java-plugin container?

[Les-Linux]

@svor is it the che-plugin-registry you are referring to ?

quay.io/che-incubator/configbump@sha256: 1 quay.io/eclipse/che--centos--mysql-57-centos7:latest 1 quay.io/eclipse/che--centos--postgresql-13-centos7@sha256: 2 quay.io/eclipse/che--traefik@sha256 1 quay.io/eclipse/che-dashboard@sha256 1 quay.io/eclipse/che-devfile-registry@sha256: 1 quay.io/eclipse/che-machine-exec:7.42.0 1 quay.io/eclipse/che-operator@sha256 1 quay.io/eclipse/che-plugin-registry@sha256: 1 quay.io/eclipse/che-server@sha256: 1 quay.io/eclipse/che-theia:7.42.0

[svor]

@Les-Linux have you updated your previous version of che to 7.42.0 or you have installed it from the scratch?

[Les-Linux]

Hey @svor , this is a new from scratch deployed instance.

[svor]

How did you deploy it, if it was chectl - which version? Also it would be nice if you provide logs from the containers from the workspace pod

[Les-Linux]

I deployed the instance from open shift via the che operator. Is there any particular log events output you require?

[Les-Linux]

logs.tar.gz

@svor , please find all container logs attached.

[svor]

@Les-Linux thank you for providing logs I've tried to deploy che via che operator 7.42.0 on OCP 4.9 and I haven't reproduce the problem. From your logs I see that plugins were not downloaded because of certificate problem:

downloading https://download.jboss.org/jbosstools/static/jdt.ls/stable/java-0.82.0-369.vsix to /plugins/sidecars/tools for component tools
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (60) SSL certificate problem: self signed certificate in certificate chain
More details here: https://curl.se/docs/sslcerts.html

@tolusha does it say something to you?

[tolusha]

That's what we saw in pluing broker for 7.41.x version ^(

[tolusha]

@svor What was the container with an error?

@Les-Linux It would be nice to try curl inside that container. I guess you can try to curl first in theia conatiner curl --verbose https://download.jboss.org/jbosstools/static/jdt.ls/stable/java-0.82.0-369.vsix

[svor]

What was the container with an error?

it's from remote-runtime-injector init continer

[Les-Linux]

hey @svor, I noticed the issue initially around debugging and when I checked plugin's I could see the error I included in this thread. Let me do a curl next and post the response.

[Les-Linux]

curl-tools.log curl-theia-ide.log curl-che-pod.log

[Les-Linux]

remote-runtime-injector

@svor , do you need me to do something specific with remote-runtime-injector ?

[Les-Linux]

from the theia-ide container, curl would seem to be working

[Les-Linux]

plugin-error persists

[svor]

@Les-Linux I see the problems with plugins view on my side as well, could you please register an issue for that

but java-debug and vscode-java plugins should work, it would be nice to see what you have in remote-runtime-injector container (you can find it in the workspace pod):

[tolusha]

@benoitf Could you point out the exact curl command line that is used to donwload plugins It would be nice to try the same command manually inside remote-runtime-injector container by @Les-Linux to figure out the problem

[Les-Linux]

hello @svor, this is the output of the logs for the remote-runtime-injector. there is a error with the log retrieval shown and I am not sure what that is about.

[Les-Linux]

@svor for debugging, I am seeing the following.. workspace3094f5477f284c17-6db45fd7b7-n4dlk-theia-ide.log workspace3094f5477f284c17-6db45fd7b7-n4dlk-tools.log

[benoitf]

@tolusha curl commands are there: https://github.com/eclipse-che/che-theia/blob/main/dockerfiles/theia-endpoint-runtime-binary/src/entrypoint.sh

https://github.com/eclipse-che/che-theia/blob/main/dockerfiles/theia-endpoint-runtime-binary/src/entrypoint.sh#L89

[Les-Linux]

@svor I submitted a github issue https://github.com/eclipse/che/issues/21043

[Les-Linux]

@tolusha i executed the curl from the theia-ide container and attach the output entrypoint.log entrypointL89.log

[svor]

@Les-Linux from your screenshot

i see that plugins were initialized. Now try:

• execute built command
• execute run-debug command
• try to run debugger