Closed huonguyenlt closed 1 month ago
When trying to accss the plugin registry url ( https://che.stengg-devcheworkspaces.com/plugin-registry/v3), the browser redirect me to https://che.stengg-devcheworkspaces.com/v3/ and the page is not working. Is it expected behavior?
That's a bug. I need to investigate
@huonguyenlt
Error: unable to verify the first certificate
Could you try in the terminal of a user workspace:
curl --cacert /tmp/che/secret/ca.crt https://che.stengg-devcheworkspaces.com
openssl verify -verbose -CAfile /tmp/che/secret/ca.crt /tmp/che/secret/ca.crt
When trying to access the plugin registry url
Dashboard doesn't use anymore pluginregistry to fetch editors definitions. That's some redundant that we need to cleanup. In general it is expected
@huonguyenlt
Error: unable to verify the first certificate
Could you try in the terminal of a user workspace:
curl --cacert /tmp/che/secret/ca.crt https://che.stengg-devcheworkspaces.com
openssl verify -verbose -CAfile /tmp/che/secret/ca.crt /tmp/che/secret/ca.crt
@tolusha the cert you mentioned does not exist. Here is the log from my empty workspace
projects $ curl --cacert /tmp/che/secret/ca.crt https://che.stengg-devcheworkspaces.com
curl: (77) error setting certificate verify locations:
CAfile: /tmp/che/secret/ca.crt
CApath: none
projects $ ls /tmp/che/secret/ca.crt
ls: cannot access '/tmp/che/secret/ca.crt': No such file or directory
projects $ ls /tmp/
composer-installer.php node-extra-certificates/ vscode-git-86ded4a1df.sock
containers-user-1234/ podman-run-1234/ vscode-ipc-73c8eb01-9350-43f2-a8ec-b8fe198b3c3f.sock
ks-script-eio8pz76 poststart-stderr.txt vscode-ipc-b6454629-be1a-41b7-9f39-43ddd5a66847.sock
ks-script-ep7jq71d poststart-stdout.txt
I found instead this cert /public-certs/kube-root-ca.crt.ca.crt
in the mount volume of the workspace pod, any chance that the cert you asked for?
Mounts:
/.git-credentials/ from devworkspace-merged-git-credentials (ro)
/checode from claim-devworkspace (rw,path="workspace296409eef9fb4a92/checode")
/config/user/profile from user-profile (ro)
/devworkspace-metadata from workspace-metadata (ro)
/etc/gitconfig from devworkspace-gitconfig (ro,path="gitconfig")
/etc/ssh/dwo_ssh_key from git-ssh-key (ro,path="dwo_ssh_key")
/etc/ssh/dwo_ssh_key.pub from git-ssh-key (ro,path="dwo_ssh_key.pub")
/etc/ssh/ssh_config from git-ssh-key (ro,path="ssh_config")
/projects from claim-devworkspace (rw,path="workspace296409eef9fb4a92/projects")
/public-certs from che-trusted-ca-certs (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-xgxlc (ro)
projects $ curl --cacert /public-certs/kube-root-ca.crt.ca.crt https://che.stengg-devcheworkspaces.com
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html
Not sure if this info relevant but I use the self signed cert for the che domain (https://eclipse.dev/che/docs/stable/administration-guide/configuring-che-with-self-signed-certificate/). So I the workspace should use this cert rather than some temp cert, right? I named the secret che.tls3 and provide to checluster as below. Is there anywhere else I have to add this cert?
spec:
networking:
domain: che.stengg-devcheworkspaces.com
tlsSecretName: che.tls3
So I the workspace should use this cert rather than some temp cert, right?
Yes, but for some reason the certificate is not propagated into workspace. Let's do it manually:
HOST=che.stengg-devcheworkspaces.com
NAMESPACE=$(kubectl get checluster -A -o "jsonpath={.items[0].metadata.namespace}")
CERTS=$(openssl s_client -showcerts -connect $HOST:443 < /dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p')
kubectl create configmap custom-certificate --from-literal registry.crt="${CERTS}" -n $NAMESPACE
kubectl label configmap custom-certificate app.kubernetes.io/component=ca-bundle app.kubernetes.io/part-of=che.eclipse.org -n $NAMESPACE
@tolusha I created the config map as you suggested in the eclipse-che namespace. I see that The certificate is then added to the configmap che-trusted-ca-certs
under user workspace namespace. And the configmap che-trusted-ca-certs
is used in the workspace pod. So the pod should see the certificate. Still the error when installing extension persists.
List of cm and secert use by workspace pod
Volumes:
devworkspace-merged-git-credentials:
Type: Secret (a volume populated by a Secret)
SecretName: devworkspace-merged-git-credentials
Optional: false
devworkspace-gitconfig:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: devworkspace-gitconfig
Optional: false
che-trusted-ca-certs:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: che-trusted-ca-certs
Optional: false
git-ssh-key:
Type: Secret (a volume populated by a Secret)
SecretName: git-ssh-key
Optional: false
user-profile:
Type: Secret (a volume populated by a Secret)
SecretName: user-profile
Optional: false
claim-devworkspace:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: claim-devworkspace
ReadOnly: false
workspace-metadata:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: workspace7e08c73f453d4f8b-metadata
Optional: true
che-gateway:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: workspace7e08c73f453d4f8b-route
Optional: false
kube-api-access-bp6nb:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
che-trusted-ca-certs config map
apiVersion: v1
data:
custom-certificate.registry.crt: |-
-----BEGIN CERTIFICATE-----
MIIGVTCCBT2gAwIBAgITHQAAM0gLriknEHbtxwAAAAAzSDANBgkqhkiG9w0BAQsF
7E4VFyOJphicizPSryPFMqFrEcVvW8mlJuwWXUzTvGHDprkbic/hkqI=
-----END CERTIFICATE-----
kube-root-ca.crt.ca.crt: |
-----BEGIN CERTIFICATE-----
MIIC/jCCAeagAwIBAgIBADANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQDEwprdWJl
mhTlqHd5Jy0DSn7ARQjZzAElrpdpBeYlO29uDmLDVeRNwpDJrcO926CL0NYL71nc
ATw=
-----END CERTIFICATE-----
kind: ConfigMap
metadata:
annotations:
che.eclipse.org/included-configmaps: kube-root-ca.crt-247215612.custom-certificate-256823490
controller.devfile.io/mount-as: file
controller.devfile.io/mount-path: /public-certs
creationTimestamp: "2024-08-26T13:33:53Z"
labels:
app.kubernetes.io/component: user-settings
app.kubernetes.io/name: eclipse-che
app.kubernetes.io/part-of: che.eclipse.org
controller.devfile.io/mount-to-devworkspace: "true"
controller.devfile.io/watch-configmap: "true"
name: che-trusted-ca-certs
namespace: lethienhuong-nguyen-stengg-com-che-0tv1zl
resourceVersion: "256823528"
uid: 0277541c-201a-4578-9c18-4f3fe1ce3eb4
custom-certificate config map
apiVersion: v1
data:
registry.crt: |-
-----BEGIN CERTIFICATE-----
MIIGVTCCBT2gAwIBAgITHQAAM0gLriknEHbtxwAAAAAzSDANBgkqhkiG9w0BAQsF
7E4VFyOJphicizPSryPFMqFrEcVvW8mlJuwWXUzTvGHDprkbic/hkqI=
-----END CERTIFICATE-----
kind: ConfigMap
metadata:
creationTimestamp: "2024-09-11T08:56:32Z"
labels:
app.kubernetes.io/component: ca-bundle
app.kubernetes.io/part-of: che.eclipse.org
name: custom-certificate
namespace: eclipse-che
resourceVersion: "256823490"
uid: 4fc8976e-829f-41fc-ba37-ea7b3b95e16c
error message
2024-09-11 17:14:19.444 [warning] [eclipse-che.port]: View container 'endpoints' does not exist and all views registered to it will be added to 'Explorer'.
2024-09-11 17:14:19.969 [info] Updating additional builtin extensions cache
2024-09-11 17:14:21.350 [error] [Extension Host] Unable to create telemetry client: "DEVWORKSPACE_TELEMETRY_BACKEND_PORT" is not set.
2024-09-11 17:15:03.682 [error] Error: unable to verify the first certificate
at TLSSocket.onConnectSecure (node:_tls_wrap:1674:34)
at TLSSocket.emit (node:events:518:28)
at TLSSocket._finishInit (node:_tls_wrap:1085:8)
at ssl.onhandshakedone (node:_tls_wrap:871:12)
2024-09-11 17:15:03.683 [error] unable to verify the first certificate: Error: unable to verify the first certificate
at TLSSocket.onConnectSecure (node:_tls_wrap:1674:34)
at TLSSocket.emit (node:events:518:28)
at TLSSocket._finishInit (node:_tls_wrap:1085:8)
at ssl.onhandshakedone (node:_tls_wrap:871:12)
Could you try in the terminal of a user workspace:
curl --cacert /public-certs/custom-certificate.registry.crt https://che.stengg-devcheworkspaces.com
@vitaliy-guliy Do you happen to know the cause of the error?
@tolusha the problem fixed. It is because the root and intermediate certs is missing in the ca chain cert. Thank alot for your help
Thank you for letting me know. Could provide more details, how did you figure that out? That would be helpful for others to detect the same problem.
@tolusha I tried 2 commands as below. I then checked again the ca chain cert and saw that the intermediate and root certificates are not included. So I created the new ca chain certs with all the certs included, and the I created new config map with required labels. As soon as the cert was added to the workspace pod, I could install extension with no error $ openssl verify -verbose -CAfile /public-certs/custom-certificate.registry.crt /public-certs/custom-certificate.registry.crt
error 20 at 0 depth lookup: unable to get local issuer certificate
error /public-certs/custom-certificate.registry.crt: verification failed
$ openssl s_client -connect che.stengg-devcheworkspaces.com:443 -showcerts
depth=0 C = SG, ST = SG, L = Singapore, O = Singapore Technologies Engineering Ltd, OU = Corporate, CN = che.stengg-devcheworkspaces.com, emailAddress = appsvc.admin@stengg.com
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 C = SG, ST = SG, L = Singapore, O = Singapore Technologies Engineering Ltd, OU = Corporate, CN = che.stengg-devcheworkspaces.com, emailAddress = appsvc.admin@stengg.com
verify error:num=21:unable to verify the first certificate
verify return:1
depth=0 C = SG, ST = SG, L = Singapore, O = Singapore Technologies Engineering Ltd, OU = Corporate, CN = che.stengg-devcheworkspaces.com, emailAddress = appsvc.admin@stengg.com
verify return:1
---
---
SSL handshake has read 2185 bytes and written 409 bytes
Verification error: unable to verify the first certificate
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 21 (unable to verify the first certificate)
---
Describe the bug
After setup the che cluster to use the embedded Open VSX registry instance in the plugin-registry, I can see the list of available extensions. However, when trying to install them, I get the error
Che version
7.89
Steps to reproduce
config checluster to use the embedded open vsx
avaible extenstions shows
click the install button to install any extension, then got the certificate error log
Expected behavior
Should be install successfully
Runtime
other (please specify in additional context)
Screenshots
Installation method
other (please specify in additional context)
Environment
Amazon
Eclipse Che Logs
Additional context
runtime: eks installation method: helm (che-operator)