Closed mmeliani closed 5 years ago
cc @eivantsov
logs from che pod :
Bootstrapping RuntimeIdentityImpl{workspaceId='workspacenlnwwi6d474ihust', envName='default', ownerId='47f31d28-d997-4354-9584-0dab05f6cb59'}:dev-machine. Creating folder for bootstrapper
2019-02-18 10:05:35,558[ineSharedPool-0] [DEBUG] [w.i.k.b.KubernetesBootstrapper 167] - Bootstrapping RuntimeIdentityImpl{workspaceId='workspacenlnwwi6d474ihust', envName='default', ownerId='47f31d28-d997-4354-9584-0dab05f6cb59'}:dev-machine. Downloading bootstrapper binary
2019-02-18 10:05:37,629[ineSharedPool-0] [DEBUG] [w.i.k.b.KubernetesBootstrapper 187] - Bootstrapping RuntimeIdentityImpl{workspaceId='workspacenlnwwi6d474ihust', envName='default', ownerId='47f31d28-d997-4354-9584-0dab05f6cb59'}:dev-machine. Creating config file
2019-02-18 10:05:39,239[ineSharedPool-0] [DEBUG] [w.i.k.b.KubernetesBootstrapper 113] - Bootstrapping RuntimeIdentityImpl{workspaceId='workspacenlnwwi6d474ihust', envName='default', ownerId='47f31d28-d997-4354-9584-0dab05f6cb59'}:dev-machine. Launching bootstrapper process
2019-02-18 10:05:39,570[uestProcessor-8] [ERROR] [l.ErrorInstallerLogEventLogger 43] - Installer `org.eclipse.che.exec` error from machine=`dev-machine` owner=`47f31d28-d997-4354-9584-0dab05f6cb59` env=`default` workspace=`workspacenlnwwi6d474ihust` text=`mkdir: cannot create directory ‘/home/user/che’: Permission denied` time=`2019-02-18T10:05:39.565857146Z`
2019-02-18 10:05:40,339[uestProcessor-7] [ERROR] [l.ErrorInstallerLogEventLogger 43] - Installer `org.eclipse.che.exec` error from machine=`dev-machine` owner=`47f31d28-d997-4354-9584-0dab05f6cb59` env=`default` workspace=`workspacenlnwwi6d474ihust` text=`tar: Child returned status 1` time=`2019-02-18T10:05:40.336409355Z`
2019-02-18 10:05:40,349[estProcessor-10] [ERROR] [l.ErrorInstallerLogEventLogger 43] - Installer `org.eclipse.che.exec` error from machine=`dev-machine` owner=`47f31d28-d997-4354-9584-0dab05f6cb59` env=`default` workspace=`workspacenlnwwi6d474ihust` text=`/bin/sh: 244: /home/user/che/exec-agent/che-exec-agent: not found` time=`2019-02-18T10:05:40.336821017Z`
2019-02-18 10:05:40,349[uestProcessor-8] [ERROR] [l.ErrorInstallerLogEventLogger 43] - Installer `org.eclipse.che.exec` error from machine=`dev-machine` owner=`47f31d28-d997-4354-9584-0dab05f6cb59` env=`default` workspace=`workspacenlnwwi6d474ihust` text=`gzip: stdin: unexpected end of file` time=`2019-02-18T10:05:40.336104079Z`
2019-02-18 10:05:40,350[uestProcessor-9] [ERROR] [l.ErrorInstallerLogEventLogger 43] - Installer `org.eclipse.che.exec` error from machine=`dev-machine` owner=`47f31d28-d997-4354-9584-0dab05f6cb59` env=`default` workspace=`workspacenlnwwi6d474ihust` text=`tar: Error is not recoverable: exiting now` time=`2019-02-18T10:05:40.336427731Z`
2019-02-18 10:05:40,351[aceSharedPool-5] [WARN ] [.i.k.KubernetesInternalRuntime 249] - Failed to start Kubernetes runtime of workspace workspacenlnwwi6d474ihust. Cause: Installation 'org.eclipse.che.exec' failed, script exit code is 127
2019-02-18 10:05:40,656[aceSharedPool-5] [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 417] - Workspace 'user:wksp-f2cd' with id 'workspacenlnwwi6d474ihust' start failed
tls:
this is the secret i'm using for che ingress endpoint , and i'm only giving ca.crt in the CHE_SELFSIGNEDCERT , should the workspace pod be aware of those tls certs in the secret name or not ? @sleshchenko i saw ur commit about this en entrypoint, how can i do this ?
@mmeliani
Che Server automatically creates the cert secret for workspace pods if CHE_SELF__SIGNED__CERT
is set. See https://github.com/eclipse/che/blob/master/infrastructures/kubernetes/src/main/java/org/eclipse/che/workspace/infrastructure/kubernetes/provision/CertificateProvisioner.java
And in logs you shared I can see that cert file is bound
Certificate File /tmp/che/secret/ca.crt will be used for binaries downloading
It's needed to find a way to download exec-agent
binaries from workspace container manually and check what's wrong with binaries/CheServer-WorkspacePod connection even after certificates using.
An idea that I have - you can modify installers script and silent option for curl command. Will try and provide you detailed instructions.
@mmeliani I've created a PR where I added -f
and -S
options for curl. It can help to understand what happened during curl command execution in your case.
Not to wait my PR merged, you are able to do the same updates on your Che installation, you can use swagger to update installers scripts quite easly(It's available on by the following URL ${CHE_HOST}/swagger/
). Not that your changes will be overridden after Che Server restart. The first exception in your case occured in org.eclipse.che.exec
, so you can update only this installer.
@sleshchenko did not understand how to update installers scripts using the api? how that possible ? could u please show an example how to update it ?
@mmeliani
{YOUR_CHE_HOST}/swagger/#!/installer/getInstaller
org.eclipse.che.exec
key for getting exec installer JSON.curl -s
with curl -sSf
{YOUR_CHE_HOST}/swagger/#!/installer/update
and use updated JSON as request body.@mmeliani New error message that I see
{"jsonrpc":"2.0","method":"installer/log","params":{"machineName":"dev-machine","runtimeId":{"workspaceId":"workspacegtpqri3p4awwtuhl","envName":"default","ownerId":"47f31d28-d997-4354-9584-0dab05f6cb59"},"time":"2019-02-19T15:13:17.712260179Z","text":"curl: (37) Couldn't open file /home/user/che/exec-agent-linux_amd64.tar.gz","installer":"org.eclipse.che.exec","stream":"STDERR"}}
curl: (37) Couldn't open file /home/user/che/exec-agent-linux_amd64.tar.gz
But it does not help us to find a cause. Actually, we don't see an original error that happens during binaries downloading because installer script is too complicated and I missed that, in fact, there is another curl that does downloading, and that one we updated just help unarchive binaries.
You can update installers with the following changes
It's installers content with the following changes applied https://github.com/eclipse/che/pull/12704
i investigated further and was a permission denied so always set security context to 1000 in order to start correctly workspaces Thanks for you support will close the ticket
Description
I started che chart using che-server:latest ( same for nightly ) with tls enabled and self signed certificate, initially che-server pod doesn't come up and got error :
Caused by: java.lang.RuntimeException: Exception while retrieving OpenId configuration from endpoint: https://keycloak.shared.devlab.test.vlab.com/auth/realms/che/.well-known/openid-configuration
i managed to solve that by creating a generic secret that wrap ca.crt and add appropriate env variable CHE_SELFSIGNEDCERT in the deployments and che-server pod started correctly, but i got antoher error while creating the workspace and it's probably related to certs also, So what about the secret i'm using for ingress ? should che be aware of it also ? where am I going wrong ?
Diagnostics