Closed svrajasekar closed 4 years ago
hi @svrajasekar
Have you check the installation guide? https://www.eclipse.org/che/docs/che-7/running-che-locally/
Also it will be very helpful if you provide any logs. How to retrieve them: https://www.eclipse.org/che/docs/che-7/retrieving-che-logs/
Pls fill in form to have more clear picture of what is going on.
Describe the bug
Eclipse che server installation on CentOS 7 timesout.
Che version
* [ ] latest * [ ] nightly * [X] other: please specify 7.4.0
Steps to reproduce
- Install Openshift 3.11 on CentOS 7
- Install chectl
- chectl server:start --platform=openshift --domain=127.0.0.1.nip.io --multiuser
Expected behavior
Che server in multiuser mode must be up and running
Runtime
* [ ] kubernetes (include output of `kubectl version`)
Client Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.0+d4cacc0", GitCommit:"d4cacc0", GitTreeState:"clean", BuildDate:"2018-10-10T16:38:01Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.0+d4cacc0", GitCommit:"d4cacc0", GitTreeState:"clean", BuildDate:"2019-12-13T09:16:09Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}
* [ ] Openshift (include output of `oc version`)
oc v3.11.0+0cbc58b kubernetes v1.11.0+d4cacc0 features: Basic-Auth GSSAPI Kerberos SPNEGO
Server https://127.0.0.1:8443 kubernetes v1.11.0+d4cacc0
* [ ] minikube (include output of `minikube version` and `kubectl version`) * [ ] minishift (include output of `minishift version` and `oc version`) * [ ] docker-desktop + K8S (include output of `docker version` and `kubectl version`) * [ ] other: (please specify)
Screenshots
chectl server:start --platform=openshift --domain=127.0.0.1.nip.io --multiuser ✔ Verify Kubernetes API...OK (it's OpenShift) ✔ 👀 Looking for an already existing Che instance ✔ Verify if Che is deployed into namespace "che"...it is not ✔ ✈️ Openshift preflight checklist ✔ Verify if oc is installed...done. ✔ Verify if openshift is running...done. ✔ 🏃 Running the Che Operator ✔ Copying operator resources...done. ✔ Create Namespace (che)...done. ✔ Create ServiceAccount che-operator in namespace che...done. ✔ Create Role che-operator in namespace che...done. ✔ Create ClusterRole che-operator...done. ✔ Create RoleBinding che-operator in namespace che...done. ✔ Create ClusterRoleBinding che-operator...done. ✔ Create CRD checlusters.org.eclipse.che...done. ✔ Waiting 5 seconds for the new Kubernetes resources to get flushed...done. ✔ Create deployment che-operator in namespace che...done. ✔ Create Che Cluster eclipse-che in namespace che...done. ❯ ✅ Post installation checklist ✔ PostgreSQL pod bootstrap ✔ scheduling...done. ✔ downloading images...done. ✔ starting...done. ✔ Keycloak pod bootstrap ✔ scheduling...done. ✔ downloading images...done. ✔ starting...done. ✔ Devfile registry pod bootstrap ✔ scheduling...done. ✔ downloading images...done. ✔ starting...done. ✔ Plugin registry pod bootstrap ✔ scheduling...done. ✔ downloading images...done. ✔ starting...done. ❯ Che pod bootstrap ✔ scheduling...done. ✔ downloading images...done. ✖ starting → ERR_TIMEOUT: Timeout set to pod ready timeout 130000 Retrieving Che Server URL Che status check Error: ERR_TIMEOUT: Timeout set to pod ready timeout 130000 at KubeHelper.
(~/chectl/lib/api/kube.js:615:19) at Generator.next ( ) at fulfilled (~/chectl/node_modules/tslib/tslib.js:107:62) Installation method
* [X ] chectl * [ ] che-operator * [ ] minishift-addon * [ ] I don't know
Environment
* [ ] my computer * [ ] Windows * [X ] Linux * [ ] macOS * [ ] Cloud * [ ] Amazon * [ ] Azure * [ ] GCE * [ ] other (please specify) * [ ] other: please specify
Additional context
Keycloack didn't start properly.
9m 10m 2 keycloak-5cc6777cdd-s5xz2.15e0c571f6b478d1 Pod spec.containers{keycloak} Warning Unhealthy kubelet, localhost Readiness probe failed: Get http://172.17.0.10:8080/auth/js/keycloak.js: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Could you show its logs https://www.eclipse.org/che/docs/che-7/viewing-external-service-logs/#viewing-keycloak-server-logs_viewing-keycloak-logs
it seems fine. What about che server logs? https://www.eclipse.org/che/docs/che-7/viewing-che-server-logs/
Since che server log is pretty huge. I will be sending it in parts.
a wrong route
http://keycloak-che.127.0.0.1.nip.io/auth/realms/che/.well-known/openid-configuration
my guess you installed openshift with a wrong route suffix which is 127.0.0.1
@ibuziuk Do you have any ideas?
Please help me out. I need to desperately set this up. I have no clue as to what to do next. I tried with Minikube, Minishift, Kubernetes and Openshift. Everywhere I faced timeout issue. I am using CentOS 7 VM with 16 gig ram and 4 vcpus.
@svrajasekar If you are running in minishift/minikube, --domain=127.0.0.1.nip.io
is incorrect I think. It should be set to the IP of the cluster itself (you can get this using e.g. minishift ip
). You are currently setting it to localhost, which is probably what causes your issue. I believe you can also use --platform=[minikube|minishift]
to have chectl
handle it automatically.
I am using Openshift 3.11 only and not minishift. Please help me to install Eclipse-che successfully on my host. I even assigned the --public-hostname to the public ip. But no use. Today I tried to install using che-operator. Except che-server pod, all other pods work properly. The che pod times out. Also if you looked at the log that I had sent, you could see a lot exceptions relate to jgroups and ch.qos.logback. It throws illegalstate exception. I don't know to proceed further. I need to set this up asap to my company.
@svrajasekar --domain
should be set to the hostname of your cluster -- where do you navigate to access your cluster via browser, for example?
You can see in the Che server logs
Retrieving OpenId configuration from endpoint: http://keycloak-che.127.0.0.1.nip.io/auth/realms/che/.well-known/openid-configuration
The Che server is trying to communicate with keycloak by pinging localhost, which will obviously not work since keycloak is a separate Pod on the cluster.
E.g. on my cluster:
$ oc get route keycloak
NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD
keycloak keycloak-amisevsk-che-2.<domain-name> keycloak <all> None
In your cluster you should see something like http://keycloak-che.<your-cluster-domain>
, you need to set --domain=<your-cluster-domain>
Step "Preparing OpenShift 3 for installing Che" in the docs covers retrieving your --domain
.
@svrajasekar I'm having the exact same issue.
@amisevsk The domain is set - 127.0.0.1.nip.io
I noticed that when I hit the endpoint from the error logs: http://keycloak-che.127.0.0.1.nip.io/auth/realms/che/.well-known/openid-configuration I get the same error.
However, when I update the URL to point to port 8080 on the endpoint, it does seem to retrieve json data.
I updated the the keycloak service to also forward port 80 to 8080 which enables the original url to return the json. However, the sever still fails.
Yes I have been banging my head on this issue.
Today I brought up Openshift 3 like this oc cluster up --public-hostname=osmaster.maantt.com
It came up successfully
I then created a project named "che"
I executed the below command to install Eclipse/che
chectl server:start --platform=openshift --installer=operator --domain=osmaster.maantt.com --multiuser
Still no luck. It times out.
On top of it, I had a look at the logs, it still shows http://keycloak-che.127.0.0.1.nip.io
I don't know why?
You need to help me out. I am desperate and I have to set it up fast. I have already spent a week on this.
Hi What was the last stable version of eclipse/che that could be launched directly using docker? Today I brought up eclipse/che:6.13.0 successfully. But when I access the application in the browser, it frequently displays "Could not load plugins" error. How to get rid of this?
Atleast this could save my face in the organization. Please help me out.
@svrajasekar could you try
> oc cluster up --public-hostname=osmaster.maantt.com --routing-suffix=osmaster.maantt.com.nip.io
Tried all the combos. Nothing seem to work. Keycloak does not seem to start properly though it's status shows as "running".
Because of the above issue, che server fails to start.
Please let me know any old stable version that can be used straight from Docker instead of Kubernetes or Openshift
All the old versions starting 6.1.0 till 6.19.0 fails because these point to keycloak.openshift which is not in the repository.
I will have to explore alternative to eclipse/che I guess. I have already waster 10 days on this.
I have the following entries in /etc/hosts file
192.168.70.101 che-che.chemaster.maantt.com 192.168.70.101 keycloak-che.chemaster.maantt.com 192.168.70.101 postgres-che.chemaster.maantt.com
I am able to ping using the hostname.
But chectl reports "UnknownHostException" for keycloak-che.chemaster.maantt.com
May I know why?
/cc @sleshchenko
@svrajasekar Let's stick to one platform and try from the very beginning. Please try to provide full information about your environment and commands you executing to get your Che, if we go this way - maybe we will be able to reproduce your case on our side.
Atlast after a great battle that lasted for more than 2 weeks, I could successfully bring up eclipse/che on openshift 3
Now when I access the eclipse che server in the browser, I get the below error:
HTTP Status 401 – Unauthorized Type Status Report Message Authorization token is missing Description The request has not been applied because it lacks valid authentication credentials for the target resource.
Can you please help me solve this?
@svrajasekar Are you able to share HAR? Also please do it from incognito window. Before attaching it to the issue make sure that you do not expose some private info.
@sleshchenko Actually I noticed that Che server is not communicating to Keycloak using public url when installing. I had to use keycloak cluster ip. It then communicates and successfully installs che server. I guess it is a bug. This is the reason I get 401 - Unauthorized error.
When deploying che-server, I used the following syntax
oc new-app -f che-server-template.yaml -p ROUTING_SUFFIX=<PUBLIC-IP>.nip.io -p CHE_KEYCLOAK_AUTHSERVERURL=http://<CLUSTER-IP>:5050/auth -p CHE_MULTIUSER=true -p PROTOCOL=https -p WS_PROTOCOL=wss -p TLS=true; \
Can I get a quick fix for this or workaround?
I tried to use https://keycloak-che.<PUBLIC-IP>/auth in the CHE_KEYCLOAK_AUTHSERVERURL. But it throws UnknownHostException.
Are we talking about minishift?
Could you try https://keycloak-che.{PUBLIC-IP}.nip.io/auth
?
No, it is Openshift 3.11
I tried with https://keycloak-che.{PUBLIC-IP}.nip.io/auth as well. Throws UnknownHostException
It only connects to Cluster IP
Atlast I could successfully install Eclipse Che using external keycloak server.
But when I launch eclipse che in the browser, I don't find any items in the stacks page
Why?
I have been asking for help for more than a week. But no one seems to suggest a positive step to resolve my concerns. Everybody is saying "try this" and "try that".
@svrajasekar Yes, sorry about that. Nobody really understands the installation you have and what is wrong. If you send me an email - we could try to organize a quick meeting if it works for you
Atlast I could successfully install Eclipse Che using external keycloak server.
But when I launch eclipse che in the browser, I don't find any items in the stacks page
Can you help me out?
I have no idea. Maybe HAR could show more
Can't start a new workspace. Complains that it is unable to install plugins due to timeout. I even hosted the local che-plugin-registry built using --offline flag. Still no use. I need your help on this. One issue after another pops up. Don't know when it is going to function properly.
My VM is hosted in AWS. It has 32 gig ram with 8 vCpus.
I installed Openshift 3.11 successfully and could start it properly.
Regards
Hi,
Review for installing oc and chectl: https://www.eclipse.org/che/docs/che-7/installing-che-on-openshift-3-using-the-operator/
See more bug in your case: https://docs.openshift.com/container-platform/3.11/release_notes/ocp_3_11_release_notes.html#ocp-3-11-43
On Sun, Dec 29, 2019, 19:58 Rajasekar S V notifications@github.com wrote:
Can't start a new workspace. Complains that it is unable to install plugins due to timeout. I even hosted the local che-plugin-registry built using --offline flag. Still no use. I need your help on this. One issue after another pops up. Don't know when it is going to function properly.
My VM is hosted in AWS. It has 32 gig ram with 8 vCpus.
I installed Openshift 3.11 successfully and could start it properly.
Regards
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/eclipse/che/issues/15481?email_source=notifications&email_token=AEYAML2QEAX74FZZ57FZ7YTQ3CNHBA5CNFSM4J23M2E2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHY63CI#issuecomment-569503113, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEYAML4YU657GIEG62S4BULQ3CNHBANCNFSM4J23M2EQ .
I don't know how many times to iterate. I followed the instructions stated in the Che Getting Started Guid step by step. Still I faced and still facing issues. Nobody has helped me so far. Everyone says "Try this" and "Try that". But no clear instruction.
There is a bug in chectl tool. When it starts up Eclipse Che, every other pod starts successfully. But not che pod. The reason: Che pod tries to communicate to Keycloak pod using the published route. It will not work. Because no 2 pods within a cluster can communicate with each other over public url.
To address this, I had to install Keycloak server separately outside the cluster in my host and point this url to CHE_KEYLOAK_AUTHSERVERURL of the che pod. This solved the timeout problem,
Also we can's install devfile and plugin registries inside the cluster as it will hit the same problem stated above. Either host it outside the cluster or use the public url. This solved the 2nd problem
The last problem I am facing now is that the workspace is started because of timeout issues. Which means Eclipse Che is trying to download the plugin images. But somewhere it times out. I am yet get the help for this.
Regards
Sorry for the type:
The last problem I am facing now is that the workspace is NOT started because of timeout issues. Which means Eclipse Che is trying to download the plugin images. But somewhere it times out. I am yet get the help for this.
Hi,
Could you wonder permission for get image!?
On Sun, Dec 29, 2019, 23:16 Rajasekar S V notifications@github.com wrote:
Sorry for the type:
The last problem I am facing now is that the workspace is NOT started because of timeout issues. Which means Eclipse Che is trying to download the plugin images. But somewhere it times out. I am yet get the help for this.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/eclipse/che/issues/15481?email_source=notifications&email_token=AEYAML7F27MXERTE7FGBT2TQ3DENDA5CNFSM4J23M2E2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHZC4QA#issuecomment-569519680, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEYAML2Q6GKASWM55HTV5CLQ3DENDANCNFSM4J23M2EQ .
After I launched eclipse che and created a workspace using one of the available stacks, it timed out stating "Plugin installation timed out"
Next time, when the workspace was being started, I looked at the che-init-plugin-broker docker logs:
docker logs f4811284c186 2019/12/31 19:37:42 Broker configuration 2019/12/31 19:37:42 Push endpoint: ws://che-che./api/websocket 2019/12/31 19:37:42 Auth enabled: true 2019/12/31 19:37:42 Runtime ID: 2019/12/31 19:37:42 Workspace: workspacetaac25syghuc0u6y 2019/12/31 19:37:42 Environment: default 2019/12/31 19:37:42 OwnerId: 3d5836ba-ba9a-4a53-8732-c6e5db1b6e25 2019/12/31 19:37:42 Couldn't connect to endpoint 'ws://che-che./api/websocket', due to error 'dial tcp: lookup che-che. on 172.30.0.2:53: server misbehaving'
If you notice the ws:// endpoint, it strips the routing suffix and tries to open the socket. It is where it times out and fails.
Any solution to this problem?
Hi,
some doc to refer for troubleshooting. https://technology.amis.nl/2018/10/28/quick-start-with-eclipse-che-browser-based-ide-running-on-docker/
Dont forget update lastest repo.
On Wed, Jan 1, 2020, 02:45 Rajasekar S V notifications@github.com wrote:
After I launched eclipse che and created a workspace using one of the available stacks, it times out stating "Plugin installation timed out"
Next time, when the workspace was being started, I looked at the docker logs:
docker logs f4811284c186 2019/12/31 19:37:42 Broker configuration 2019/12/31 19:37:42 Push endpoint: ws://che-che./api/websocket 2019/12/31 19:37:42 Auth enabled: true 2019/12/31 19:37:42 Runtime ID: 2019/12/31 19:37:42 Workspace: workspacetaac25syghuc0u6y 2019/12/31 19:37:42 Environment: default 2019/12/31 19:37:42 OwnerId: 3d5836ba-ba9a-4a53-8732-c6e5db1b6e25 2019/12/31 19:37:42 Couldn't connect to endpoint 'ws://che-che./api/websocket', due to error 'dial tcp: lookup che-che. on 172.30.0.2:53: server misbehaving'
If you notice the ws:// endpoint, it strips the routing suffix and tries to open the socket. It is where it times out and fails.
Any solution to this problem?
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/eclipse/che/issues/15481?email_source=notifications&email_token=AEYAMLZIIMUWZRLO55AFZI3Q3OOPBA5CNFSM4J23M2E2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEH4TQSY#issuecomment-569981003, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEYAMLY27CNTGJ5YIXO6IQDQ3OOPBANCNFSM4J23M2EQ .
@IveJ Thanks for the troubleshooting doc. But the steps shown in the doc talks about starting eclipse che in a docker container. The old way. Now that Eclipse Che latest version cannot be started without either Kubernetes or Openshift.
Second, the doc only refers to single user che. I didn't face any problems with Che 6.19.0 starting in single user mode.
The question I asked is why Eclipse Che strips the routing suffix from the exposed route url when trying to open socket connection?
Especially the push endpoint.
Please help me. I am desperate.
Hi,
As you can see on doc che-plugin-docker, this broker runs as an init container on the workspace pod. Its job is to take in a list of plugin identifiers (either references to a plugin in the registry or a link to a plugin meta.yaml) and ensure that the correct .vsix and .theia extenions are downloaded into the /plugins directory, for each plugin requested for the workspace. And more understand when you look at in dockerfile backward in registry. So think of timeout occur for unsufficent extension during download.
https://github.com/eclipse/che-plugin-broker
https://www.eclipse.org/che/docs/che-7/che-workspaces-architecture/
On Wed, Jan 1, 2020, 10:03 Rajasekar S V notifications@github.com wrote:
@IveJ https://github.com/IveJ Thanks for the troubleshooting doc. But the steps shown in the doc talks about starting eclipse che in a docker container. The old way. Now that Eclipse Che latest version cannot be started without either Kubernetes or Openshift.
Second, the doc only refers to single user che. I didn't face any problems with Che 6.19.0 starting in single user mode.
The question I asked is why Eclipse Che strips the routing suffix from the exposed route url when trying to open socket connection?
Especially the push endpoint.
Please help me. I am desperate.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/eclipse/che/issues/15481?email_source=notifications&email_token=AEYAML2LFTDE3MWXYRLCQZ3Q3QBWVA5CNFSM4J23M2E2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEH44FVY#issuecomment-570016471, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEYAML2RQG4ACX7ALEN2TFTQ3QBWVANCNFSM4J23M2EQ .
@IveJ Please note that I have not contributed anything to Eclipse Che development. I have been trying hard to setup for my company. It is going to be 3 weeks and still I have not got any help.
What you are doing is, sending me the link to documentation. What I want is a solution.
Please let me know if you can provide a solution or not?
I can send you whatever the information you want.
I have achieved 90% out of my own initiative. Atleast on this last issue, I hope you can help me out.
Hi Raja,
Follow quick start guide in doc for your enviroment: https://www.eclipse.org/che/docs/che-7/che-quick-starts/
Notice to your project deployment, with ssl option, https or wss. Url as path your project.
On Wed, Jan 1, 2020, 16:16 Rajasekar S V notifications@github.com wrote:
@IveJ https://github.com/IveJ Please note that I have not contributed anything to Eclipse Che development. I have been trying hard to setup for my company. It is going to be 3 weeks and still I have not got any help.
What you are doing is, sending me the link to documentation. What I want is a solution.
Please let me know if you can provide a solution or not?
I can send you whatever the information you want.
I have achieved 90% out of my own initiative. Atleast on this last issue, I hope you can help me out.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/eclipse/che/issues/15481?email_source=notifications&email_token=AEYAML7L4JFGEQOWR7Q4EP3Q3RNNJA5CNFSM4J23M2E2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEH5BDCQ#issuecomment-570036618, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEYAML2QXXB4W5VV7ER37NTQ3RNNJANCNFSM4J23M2EQ .
@svrajasekar
Sorry that you're still stuck at the same place too. This would have been a good tool for my development team but there just doesn't seem to be a fix right now for implementation on Openshift 3. The odd thing for me is that it installs fine on a Minishift deployment in a similar environment.
Are you able to deploy it using Minishift as well?
@svrajasekar I finally got it to run on the Openshift 3.11 cluster.
I'm not sure if this helps or why it worked but it possibly could work for you as well. I've tried several different combinations of the the oc cluster up
command but the last 2 sequences worked.
First I launched the cluster with
oc cluster up --public-hostname=<ip-address> --routing-suffix=<ip-address>.nip.io
This just kept forwarded me to https://127.0.0.1:8443/console/
when I tried to access the console at https://<ip-address>:8443/console/
and never allowed me to successfully login.
Afterwards I took the cluster down with:
oc cluster down
Finally, I relaunched the cluster with:
oc cluster up --public-hostname=<hostname> --routing-suffix=<hostname>.nip.io
This is similar to what you tried previously. However, when I tried to reach the console at https://<hostname>:8443/console/
it forwarded me to https://<ip-address>:8443/console/
and allowed me to successfully login.
From there I was able to setup my users, roles, and the che project and then install the che7 as in the documentation.
chectl server:start --platform=openshift --installer=operator --domain=<OPENSHIFT_APPS_URL>
Again, sounds trivial, but it worked for me.
Also note that chectl has updated since you opened this issue. I just ran:
chectl update stable
Good luck!
I have been trying hard to setup Eclipse-che on Openshift 3 over the last few days. But could not successfully do it.
I tried in Minikube, Minishift, Kubernetes and Openshift. It failed in all.
Che-server pod crashes in multiuser mode. I tried with various versions of Openshift and Eclipse che. But no use.
I am able to start Eclipse-che in single user mode. But in multiuser mode, che-server pod crashes.
I need help desperately.