search

eclipse-che / che

Kubernetes based Cloud Development Environments for Enterprise Teams
http://eclipse.org/che
Eclipse Public License 2.0
6.96k stars 1.19k forks source link

After docker desktop is restarted Che is not available #16134

Closed ke-liu closed 3 years ago

ke-liu commented 4 years ago

Installed the eclipse Che recently, unfortunately server crashes and keeps losing my workspace every day…

Describe the bug

Installed via chectl with docker desktop (latest version 2.2.0.3) on my Mac pro 10.14.6 (with v1.15.5 Kubernetes: tried both minikube and docker-desktop). The eclipse Che installed successful, created workspace with no issue. But each day I log back in my Mac, I lost the URL connection: http://che-che.x.x.x.x.nip.io/ Try to stop and start server:

chectl server:start --platform=docker-desktop

✖ Check Eclipse Che server status

… › Error: E_CHECK_CHE_STATUS_FAIL - Failed to check Eclipse Che status (URL: http://che-che.x.x.x.x.nip.io). E_CHE_API_NO_RESPONSE - Endpoint: http://che-che.x.x.x.x.nip.io/api/system/state - Error message: › connect EHOSTUNREACH x.x.x.x:80 - Local (x.x.y.y:50622)

Nothing works since in terms of chectl commands, restarting dockers and Kubernetes would not help at all. I have to reset Kubernetes cluster, then start the Che server agin.. Lost all workspace and work.

Che version

chectl/7.8.0 darwin-x64 node-v10.18.1

Steps to reproduce

log off the mac for certain period of time (12 hours), or maybe switching networks. Each night I finish work while che is working, log back on the second day, che server stops running.

Expected behavior

Che server should always up and running , service should be restarted once detected crash/failure for example when mac reboots... there should be mechanism to volume mount the workspaces to host machine (mac), incase user has to reset kubernetes cluster, the workspaces remain.

Runtime

kubectl version Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"06ad960bfd03b39c8310aaf92d1e7c12ce618213", GitTreeState:"clean", BuildDate:"2020-02-13T18:06:54Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.5", GitCommit:"20c265fef0741dd71a66480e35bd69f18351daea", GitTreeState:"clean", BuildDate:"2019-10-15T19:07:57Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}

minikube version minikube version: v1.7.2 commit: 50d543b5fcb0e1c0d7c27b1398a9a9790df09dfb

docker version Client: Docker Engine - Community Version: 19.03.5 API version: 1.40 Go version: go1.12.12 Git commit: 633a0ea Built: Wed Nov 13 07:22:34 2019 OS/Arch: darwin/amd64 Experimental: false

Server: Docker Engine - Community Engine: Version: 19.03.5 API version: 1.40 (minimum version 1.12) Go version: go1.12.12 Git commit: 633a0ea Built: Wed Nov 13 07:29:19 2019 OS/Arch: linux/amd64 Experimental: false containerd: Version: v1.2.10 GitCommit: b34a5c8af56e510852c35414db4c1f4fa6172339 runc: Version: 1.0.0-rc8+dev GitCommit: 3e425f80a8c931f88e6d94a8c831b9d5aa481657 docker-init: Version: 0.18.0 GitCommit: fec3683 Kubernetes: Version: v1.15.5 StackAPI: v1beta2

Screenshots

Screen Shot 2020-02-25 at 10 47 11 AM

Screen Shot 2020-02-25 at 10 47 37 AM Screen Shot 2020-02-25 at 10 49 31 AM

Installation method

chectl

Environment

macOS: 10.14.6

l0rd commented 4 years ago

@ke-liu you should not lose your data when k8s get stopped/restarted. We need to figure out what's happening here.

You have provided chectl server:start output and that's good but would you mind providing the output of the following commands as well: kubectl --all-namespaces=true get pod,ing kubectl -n che-che logs -f pod/che-<pod-suffix> kubectl -n che-che describe pod/che-<pod-suffix>

ke-liu commented 4 years ago

Thank you for looking this for me, here are the outputs when server failed to start:

kubectl --all-namespaces=true get pod,ing

NAMESPACE       NAME                                            READY   STATUS    RESTARTS   AGE
che             pod/che-6c8c55bcf5-q9pw9                        1/1     Running   2          4d5h
che             pod/devfile-registry-78675f7854-jdxv6           1/1     Running   0          4d5h
che             pod/plugin-registry-bfcf5644b-nvgpb             1/1     Running   2          4d5h
docker          pod/compose-7b7c5cbbcc-5p2vp                    1/1     Running   0          4d6h
docker          pod/compose-api-dbbf7c5db-x8dgp                 1/1     Running   0          4d6h
ingress-nginx   pod/nginx-ingress-controller-688987f6c9-8nthf   1/1     Running   1          4d5h
kube-system     pod/coredns-5c98db65d4-7xkxs                    1/1     Running   3          4d6h
kube-system     pod/coredns-5c98db65d4-hqmwm                    1/1     Running   3          4d6h
kube-system     pod/etcd-docker-desktop                         1/1     Running   0          4d6h
kube-system     pod/kube-apiserver-docker-desktop               1/1     Running   0          4d6h
kube-system     pod/kube-controller-manager-docker-desktop      1/1     Running   0          4d6h
kube-system     pod/kube-proxy-n474b                            1/1     Running   0          4d6h
kube-system     pod/kube-scheduler-docker-desktop               1/1     Running   1          4d6h

NAMESPACE   NAME                                  HOSTS                                      ADDRESS     PORTS   AGE
che         ingress.extensions/che-ingress        che-che.161.44.255.0.nip.io                localhost   80      4d5h
che         ingress.extensions/devfile-registry   devfile-registry-che.161.44.255.0.nip.io   localhost   80      4d5h
che         ingress.extensions/plugin-registry    plugin-registry-che.161.44.255.0.nip.io    localhost   80      4d5h

===========================================================

kubectl -n che logs pod/che-6c8c55bcf5-q9pw9

Using embedded assembly.
Listening for transport dt_socket at address: 8000
24-Feb-2020 15:04:21.227 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version name:   Apache Tomcat/8.5.46
24-Feb-2020 15:04:21.235 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built:          Sep 16 2019 18:16:19 UTC
24-Feb-2020 15:04:21.236 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version number: 8.5.46.0
24-Feb-2020 15:04:21.236 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name:               Linux
24-Feb-2020 15:04:21.236 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version:            4.19.76-linuxkit
24-Feb-2020 15:04:21.237 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture:          amd64
24-Feb-2020 15:04:21.238 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home:             /usr/local/openjdk-8
24-Feb-2020 15:04:21.238 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version:           1.8.0_232-b09
24-Feb-2020 15:04:21.238 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor:            Oracle Corporation
24-Feb-2020 15:04:21.238 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE:         /home/user/eclipse-che/tomcat
24-Feb-2020 15:04:21.239 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME:         /home/user/eclipse-che/tomcat
24-Feb-2020 15:04:21.241 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.config.file=/home/user/eclipse-che//tomcat/conf/logging.properties
24-Feb-2020 15:04:21.241 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
24-Feb-2020 15:04:21.242 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:MaxRAMFraction=2
24-Feb-2020 15:04:21.243 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:+UseParallelGC
24-Feb-2020 15:04:21.244 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:MinHeapFreeRatio=10
24-Feb-2020 15:04:21.244 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:MaxHeapFreeRatio=20
24-Feb-2020 15:04:21.245 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:GCTimeRatio=4
24-Feb-2020 15:04:21.246 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:AdaptiveSizePolicyWeight=90
24-Feb-2020 15:04:21.246 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:+UnlockExperimentalVMOptions
24-Feb-2020 15:04:21.247 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:+UseCGroupMemoryLimitForHeap
24-Feb-2020 15:04:21.248 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dsun.zip.disableMemoryMapping=true
24-Feb-2020 15:04:21.248 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xms20m
24-Feb-2020 15:04:21.249 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dche.docker.network=bridge
24-Feb-2020 15:04:21.250 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dport.http=8080
24-Feb-2020 15:04:21.250 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dche.home=/home/user/eclipse-che/
24-Feb-2020 15:04:21.250 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dche.logs.dir=/data/logs
24-Feb-2020 15:04:21.251 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dche.logs.level=INFO
24-Feb-2020 15:04:21.261 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djuli-logback.configurationFile=file:/home/user/eclipse-che//tomcat/conf/tomcat-logger.xml
24-Feb-2020 15:04:21.261 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djdk.tls.ephemeralDHKeySize=2048
24-Feb-2020 15:04:21.261 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
24-Feb-2020 15:04:21.262 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dorg.apache.catalina.security.SecurityListener.UMASK=0022
24-Feb-2020 15:04:21.262 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -agentlib:jdwp=transport=dt_socket,address=8000,server=y,suspend=n
24-Feb-2020 15:04:21.262 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dche.local.conf.dir=/etc/conf
24-Feb-2020 15:04:21.263 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dignore.endorsed.dirs=
24-Feb-2020 15:04:21.264 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.base=/home/user/eclipse-che//tomcat
24-Feb-2020 15:04:21.266 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.home=/home/user/eclipse-che//tomcat
24-Feb-2020 15:04:21.267 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.io.tmpdir=/home/user/eclipse-che//tomcat/temp
24-Feb-2020 15:04:21.440 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8080"]
24-Feb-2020 15:04:21.467 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read
24-Feb-2020 15:04:21.485 INFO [main] org.apache.catalina.startup.Catalina.load Initialization processed in 1459 ms
24-Feb-2020 15:04:21.523 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
24-Feb-2020 15:04:21.523 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Apache Tomcat/8.5.46
24-Feb-2020 15:04:21.597 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/home/user/eclipse-che/tomcat/webapps/docs.war]
24-Feb-2020 15:04:22.899 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/home/user/eclipse-che/tomcat/webapps/docs.war] has finished in [1,301] ms
24-Feb-2020 15:04:22.900 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/home/user/eclipse-che/tomcat/webapps/workspace-loader.war]
24-Feb-2020 15:04:22.952 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/home/user/eclipse-che/tomcat/webapps/workspace-loader.war] has finished in [51] ms
24-Feb-2020 15:04:22.953 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/home/user/eclipse-che/tomcat/webapps/dashboard.war]
24-Feb-2020 15:04:23.279 INFO [localhost-startStop-1] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
24-Feb-2020 15:04:23.287 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/home/user/eclipse-che/tomcat/webapps/dashboard.war] has finished in [334] ms
24-Feb-2020 15:04:23.287 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/home/user/eclipse-che/tomcat/webapps/swagger.war]
24-Feb-2020 15:04:23.338 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/home/user/eclipse-che/tomcat/webapps/swagger.war] has finished in [51] ms
24-Feb-2020 15:04:23.341 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/home/user/eclipse-che/tomcat/webapps/api.war]
2020-02-24 15:04:31,690[ost-startStop-1]  [INFO ] [.e.c.c.d.JNDIDataSourceFactory 63]   - This=org.eclipse.che.core.db.h2.H2SQLJndiDataSourceFactory@dccc5ef obj=ResourceRef[className=javax.sql.DataSource,factoryClassLocation=null,factoryClassName=org.apache.naming.factory.ResourceFactory,{type=scope,content=Shareable},{type=auth,content=Container},{type=singleton,content=true},{type=factory,content=org.eclipse.che.api.CommonJndiDataSourceFactory}] name=che Context=org.apache.naming.NamingContext@762b1938 environment={}
2020-02-24 15:04:39,659[ost-startStop-1]  [INFO ] [o.f.c.i.d.DbSupportFactory 44]       - Database: jdbc:h2:che (H2 1.4)
2020-02-24 15:04:39,820[ost-startStop-1]  [INFO ] [o.f.c.i.util.VersionPrinter 44]      - Flyway 4.2.0 by Boxfuse
2020-02-24 15:04:39,825[ost-startStop-1]  [INFO ] [o.f.c.i.d.DbSupportFactory 44]       - Database: jdbc:h2:che (H2 1.4)
2020-02-24 15:04:39,929[ost-startStop-1]  [INFO ] [i.f.CustomSqlMigrationResolver 158]  - Searching for sql scripts in locations [classpath:che-schema]
2020-02-24 15:04:40,120[ost-startStop-1]  [INFO ] [o.f.c.i.command.DbValidate 44]       - Successfully validated 55 migrations (execution time 00:00.198s)
2020-02-24 15:04:40,169[ost-startStop-1]  [INFO ] [o.f.c.i.command.DbMigrate 44]        - Current version of schema "PUBLIC": 7.6.0.1
2020-02-24 15:04:40,180[ost-startStop-1]  [INFO ] [o.f.c.i.command.DbMigrate 44]        - Schema "PUBLIC" is up to date. No migration necessary.
2020-02-24 15:04:43,910[ost-startStop-1]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 174]   - Configured factories for environments: '[kubernetes, no-environment]'
2020-02-24 15:04:43,912[ost-startStop-1]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 175]   - Registered infrastructure 'kubernetes'
2020-02-24 15:04:44,045[ost-startStop-1]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 661]   - Infrastructure is tracking 0 active runtimes
2020-02-24 15:04:44,083[ost-startStop-1]  [INFO ] [o.e.c.a.c.u.ApiInfoLogInformer 36]   - Eclipse Che Api Core: Build info '7.8.0' scmRevision '833d97eea12e87458908d44325277c57193f55ab' implementationVersion '7.8.0'
24-Feb-2020 15:04:44.830 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/home/user/eclipse-che/tomcat/webapps/api.war] has finished in [21,489] ms
24-Feb-2020 15:04:44.833 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/home/user/eclipse-che/tomcat/webapps/ROOT.war]
24-Feb-2020 15:04:48.579 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/home/user/eclipse-che/tomcat/webapps/ROOT.war] has finished in [3,746] ms
24-Feb-2020 15:04:48.613 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"]
24-Feb-2020 15:04:48.691 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 27239 ms

===========================================================

kubectl -n che describe pod/che-6c8c55bcf5-q9pw9

Name:           che-6c8c55bcf5-q9pw9
Namespace:      che
Priority:       0
Node:           docker-desktop/192.168.65.3
Start Time:     Fri, 21 Feb 2020 11:58:16 -0500
Labels:         app=che
                component=che
                pod-template-hash=6c8c55bcf5
Annotations:    <none>
Status:         Running
IP:             10.1.0.90
IPs:            <none>
Controlled By:  ReplicaSet/che-6c8c55bcf5
Init Containers:
  fmp-volume-permission:
    Container ID:  docker://d0e20cf42e81ddf7e53d48094bf758fb88119d6aaf299e37265d52f24e9a91e0
    Image:         busybox
    Image ID:      docker-pullable://busybox@sha256:6915be4043561d64e0ab0f8f098dc2ac48e077fe23f488ac24b665166898115a
    Port:          <none>
    Host Port:     <none>
    Command:
      chmod
      777
      /data
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Mon, 24 Feb 2020 10:04:17 -0500
      Finished:     Mon, 24 Feb 2020 10:04:17 -0500
    Ready:          True
    Restart Count:  1
    Environment:    <none>
    Mounts:
      /data from che-data-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from che-token-kz46z (ro)
Containers:
  che:
    Container ID:   docker://c8ebc9395f2857d5cfac58fc43082112d6f4d2b69f0a490b6fdf586d5b6eb747
    Image:          quay.io/eclipse/che-server:7.8.0
    Image ID:       docker-pullable://quay.io/eclipse/che-server@sha256:7e582353177633e4d554463aee513c4260e7afdd6733284831d6af8f01bc7e88
    Ports:          8080/TCP, 8000/TCP, 8888/TCP, 8087/TCP
    Host Ports:     0/TCP, 0/TCP, 0/TCP, 0/TCP
    State:          Running
      Started:      Mon, 24 Feb 2020 10:04:19 -0500
    Last State:     Terminated
      Reason:       Error
      Exit Code:    137
      Started:      Mon, 24 Feb 2020 10:03:44 -0500
      Finished:     Mon, 24 Feb 2020 10:04:14 -0500
    Ready:          True
    Restart Count:  2
    Limits:
      memory:  600Mi
    Requests:
      memory:   256Mi
    Liveness:   http-get http://:8080/api/system/state delay=120s timeout=10s period=10s #success=1 #failure=3
    Readiness:  http-get http://:8080/api/system/state delay=15s timeout=60s period=10s #success=1 #failure=3
    Environment Variables from:
      che  ConfigMap  Optional: false
    Environment:
      OPENSHIFT_KUBE_PING_NAMESPACE:  che (v1:metadata.namespace)
    Mounts:
      /data from che-data-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from che-token-kz46z (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  che-data-volume:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  che-data-volume
    ReadOnly:   false
  che-token-kz46z:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  che-token-kz46z
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>
l0rd commented 4 years ago

What happens if you try to open http://che-che.161.44.255.0.nip.io in your browser?

l0rd commented 4 years ago

I have changed the title because there is no evidence of a che-server crash.

ke-liu commented 4 years ago

fine with the title changing.. but the only reason I restart the docker desktop is because the che stops working. if i go: http://che-che.161.44.255.0.nip.io, browser will try load the page and eventually timeout, then retry again, and again...

ke-liu commented 4 years ago

i suspect server crash is because every che command i try indicating something wrong witht he server, for example:

chectl workspace:start --devfile ~/mysite_dcos/Devfile.yaml ✔ Retrieving Eclipse Che Server URL...http://che-che.161.44.255.0.nip.io ✖ Verify if Eclipse Che server is running → E_SRV_NOT_RUNNING - Eclipse Che Server is not available by http://che-che.161.44.255.0.nip.io Create workspace from Devfile ~/mysite_dcos/Devfile.yaml › Error: E_SRV_NOT_RUNNING - Eclipse Che Server is not available by http://che-che.161.44.255.0.nip.io

l0rd commented 4 years ago

Right, chectl is not able to connect to the che-server. We need to figure out why.

I was actually wrong about the server not crashing. The che-server got restarted 3 times. And last time it stopped with Exit Code: 137 that means "Out of Memory"...

How much memory are you setting for Docker Desktop?

image

ke-liu commented 4 years ago

earlier i gave 2G memory, the workspace not even be able to be created... then i increased to 8G, workspaces are creating fine, but the server issues happened... Screen Shot 2020-02-25 at 6 56 10 PM

tolusha commented 4 years ago

@ke-liu Could you run pls

$ kubectl get pv  -n che
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM               STORAGECLASS   REASON   AGE
pvc-0db53bda-b0e2-43d0-9d32-59842d401ea0   1Gi        RWO            Delete           Bound    che/postgres-data   standard                44m

and then

$ kubectl get pv pvc-0db53bda-b0e2-43d0-9d32-59842d401ea0 -n che -o=yaml
ke-liu commented 4 years ago

kubectl get pv -n che

NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                     STORAGECLASS   REASON   AGE
pvc-030c931c-660f-41a9-88c0-a03ed710ed21   1Gi        RWO            Delete           Bound    che/che-data-volume       hostpath                4d22h
pvc-fb9108c8-9952-418d-9bea-efe4f311e722   1Gi        RWO            Delete           Bound    che/claim-che-workspace   hostpath                4d22h

=====================================================

kubectl get pv pvc-030c931c-660f-41a9-88c0-a03ed710ed21 -n che -o=yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    docker.io/hostpath: /Users/kel2/.docker/Volumes/che-data-volume/pvc-030c931c-660f-41a9-88c0-a03ed710ed21
    pv.kubernetes.io/provisioned-by: docker.io/hostpath
  creationTimestamp: "2020-02-21T16:58:16Z"
  finalizers:
  - kubernetes.io/pv-protection
  name: pvc-030c931c-660f-41a9-88c0-a03ed710ed21
  resourceVersion: "2441"
  selfLink: /api/v1/persistentvolumes/pvc-030c931c-660f-41a9-88c0-a03ed710ed21
  uid: 9d531aaf-b84f-42ae-934e-9afbdc78dac0
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 1Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: che-data-volume
    namespace: che
    resourceVersion: "2424"
    uid: 030c931c-660f-41a9-88c0-a03ed710ed21
  hostPath:
    path: /Users/kel2/.docker/Volumes/che-data-volume/pvc-030c931c-660f-41a9-88c0-a03ed710ed21
    type: ""
  persistentVolumeReclaimPolicy: Delete
  storageClassName: hostpath
  volumeMode: Filesystem
status:
  phase: Bound

===================================================================

kubectl get pv pvc-fb9108c8-9952-418d-9bea-efe4f311e722 -n che -o=yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    docker.io/hostpath: /Users/kel2/.docker/Volumes/claim-che-workspace/pvc-fb9108c8-9952-418d-9bea-efe4f311e722
    pv.kubernetes.io/provisioned-by: docker.io/hostpath
  creationTimestamp: "2020-02-21T17:05:55Z"
  finalizers:
  - kubernetes.io/pv-protection
  name: pvc-fb9108c8-9952-418d-9bea-efe4f311e722
  resourceVersion: "3164"
  selfLink: /api/v1/persistentvolumes/pvc-fb9108c8-9952-418d-9bea-efe4f311e722
  uid: 4ce65d08-6dea-4ad3-9e67-824933307543
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 1Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: claim-che-workspace
    namespace: che
    resourceVersion: "3158"
    uid: fb9108c8-9952-418d-9bea-efe4f311e722
  hostPath:
    path: /Users/kel2/.docker/Volumes/claim-che-workspace/pvc-fb9108c8-9952-418d-9bea-efe4f311e722
    type: ""
  persistentVolumeReclaimPolicy: Delete
  storageClassName: hostpath
  volumeMode: Filesystem
status:
  phase: Bound
sblumenstock-doubleSlash commented 4 years ago

Same problem that I have. Annoying problem when working with che.

sblumenstock-doubleSlash commented 3 years ago

Is there any new information on this topic? The problem is still with me today.

tsmaeder commented 3 years ago

I have had the same problem recently with docker-desktop for Windows. The problem seems to be that the IP of the cluster can change when either docker or the whole machine is restarted. Che remembers the hostname for the cluster when being installed. When the IP address changes, the hostname used via nip.io changes as well. So che is still running, but it is no longer reacheable.

tsmaeder commented 3 years ago

I was able to get che back to a semblance of working by doing 

kubectl edit checluster -n <che namespace, on docker for windows it's 'che'>

and replacing the old nip.io address with the new one. However, the self-signed certificates for the various services no longer seem to work: going to the various services with a browser (keykloak, plugin/devfile registry) seems to fix the problem. Unfortunately, https://github.com/eclipse/che/issues/18582 prevented me from verifying that the above workaround works correctly.

che-bot commented 3 years ago

Issues go stale after 180 days of inactivity. lifecycle/stale issues rot after an additional 7 days of inactivity and eventually close.

Mark the issue as fresh with /remove-lifecycle stale in a new comment.

If this issue is safe to close now please do so.

Moderators: Add lifecycle/frozen label to avoid stale mode.