timeout expired waiting for volumes to attach/mount for pod "eclipse-che"/"postgres-1-56ffs". list of unattached/unmounted volumes=[postgres-data]

mrmStack commented 6 years ago

I try to deploy elipse che multiuser in openshift origin 3.9.0, and I follow the steps of the documents of eclipse che in openshift, but doesnt't work.

https://www.eclipse.org/che/docs/openshift-multi-user.html

after time out 10 min the pod fail and "oc get events" shows this:

Warning FailedMount kubelet, che-nodo.app.es Unable to mount volumes for pod "postgres-1-56ffs_eclipse-che(8e8c10b6-6419-11e8-b655-001a4aab0006)": timeout expired waiting for volumes to attach/mount for pod "eclipse-che"/"postgres-1-56ffs". list of unattached/unmounted volumes=[postgres-data]

PersistentVolumeClaim Normal FailedBinding persistentvolume-controller no persistent volumes available for this claim and no storage class is set

I don't know what's the problem, somebody helps's me...

thanks regards.

mrmStack commented 6 years ago

LAST SEEN   FIRST SEEN   COUNT     NAME                                 KIND                    SUBOBJECT                     TYPE      REASON                        SOURCE                        MESSAGE
11m         11m          1         postgres-1-56ffs.153373ec11199ee3    Pod                                                   Normal    Scheduled                     default-scheduler             Successfully assigned postgres-1-56ffs to che-nodo.app.es
11m         11m          1         postgres-1-56ffs.153373ec229829ce    Pod                                                   Normal    SuccessfulMountVolume         kubelet, che-nodo.app.es    MountVolume.SetUp succeeded for volume "default-token-t7flx" 
2m          9m           4         postgres-1-56ffs.15337408b5502dab    Pod                                                   Warning   FailedMount                   kubelet, che-nodo.app.es    Unable to mount volumes for pod "postgres-1-56ffs_eclipse-che(8e8c10b6-6419-11e8-b655-001a4aab0006)": timeout expired waiting for volumes to attach/mount for pod "eclipse-che"/"postgres-1-56ffs". list of unattached/unmounted volumes=[postgres-data]
27s         27s          1         postgres-1-56ffs.15337487c22f4a56    Pod                                                   Warning   FailedMount                   kubelet, che-node01.pic.es    Unable to mount volumes for pod "postgres-1-56ffs_eclipse-che(8e8c10b6-6419-11e8-b655-001a4aab0006)": timeout expired waiting for volumes to attach/mount for pod "eclipse-che"/"postgres-1-56ffs". list of unattached/unmounted volumes=[postgres-data default-token-t7flx]
11m         11m          1         postgres-1-deploy.153373eb61b04081   Pod                                                   Normal    Scheduled                     default-scheduler             Successfully assigned postgres-1-deploy to che-nodo.app.es
11m         11m          1         postgres-1-deploy.153373eb6911472d   Pod                                                   Normal    SuccessfulMountVolume         kubelet, che-nodo.app.es    MountVolume.SetUp succeeded for volume "deployer-token-pdtcd" 
11m         11m          1         postgres-1-deploy.153373ebeaf37164   Pod                     spec.containers{deployment}   Normal    Pulled                        kubelet, che-nodo.app.es    Container image "openshift/origin-deployer:v3.9.0" already present on machine
11m         11m          1         postgres-1-deploy.153373ebedf2b4da   Pod                     spec.containers{deployment}   Normal    Created                       kubelet, che-nodo.app.es    Created container
11m         11m          1         postgres-1-deploy.153373ebf6a0540d   Pod                     spec.containers{deployment}   Normal    Started                       kubelet, che-nodo.app.es    Started container
11m         11m          1         postgres-1.153373ec10b7d5ec          ReplicationController                                 Normal    SuccessfulCreate              replication-controller        Created pod: postgres-1-56ffs
1m          1m           1         postgres-1.153374780b1c00e3          ReplicationController                                 Normal    SuccessfulDelete              replication-controller        Deleted pod: postgres-1-56ffs
1m          11m          43        postgres-data.153373eb61203c9d       PersistentVolumeClaim                                 Normal    FailedBinding                 persistentvolume-controller   no persistent volumes available for this claim and no storage class is set
11m         11m          1         postgres.153373eb5f576699            DeploymentConfig                                      Normal    DeploymentCreated             deploymentconfig-controller   Created new replication controller "postgres-1" for version 1
1m          1m           1         postgres.1533747809343101            DeploymentConfig                                      Normal    ReplicationControllerScaled   deploymentconfig-controller   Scaled replication controller "postgres-1" from 1 to 0

ghost commented 6 years ago

@mrmStack it looks like you do not have available PVs that can satisfy PVC that Postgres requires:

no persistent volumes available for this claim and no storage class is set

Can you log in as admin and run:

oc get pv

mrmStack commented 6 years ago

hi @eivantsov

[root@che-astro multi]# oc get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pv0001 5Gi RWO Recycle Bound openshift-ansible-service-broker/etcd
captura de pantalla de 2018-05-30 20-23-09 26m

mrmStack commented 6 years ago

I created the pv whith a yaml template like this:

apiVersion: v1 kind: PersistentVolume metadata: name: pv0001 spec: capacity: storage: 5Gi accessModes:

ReadWriteOnce nfs: path: /tmp server: 172.17.0.2 persistentVolumeReclaimPolicy: Recycle

mrmStack commented 6 years ago

thanks so much!

ghost commented 6 years ago

@mrmStack you will need more pvs - https://www.eclipse.org/che/docs/openshift-admin-guide.html#storage-overview

mrmStack commented 6 years ago

Ok, but how I bind the pv whith the pvc in the template? because I will create in eclipse-che project the pvc, but I dont know how I can binding both.

thanks regards

mrmStack commented 6 years ago

would you mind some example for a deploy like this?

ghost commented 6 years ago

@mrmStack you just need more PVs and that's it. Once a PVC is created, OpenShift will automatically bind it to a free PV that can satisfy the claim.

mrmStack commented 6 years ago

ok!, then...I have to create one pvc for each pv?, or I can create one pv in ReadWriteMany and bind all pvc's in one pv?...because the pvc should create inside the project....I'm wrong??

Che server claims 1GB to store logs and initial workspace stacks. PVC mode is RWO. Keycloak needs 2 PVCs, 1Gb each to store logs and Keycloak data. Postgres needs one 1GB PVC to store db.

thanks so much @eivantsov !! I try now your steps!!

ghost commented 6 years ago

@mrmStack you don't need to create any PVCs. IJust make sure you have at least 5 PVs.

Once Che is deployed, a unique PVC strategy is used by default, which means start of a new workspace will require a new PV.

However, you can reconfigure it - https://www.eclipse.org/che/docs/openshift-admin-guide.html#common-pvc-strategy

mrmStack commented 6 years ago

hi @eivantsov ,

first stuff, thanks for alll I get to run postgres & and keycloak, but che server, doesn't work. It's the same method that I used to deploy postgres and keyckIoak??, beacuse in the documents of eclipse che I read this... I try but none.

oc apply -f pvc/che-server-pvc.yaml oc new-app -f che-server-template.yaml -p ROUTING_SUFFIX=${ROUTING_SUFFIX} -p CHE_MULTIUSER=true oc set volume dc/che --add -m /data --name=che-data-volume --claim-name=che-data-volume

thanks regards

ghost commented 6 years ago

@mrmStack yes, this is how you deploy Che server. You may check pod logs

mrmStack commented 6 years ago

che server is the only doesn't works, is necesary that I must put the flag https if my master have address https://che-astrto.app.es:8443 ??

`LAST SEEN   FIRST SEEN   COUNT     NAME                               KIND                    SUBOBJECT                     TYPE      REASON                           SOURCE                        MESSAGE
19m         19m          1         che-1-deploy.1533c2d1a477ec6e      Pod                                                   Normal    Scheduled                        default-scheduler             Successfully assigned che-1-deploy to che-node01.pic.es
19m         19m          1         che-1-deploy.1533c2d1b674e7be      Pod                                                   Normal    SuccessfulMountVolume            kubelet, che-node01.pic.es    MountVolume.SetUp succeeded for volume "deployer-token-vcwtk" 
19m         19m          1         che-1-deploy.1533c2d210807e27      Pod                     spec.containers{deployment}   Normal    Pulled                           kubelet, che-node01.pic.es    Container image "openshift/origin-deployer:v3.9.0" already present on machine
19m         19m          1         che-1-deploy.1533c2d2145169fe      Pod                     spec.containers{deployment}   Normal    Created                          kubelet, che-node01.pic.es    Created container
19m         19m          1         che-1-deploy.1533c2d21cbe351e      Pod                     spec.containers{deployment}   Normal    Started                          kubelet, che-node01.pic.es    Started container
18m         18m          1         che-1-deploy.1533c2d618802ceb      Pod                     spec.containers{deployment}   Normal    Killing                          kubelet, che-node01.pic.es    Killing container with id docker://deployment:Need to kill Pod
19m         19m          1         che-1-xjqn6.1533c2d236415764       Pod                                                   Normal    Scheduled                        default-scheduler             Successfully assigned che-1-xjqn6 to che-node01.pic.es
19m         19m          1         che-1-xjqn6.1533c2d245e80109       Pod                                                   Normal    SuccessfulMountVolume            kubelet, che-node01.pic.es    MountVolume.SetUp succeeded for volume "che-token-dzjrc" 
19m         19m          1         che-1-xjqn6.1533c2d2a5d076df       Pod                     spec.containers{che}          Normal    Pulling                          kubelet, che-node01.pic.es    pulling image "docker.io/eclipse/che-server:nightly"
18m         18m          1         che-1-xjqn6.1533c2d30ce7c5b3       Pod                     spec.containers{che}          Normal    Pulled                           kubelet, che-node01.pic.es    Successfully pulled image "docker.io/eclipse/che-server:nightly"
18m         18m          1         che-1-xjqn6.1533c2d310dcf84a       Pod                     spec.containers{che}          Normal    Created                          kubelet, che-node01.pic.es    Created container
18m         18m          1         che-1-xjqn6.1533c2d318edfd3a       Pod                     spec.containers{che}          Normal    Started                          kubelet, che-node01.pic.es    Started container
18m         18m          1         che-1-xjqn6.1533c2dae3c6db5e       Pod                     spec.containers{che}          Warning   Unhealthy                        kubelet, che-node01.pic.es    Readiness probe failed: Get http://10.129.0.248:8080/api/system/state: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
14m         18m          27        che-1-xjqn6.1533c2dc8d37c441       Pod                     spec.containers{che}          Warning   Unhealthy                        kubelet, che-node01.pic.es    Readiness probe failed: HTTP probe failed with statuscode: 500
17m         18m          3         che-1-xjqn6.1533c2dfad6600d0       Pod                     spec.containers{che}          Warning   Unhealthy                        kubelet, che-node01.pic.es    Liveness probe failed: HTTP probe failed with statuscode: 500
19m         19m          1         che-1.1533c2d235d6a32e             ReplicationController                                 Normal    SuccessfulCreate                 replication-controller        Created pod: che-1-xjqn6
18m         18m          1         che-1.1533c2d6e5588aa0             ReplicationController                                 Normal    SuccessfulDelete                 replication-controller        Deleted pod: che-1-xjqn6
18m         18m          1         che-2-deploy.1533c2d6321914ad      Pod                                                   Normal    Scheduled                        default-scheduler             Successfully assigned che-2-deploy to che-node01.pic.es
18m         18m          1         che-2-deploy.1533c2d640af90cb      Pod                                                   Normal    SuccessfulMountVolume            kubelet, che-node01.pic.es    MountVolume.SetUp succeeded for volume "deployer-token-vcwtk" 
18m         18m          1         che-2-deploy.1533c2d6c0ea7fa1      Pod                     spec.containers{deployment}   Normal    Pulled                           kubelet, che-node01.pic.es    Container image "openshift/origin-deployer:v3.9.0" already present on machine
18m         18m          1         che-2-deploy.1533c2d6c43c6ec6      Pod                     spec.containers{deployment}   Normal    Created                          kubelet, che-node01.pic.es    Created container
18m         18m          1         che-2-deploy.1533c2d6cc87d069      Pod                     spec.containers{deployment}   Normal    Started                          kubelet, che-node01.pic.es    Started container
18m         18m          1         che-2-xh4n6.1533c2d6efc69287       Pod                                                   Normal    Scheduled                        default-scheduler             Successfully assigned che-2-xh4n6 to che-node01.pic.es
18m         18m          1         che-2-xh4n6.1533c2d6fcf8f30b       Pod                                                   Normal    SuccessfulMountVolume            kubelet, che-node01.pic.es    MountVolume.SetUp succeeded for volume "che-token-dzjrc" 
18m         18m          1         che-2-xh4n6.1533c2d6fe378152       Pod                                                   Normal    SuccessfulMountVolume            kubelet, che-node01.pic.es    MountVolume.SetUp succeeded for volume "pv-che" 
18m         18m          1         che-2-xh4n6.1533c2d77e38c055       Pod                     spec.containers{che}          Normal    Pulling                          kubelet, che-node01.pic.es    pulling image "docker.io/eclipse/che-server:nightly"
18m         18m          1         che-2-xh4n6.1533c2d7dd3b415e       Pod                     spec.containers{che}          Normal    Pulled                           kubelet, che-node01.pic.es    Successfully pulled image "docker.io/eclipse/che-server:nightly"
18m         18m          1         che-2-xh4n6.1533c2d7e105f3cc       Pod                     spec.containers{che}          Normal    Created                          kubelet, che-node01.pic.es    Created container
18m         18m          1         che-2-xh4n6.1533c2d7e9bee802       Pod                     spec.containers{che}          Normal    Started                          kubelet, che-node01.pic.es    Started container
17m         17m          1         che-2-xh4n6.1533c2e124f54f94       Pod                     spec.containers{che}          Warning   Unhealthy                        kubelet, che-node01.pic.es    Readiness probe failed: Get http://10.129.0.250:8080/api/system/state: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
17m         17m          3         che-2-xh4n6.1533c2e24f04e5b7       Pod                     spec.containers{che}          Warning   Unhealthy                        kubelet, che-node01.pic.es    Readiness probe failed: HTTP probe failed with statuscode: 500
17m         17m          3         che-2-xh4n6.1533c2e396a525f8       Pod                     spec.containers{che}          Warning   Unhealthy                        kubelet, che-node01.pic.es    Liveness probe failed: HTTP probe failed with statuscode: 500
8m          17m          47        che-2-xh4n6.1533c2e94ad2e84c       Pod                     spec.containers{che}          Warning   Unhealthy                        kubelet, che-node01.pic.es    Readiness probe failed: Get http://10.129.0.250:8080/api/system/state: dial tcp 10.129.0.250:8080: getsockopt: connection refused
18m         18m          1         che-2.1533c2d6ecf7db91             ReplicationController                                 Normal    SuccessfulCreate                 replication-controller        Created pod: che-2-xh4n6
8m          8m           1         che-2.1533c362b1075f76             ReplicationController                                 Normal    SuccessfulDelete                 replication-controller        Deleted pod: che-2-xh4n6
33m         33m          1         che-3-deploy.1533c20ce9460f4a      Pod                                                   Normal    Scheduled                        default-scheduler             Successfully assigned che-3-deploy to che-node01.pic.es
33m         33m          1         che-3-deploy.1533c20cf5e991b0      Pod                                                   Normal    SuccessfulMountVolume            kubelet, che-node01.pic.es    MountVolume.SetUp succeeded for volume "deployer-token-vcwtk" 
33m         33m          1         che-3-deploy.1533c20d71c22c37      Pod                     spec.containers{deployment}   Normal    Pulled                           kubelet, che-node01.pic.es    Container image "openshift/origin-deployer:v3.9.0" already present on machine
33m         33m          1         che-3-deploy.1533c20d7512a09e      Pod                     spec.containers{deployment}   Normal    Created                          kubelet, che-node01.pic.es    Created container
33m         33m          1         che-3-deploy.1533c20d7d5af03d      Pod                     spec.containers{deployment}   Normal    Started                          kubelet, che-node01.pic.es    Started container
32m         32m          1         che-3-deploy.1533c21130a49121      Pod                     spec.containers{deployment}   Normal    Killing                          kubelet, che-node01.pic.es    Killing container with id docker://deployment:Need to kill Pod
33m         33m          1         che-3-mbl64.1533c20d970a4af8       Pod                                                   Normal    Scheduled                        default-scheduler             Successfully assigned che-3-mbl64 to che-node01.pic.es
33m         33m          1         che-3-mbl64.1533c20da9e2dc2a       Pod                                                   Normal    SuccessfulMountVolume            kubelet, che-node01.pic.es    MountVolume.SetUp succeeded for volume "che-token-dzjrc" 
33m         33m          1         che-3-mbl64.1533c20dab641b1d       Pod                                                   Normal    SuccessfulMountVolume            kubelet, che-node01.pic.es    MountVolume.SetUp succeeded for volume "pv-che" 
33m         33m          2         che-3-mbl64.1533c20e4384fd38       Pod                     spec.containers{che}          Normal    Pulling                          kubelet, che-node01.pic.es    pulling image "docker.io/eclipse/che-server:nightly"
33m         33m          2         che-3-mbl64.1533c20ea65d60bd       Pod                     spec.containers{che}          Normal    Pulled                           kubelet, che-node01.pic.es    Successfully pulled image "docker.io/eclipse/che-server:nightly"
33m         33m          2         che-3-mbl64.1533c20ea95ea846       Pod                     spec.containers{che}          Normal    Created                          kubelet, che-node01.pic.es    Created container
33m         33m          2         che-3-mbl64.1533c20eb1e7449d       Pod                     spec.containers{che}          Normal    Started                          kubelet, che-node01.pic.es    Started container
32m         32m          3         che-3-mbl64.1533c20f8c729b58       Pod                     spec.containers{che}          Warning   BackOff                          kubelet, che-node01.pic.es    Back-off restarting failed container
33m         33m          1         che-3.1533c20d96a3978b             ReplicationController                                 Normal    SuccessfulCreate                 replication-controller        Created pod: che-3-mbl64
32m         32m          1         che-3.1533c2112bab48ce             ReplicationController                                 Normal    SuccessfulDelete                 replication-controller        Deleted pod: che-3-mbl64
25m         25m          1         che-data-volume.1533c27e186ea458   PersistentVolumeClaim                                 Warning   ClaimLost                        persistentvolume-controller   Bound claim has lost its PersistentVolume. Data on the volume is lost!
33m         4h           2         che.1533b5d129767743               DeploymentConfig                                      Normal    DeploymentCreated                deploymentconfig-controller   Created new replication controller "che-3" for version 3
19m         19m          1         che.1533c2d1a04f7816               DeploymentConfig                                      Normal    DeploymentCreated                deploymentconfig-controller   Created new replication controller "che-1" for version 1
18m         18m          1         che.1533c2d6112bad36               DeploymentConfig                                      Normal    DeploymentCancelled              deploymentconfig-controller   Cancelled deployment "che-1" superceded by version 2
18m         18m          7         che.1533c2d6112bd554               DeploymentConfig                                      Normal    DeploymentAwaitingCancellation   deploymentconfig-controller   Deployment of version 2 awaiting cancellation of older running deployments
18m         18m          1         che.1533c2d62ffbfe98               DeploymentConfig                                      Normal    RolloutCancelled                 deployer-controller           Rollout for "eclipse-che/che-1" cancelled
18m         18m          1         che.1533c2d630d9c830               DeploymentConfig                                      Normal    DeploymentCreated                deploymentconfig-controller   Created new replication controller "che-2" for version 2
8m          8m           1         che.1533c362afde1602               DeploymentConfig                                      Normal    ReplicationControllerScaled      deploymentconfig-controller   Scaled replication controller "che-2" from 1 to 0

`

thanks for all!!

ghost commented 6 years ago

oc get pods
oc logs $chepod

Or you can go to eclipse che project in https://che-astrto.app.es:8443/, click on Che pod icon, logs tab

mrmStack commented 6 years ago

NAME            DESIRED   CURRENT   READY     AGE
rc/che-1        0         0         0         16m
rc/che-2        0         0         0         16m
rc/keycloak-1   1         1         1         5h
rc/postgres-1   1         1         1         5h


NAME                  READY     STATUS    RESTARTS   AGE
po/che-2-deploy       0/1       Error     0          16m
po/keycloak-1-22cj8   1/1       Running   0          5h
po/postgres-1-dhjqg   1/1       Running   0          5h``

[root@che-astro templates]# oc logs po/che-2-deploy
--> Scaled older deployment che-1 down
--> Scaling che-2 to 1
error: update acceptor rejected che-2: pods for rc 'eclipse-che/che-2' took longer than 600 seconds to become available

mrmStack commented 6 years ago

i write a bash script for a request to server and the server response 200 ok to me...but I dont know if I should use the https flag for install eclipse che if my address master server openshift is this:

https://che-astrto.app.es:8443/

bash script:

url3="http://keycloak-eclipse-che.apps.openshift.che-astro.pic.es";
url4="http://10.129.0.250:8080/api/system/state";
response=$(curl -s -I -L $url4 | grep HTTP);

status=${response#* };
status=${status:0:3};

if [ "$status" != "200" ]
then
    printf "success!!!,  page seems that's working now!!!\n"
fi
url3="http://keycloak-eclipse-che.apps.openshift.che-astro.pic.es";
url4="http://10.129.0.250:8080/api/system/state";
response=$(curl -s -I -L $url4 | grep HTTP);

status=${response#* };
status=${status:0:3};

if [ "$status" != "200" ]
then
    printf "success!!!,  page seems that's working now!!!\n"
fi

ghost commented 6 years ago

@mrmStack just get Che pod logs:

mrmStack commented 6 years ago

I follow this steps:

create che pv:


apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-che
spec:
  capacity:
    storage: 1Gi
  accessModes:
  - ReadWriteOnce
  nfs:
    path: /var/lib/openshift_containers/che
    server: che-astro.pic.es
  persistentVolumeReclaimPolicy: Recycle

oc apply -f pvc/che-server-pvc.yaml

che-server-pvc.yaml:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  labels:
    app: eclipse-che
  name: che-data-volume
spec:
 accessModes:
 - ReadWriteOnce
 resources:
   requests:
     storage: 1Gi
 volumeName: "pv-che"

oc new-app -f che-server-template.yaml -p CHE_MULTIUSER=true

oc set volume dc/che --add -m /data --name=che-data-volume --claim-name=che-data-volume

mrmStack commented 6 years ago

I don't acces with graphical ui, because I am working outside the local network cluster...but the logs of the pod are this:

--> Scaled older deployment che-1 down
--> Scaling che-2 to 1
error: update acceptor rejected che-2: pods for rc 'eclipse-che/che-2' took longer than 600 seconds to become available

the steps that I followed are right?

mrmStack commented 6 years ago

Should I modify some fields of che-server template?

kind: Template
apiVersion: v1
metadata:
  name: che
  annotations:
    description: Che
objects:
- apiVersion: v1
  kind: ServiceAccount
  metadata:
    name: che
- apiVersion: v1
  kind: RoleBinding
  metadata:
    name: che
  roleRef:
    name: admin
  subjects:
  - kind: ServiceAccount
    name: che
- apiVersion: v1
  kind: Service
  metadata:
    name: che-host
  spec:
    ports:
    - name: http
      port: 8080
      protocol: TCP
      targetPort: 8080
    selector:
      app: che
- apiVersion: v1
  kind: Route
  metadata:
    name: che
  spec:
    to:
      kind: Service
      name: che-host
- apiVersion: v1
  kind: DeploymentConfig
  metadata:
    name: che
  spec:
    replicas: 1
    revisionHistoryLimit: 2
    selector:
      app: che
    strategy:
      type: ${STRATEGY}
    template:
      metadata:
        labels:
          app: che
      spec:
        containers:
        - env:
          - name: OPENSHIFT_KUBE_PING_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          - name: NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          - name: CHE_MULTIUSER
            value: "${CHE_MULTIUSER}"
          - name: CHE_HOST
            value: "che-${NAMESPACE}.${ROUTING_SUFFIX}"
          - name: CHE_PORT
            value: "8080"
          - name: CHE_API
            value: "${PROTOCOL}://che-${NAMESPACE}.${ROUTING_SUFFIX}/api"
          - name: CHE_WEBSOCKET_ENDPOINT
            value: "${WS_PROTOCOL}://che-${NAMESPACE}.${ROUTING_SUFFIX}/api/websocket"
          - name: CHE_DEBUG_SERVER
            value: "false"
          - name: CHE_INFRASTRUCTURE_ACTIVE
            value: "openshift"
          - name: CHE_INFRA_KUBERNETES_BOOTSTRAPPER_BINARY__URL
            value: "${PROTOCOL}://che-${NAMESPACE}.${ROUTING_SUFFIX}/agent-binaries/linux_amd64/bootstrapper/bootstrapper"
          - name: CHE_INFRA_KUBERNETES_MACHINE__START__TIMEOUT__MIN
            value: "5"
          - name: CHE_INFRA_KUBERNETES_MASTER__URL
            value: "${CHE_INFRA_KUBERNETES_MASTER__URL}"
          - name: CHE_INFRA_KUBERNETES_OAUTH__TOKEN
            value: "${OPENSHIFT_TOKEN}"
          - name: CHE_INFRA_KUBERNETES_USERNAME
            value: "${OPENSHIFT_USERNAME}"
          - name: CHE_INFRA_KUBERNETES_PASSWORD
            value: "${OPENSHIFT_PASSWORD}"
          - name: CHE_INFRA_OPENSHIFT_PROJECT
            value: "${CHE_INFRA_OPENSHIFT_PROJECT}"
          - name: CHE_INFRA_KUBERNETES_PVC_STRATEGY
            value: "${CHE_INFRA_KUBERNETES_PVC_STRATEGY}"
          - name: CHE_INFRA_KUBERNETES_PVC_PRECREATE__SUBPATHS
            value: "${CHE_INFRA_KUBERNETES_PVC_PRECREATE__SUBPATHS}"
          - name: CHE_INFRA_OPENSHIFT_TLS__ENABLED
            value: "${TLS}"
          - name: CHE_INFRA_KUBERNETES_TRUST__CERTS
            value: "${TLS}"
          - name: CHE_LOCAL_CONF_DIR
            value: "/etc/conf"
          - name: CHE_LOGS_DIR
            value: "/var/lib/openshift_containers/logs"
          - name: CHE_LOG_LEVEL
            value: "INFO"
          - name: CHE_KEYCLOAK_AUTH__SERVER__URL
            value: "${CHE_KEYCLOAK_AUTH__SERVER__URL}"
          - name: CHE_INFRA_OPENSHIFT_OAUTH__IDENTITY__PROVIDER
            value: "${CHE_INFRA_OPENSHIFT_OAUTH__IDENTITY__PROVIDER}"
          - name: CHE_OAUTH_GITHUB_CLIENTID
            value: "${CHE_OAUTH_GITHUB_CLIENTID}"
          - name: CHE_OAUTH_GITHUB_CLIENTSECRET
            value: "${CHE_OAUTH_GITHUB_CLIENTSECRET}"
          - name: CHE_PREDEFINED_STACKS_RELOAD__ON__START
            value: 'true'
          - name: JAVA_OPTS
            value: "-XX:MaxRAMFraction=2 -XX:+UseParallelGC -XX:MinHeapFreeRatio=10
              -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90
              -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -Dsun.zip.disableMemoryMapping=true
              -Xms20m "
          - name: CHE_WORKSPACE_AUTO_START
            value: "false"
          - name: CHE_INFRA_KUBERNETES_PVC_QUANTITY
            value: "${CHE_INFRA_KUBERNETES_PVC_QUANTITY}"
          - name: PROTOCOL
            value: "${PROTOCOL}"
          - name: ROUTING_SUFFIX
            value: "${ROUTING_SUFFIX}"
          image: ${IMAGE_CHE}:${CHE_VERSION}
          imagePullPolicy: "${PULL_POLICY}"
          livenessProbe:
            httpGet:
              path: /api/system/state
              port: 8080
              scheme: HTTP
            initialDelaySeconds: 50
            failureThreshold: 3
            timeoutSeconds: 2
          name: che
          readinessProbe:
            httpGet:
              path: /api/system/state
              port: 8080
              scheme: HTTP
            initialDelaySeconds: 25
            timeoutSeconds: 5
            failureThreshold: 5
          name: che
          ports:
          - containerPort: 8080
            name: http
            protocol: TCP
          - containerPort: 8000
            name: http-debug
          - containerPort: 8888
            name: jgroups-ping
            protocol: TCP
          resources:
            limits:
              memory: 1Gi
            requests:
              memory: 256Mi
          terminationMessagePath: "/dev/termination-log"
          terminationMessagePolicy: File
        restartPolicy: Always
        serviceAccountName: che
        terminationGracePeriodSeconds: 360
    triggers:
    - type: ConfigChange

parameters:
- name: ROUTING_SUFFIX
  displayName: Routing suffix of your OpenShift cluster
  description: This is the suffix appended to routes in your OpenShift cluster <route>-<project>.<routing_suffix>
- name: CHE_VERSION
  displayName: Eclipse Che version
  description: Eclipse Che version which defaults to nightly
  value: nightly
- name: IMAGE_CHE
  displayName: Eclipse Che server image
  description: Che server Docker image. Defaults to docker.io/eclipse/che-server
  value: docker.io/eclipse/che-server
- name: CHE_MULTIUSER
  displayName: Che Multi-user flavor
  description: False i.e. single user by default
  value: 'false'
- name: OPENSHIFT_USERNAME
  displayName: OpenShift Username
  description: OpenShift username that will be used to create workspace objects
- name: OPENSHIFT_PASSWORD
  displayName: OpenShift Password
  description: OpenShift Password that will be used to create workspace objects
- name: OPENSHIFT_TOKEN
  displayName: OpenShift token
  description: OpenShift token to be used to create workspace objects. Must be set if username/password are impossible to use. Do not set if username/pass are provided!
- name: PROTOCOL
  displayName: HTTP protocol
  description: Protocol to be used in Che communications
  value: http
- name: WS_PROTOCOL
  displayName: Websocket protocol
  description: Websocket protocol to be used in Che communications. Defaults to ws. Set to wss only if TLS is true and https protocol is defined
  value: ws
- name: TLS
  displayName: HTTPS support
  description: HTTPS support in Che
  value: 'false'
- name: CHE_INFRA_OPENSHIFT_PROJECT
  displayName: OpenShift namespace to create workspace objects
  description: Che server namespace by default. When empty, each workspace gets own namespace. In this case a privileged sa or credentials/token need to be used.
  value: '${NAMESPACE}'
- name: CHE_INFRA_KUBERNETES_PVC_QUANTITY
  displayName: Default PVC claim
  description: Default PVC claim for storage. Defaults to 1Gi
  value: '1Gi'
- name: CHE_INFRA_KUBERNETES_PVC_STRATEGY
  displayName: PVC strategy
  description: PVC strategy. Unique implies creating PVC per workspace. Common uses one PVC with subpaths in PV. Defaults to unique
  value: 'unique'
- name: CHE_KEYCLOAK_ADMIN_REQUIRE_UPDATE_PASSWORD
  displayName: Admin password update
  description: Force an admin to update password after 1st login. True by default
  value: 'true'
- name: CHE_KEYCLOAK_AUTH__SERVER__URL
  displayName: Identity provider URL
  description: URL of a remote identity provider. Defaults to Keycloak bundled with Che multi user
  value: '${PROTOCOL}://keycloak-${NAMESPACE}.${ROUTING_SUFFIX}/auth'
- name: CHE_INFRA_OPENSHIFT_OAUTH__IDENTITY__PROVIDER
  displayName: Alias of the Openshift identity provider in Keycloak
  description: Alias of the Openshift identity provider registered in Keycloak, that should be used to create workspace OpenShift resources in Openshift namespaces owned by the current Che user.
  value: 'NULL'
- name: STRATEGY
  displayName: Update Strategy
  description: Che server update strategy. Defaults to Recreate. Use Rolling only if Che deployment does not use PVC
  value: 'Recreate'
- name: PULL_POLICY
  displayName: Che server image pull policy
  description: Always pull by default. Can be IfNotPresent
  value: 'Always'
- name: CHE_INFRA_KUBERNETES_MASTER__URL
  displayName: OpenShift master URL
  description: Required only in case of multi-cluster setup. Empty string by default.
  value: ''
- name: CHE_INFRA_KUBERNETES_PVC_PRECREATE__SUBPATHS
  displayName: Pre-create subpaths in PV
  description: Applied only if a common strategy is used. Required for Kubernetes versions before 1.6
  value: 'false'
- name: CHE_OAUTH_GITHUB_CLIENTID
  displayName: GitHub Client ID
  description: GitHub oAuth app client ID. Applicable to Che single user only!
  value: ''
- name: CHE_OAUTH_GITHUB_CLIENTSECRET
  displayName: GitHub Client Secret
  description: GitHub oAuth app client servet. Applicable to Che single user only!
  value: ''
labels:
  app: che
  template: che

mrmStack commented 6 years ago

it's posible that I have a problem with ip-tables rules?

Readiness probe failed: Get http://10.129.0.254:8080/api/system/state: dial tcp 10.129.0.254:8080: getsockopt: connection refused

thanks @eivantsov

mrmStack commented 6 years ago

It's posible that the template use the deployer user and service doesn't allow create the conection with the route http://10.129.0.254:8080/api/system/state ??

`[root@che-astro templates]# oc set probe pod/che-2-deploy --liveness
error: Pod "che-2-deploy" is invalid: [spec.containers[0].livenessProbe: Required value: must specify a handler type, spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds` or `spec.tolerations` (only additions to existing tolerations)
{"Volumes":[{"Name":"deployer-token-vcwtk","HostPath":null,"EmptyDir":null,"GCEPersistentDisk":null,"AWSElasticBlockStore":null,"GitRepo":null,"Secret":{"SecretName":"deployer-token-vcwtk","Items":null,"DefaultMode":420,"Optional":null},"NFS":null,"ISCSI":null,"Glusterfs":null,"PersistentVolumeClaim":null,"RBD":null,"Quobyte":null,"FlexVolume":null,"Cinder":null,"CephFS":null,"Flocker":null,"DownwardAPI":null,"FC":null,"AzureFile":null,"ConfigMap":null,"VsphereVolume":null,"AzureDisk":null,"PhotonPersistentDisk":null,"Projected":null,"PortworxVolume":null,"ScaleIO":null,"StorageOS":null}],"InitContainers":null,"Containers":[{"Name":"deployment","Image":"openshift/origin-deployer:v3.9.0","Command":null,"Args":null,"WorkingDir":"","Ports":null,"EnvFrom":null,"Env":[{"Name":"KUBERNETES_MASTER","Value":"https://che-astro.pic.es:8443","ValueFrom":null},{"Name":"OPENSHIFT_MASTER","Value":"https://che-astro.pic.es:8443","ValueFrom":null},{"Name":"BEARER_TOKEN_FILE","Value":"/var/run/secrets/kubernetes.io/serviceaccount/token","ValueFrom":null},{"Name":"OPENSHIFT_CA_DATA","Value":"-----BEGIN CERTIFICATE-----\nMIIC6jCCAdKgAwIBAgIBATANBgkqhkiG9w0BAQsFADAmMSQwIgYDVQQDDBtvcGVu\nc2hpZnQtc2lnbmVyQDE1MjcxODY2NTgwHhcNMTgwNTI0MTgzMDU3WhcNMjMwNTIz\nMTgzMDU4WjAmMSQwIgYDVQQDDBtvcGVuc2hpZnQtc2lnbmVyQDE1MjcxODY2NTgw\nggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDCzwKCWO3UV8qAUUDiNCNL\nDyR9rZ4fDMkoqK2HfBdkB3beMZ0IaBTHmHf3BOUlC5O8Ezgvz7GqxZNOCXMv3zZZ\nMDTpc64DY4o07IiohNbpLxDJNl7/hYxOfUI30tCoEZsKHG1vROVMcykjC7sZjJlu\nZFpf7+auWerQ5wjQzInvS+D03RJHukbUdHTQqbRfQDm7IxQdl0ewGfos+RJomGhu\ntZFuYvQj1RLC1zT/jCTGLWu4qOy6trYZyIEcZCmSiXRSaWq57GzZQDZXmnua/0yY\nSCSWnfWVSXO3z3aOpnEa3/lfmXKxUb9B87TbGp4lD3pRQ4dBiLP3Q9oguFKtRaXV\nAgMBAAGjIzAhMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MA0GCSqG\nSIb3DQEBCwUAA4IBAQBlbh9akgRnJfACB1q5OkNujvdiQmTN2OysZoFfrqqiMO/p\nzMNxIYLTMuJIUev4w+d/nsUrTWDNDcTY75MGfX7eNATjG2JjffnnASh17FD6z0yA\nOOBaKu4nSgLE7xoUP2kT0lZR6dHo1t1fQDgnB5jGO1F1KlAdNcqyft4dvtTGUwuS\nndceSoObdFmcgU59DVIvk+Ssqtq3lCPjvqk7FWxtJs+A9L5GX9XG3QvyOCAJ/ECI\nDuCDZmk6DEcVS+g5TC0KDalWRdxj720xgBGnFvJ0h1mwSIBm4zOGeg2TGtIZGgmh\nYMW+ivBsQ+eK8JmF21QbTMJAYiR65WoEHelFQOec\n-----END CERTIFICATE-----\n","ValueFrom":null},{"Name":"OPENSHIFT_DEPLOYMENT_NAME","Value":"che-2","ValueFrom":null},{"Name":"OPENSHIFT_DEPLOYMENT_NAMESPACE","Value":"eclipse-che","ValueFrom":null}],"Resources":{"Limits":null,"Requests":null},"VolumeMounts":[{"Name":"deployer-token-vcwtk","ReadOnly":true,"MountPath":"/var/run/secrets/kubernetes.io/serviceaccount","SubPath":"","MountPropagation":null}],"VolumeDevices":null,"LivenessProbe":

A: {"Exec":null,"HTTPGet":null,"TCPSocket":null,"InitialDelaySeconds":0,"TimeoutSeconds":1,"PeriodSeconds":10,"SuccessThreshold":1,"FailureThreshold":3},"ReadinessProbe":null,"Lifecycle":null,"TerminationMessagePath":"/dev/termination-log","TerminationMessagePolicy":"File","ImagePullPolicy":"IfNotPresent","SecurityContext":{"Capabilities":{"Add":null,"Drop":["KILL","MKNOD","SETGID","SETUID"]},"Privileged":null,"SELinuxOptions":null,"RunAsUser":1000790000,"RunAsNonRoot":null,"ReadOnlyRootFilesystem":null,"AllowPrivilegeEscalation":null},"Stdin":false,"StdinOnce":false,"TTY":false}],"RestartPolicy":"Never","TerminationGracePeriodSeconds":10,"ActiveDeadlineSeconds":21600,"DNSPolicy":"ClusterFirst","NodeSelector":{"node-role.kubernetes.io/compute":"true"},"ServiceAccountName":"deployer","AutomountServiceAccountToken":null,"NodeName":"che-node01.pic.es","SecurityContext":{"HostNetwork":false,"HostPID":false,"HostIPC":false,"SELinuxOptions":{"User":"","Role":"","Type":"","Level":"s0:c28,c17"},"RunAsUser":null,"RunAsNonRoot":null,"SupplementalGroups":null,"FSGroup":1000790000},"ImagePullSecrets":[{"Name":"deployer-dockercfg-gzgsl"}],"Hostname":"","Subdomain":"","Affinity":null,"SchedulerName":"default-scheduler","Tolerations":null,"HostAliases":null,"PriorityClassName":"","Priority":null,"DNSConfig":null}

B: null,"ReadinessProbe":null,"Lifecycle":null,"TerminationMessagePath":"/dev/termination-log","TerminationMessagePolicy":"File","ImagePullPolicy":"IfNotPresent","SecurityContext":{"Capabilities":{"Add":null,"Drop":["KILL","MKNOD","SETGID","SETUID"]},"Privileged":null,"SELinuxOptions":null,"RunAsUser":1000790000,"RunAsNonRoot":null,"ReadOnlyRootFilesystem":null,"AllowPrivilegeEscalation":null},"Stdin":false,"StdinOnce":false,"TTY":false}],"RestartPolicy":"Never","TerminationGracePeriodSeconds":10,"ActiveDeadlineSeconds":21600,"DNSPolicy":"ClusterFirst","NodeSelector":{"node-role.kubernetes.io/compute":"true"},"ServiceAccountName":"deployer","AutomountServiceAccountToken":null,"NodeName":"che-node01.pic.es","SecurityContext":{"HostNetwork":false,"HostPID":false,"HostIPC":false,"SELinuxOptions":{"User":"","Role":"","Type":"","Level":"s0:c28,c17"},"RunAsUser":null,"RunAsNonRoot":null,"SupplementalGroups":null,"FSGroup":1000790000},"ImagePullSecrets":[{"Name":"deployer-dockercfg-gzgsl"}],"Hostname":"","Subdomain":"","Affinity":null,"SchedulerName":"default-scheduler","Tolerations":null,"HostAliases":null,"PriorityClassName":"","Priority":null,"DNSConfig":null}
`

ghost commented 6 years ago

@mrmStack that's a deployer pod. You need logs from Che pod.

mrmStack commented 6 years ago

Ok thanks!!, I show you my pod logs now

mrmStack commented 6 years ago

This is the only get me the logs....

--> Scaled older deployment che-1 down --> Scaling che-2 to 1 error: update acceptor rejected che-2: pods for rc 'eclipse-che/che-2' took longer than 600 seconds to become available.

ghost commented 6 years ago

You need to trigger deployment again and grab logs.

mrmStack commented 6 years ago

and how make some thing like this? where I have to look?

ghost commented 6 years ago

Go to Deployments > Che. Click deploy button. After that go to Pods > Che pod > Logs

mrmStack commented 6 years ago

don't show me anything.....

mrmStack commented 6 years ago

I don't understand what happening....

ghost commented 6 years ago

@mrmStack you took a deployer pod again.

mrmStack commented 6 years ago

yes I deploy again...and say me the same....

it's better that I will start deploy che since start ?

mrmStack commented 6 years ago

And the services that I deployed yesterday, the route that assigned in the Keycloak service don't bring me to any web site....

ghost commented 6 years ago

@mrmStack all I need is Che logs. Are you familiar with OpenShift?

mrmStack commented 6 years ago

sorry @eivantsov, I'm newer with openshift deploys...and I don't understand some concepts...but in the logs of the worker node whose run the containers keycloak and postgres show me this:

    at  at ch.qos.logback.core.joran.spi.Interpreter.endElement(Interpreter.java:179)
    at  at ch.qos.logback.core.joran.spi.EventPlayer.play(EventPlayer.java:62)
    at  at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:165)
    at  at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:152)
    at  at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:110)
    at  at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:53)
    at  at ch.qos.logback.classic.util.ContextInitializer.configureByResource(ContextInitializer.java:75)
    at  at ch.qos.logback.classic.util.ContextInitializer.autoConfig(ContextInitializer.java:150)
    at  at org.slf4j.impl.StaticLoggerBinder.init(StaticLoggerBinder.java:84)
    at  at org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:55)
    at  at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150)
    at  at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124)
    at  at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:412)
    at  at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:357)
    at  at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:383)
    at  at org.eclipse.che.inject.CheBootstrap.<clinit>(CheBootstrap.java:123)
    at  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at  at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at  at org.apache.catalina.core.DefaultInstanceManager.newInstance(DefaultInstanceManager.java:120)
    at  at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4667)
    at  at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5207)
    at  at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
    at  at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:752)
    at  at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:728)
    at  at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734)
    at  at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:988)
    at  at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1860)
    at  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at  at java.lang.Thread.run(Thread.java:748)
08:25:28,103 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to INFO
08:25:28,103 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@4306cb63 - Propagating INFO level on Logger[ROOT] onto the JUL framework
08:25:28,104 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [stdout] to Logger[ROOT]
08:25:28,104 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [file] to Logger[ROOT]
08:25:28,105 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.apache.catalina.loader] to OFF
08:25:28,105 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@4306cb63 - Propagating OFF level on Logger[org.apache.catalina.loader] onto the JUL framework
08:25:28,106 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.apache.catalina.session.PersistentManagerBase] to OFF
08:25:28,106 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@4306cb63 - Propagating OFF level on Logger[org.apache.catalina.session.PersistentManagerBase] onto the JUL framework
08:25:28,106 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.apache.jasper.servlet.TldScanner] to OFF
08:25:28,106 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@4306cb63 - Propagating OFF level on Logger[org.apache.jasper.servlet.TldScanner] onto the JUL framework
08:25:28,106 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration.
08:25:28,106 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@140a68cc - Registering current configuration as safe fallback point

2018-06-01 08:25:29,460[ost-startStop-1]  [INFO ] [o.a.c.startup.HostConfig 1023]       - Deployment of web application archive [/home/user/eclipse-che/tomcat/webapps/ROOT.war] has finished in [2,276] ms
2018-06-01 08:25:29,461[ost-startStop-1]  [INFO ] [o.a.c.startup.HostConfig 959]        - Deploying web application archive [/home/user/eclipse-che/tomcat/webapps/swagger.war]
2018-06-01 08:25:29,768[ost-startStop-1]  [INFO ] [o.a.c.startup.HostConfig 1023]       - Deployment of web application archive [/home/user/eclipse-che/tomcat/webapps/swagger.war] has finished in [307] ms
2018-06-01 08:25:29,769[ost-startStop-1]  [INFO ] [o.a.c.startup.HostConfig 959]        - Deploying web application archive [/home/user/eclipse-che/tomcat/webapps/docs.war]
2018-06-01 08:25:30,398[ost-startStop-1]  [INFO ] [o.a.c.startup.HostConfig 1023]       - Deployment of web application archive [/home/user/eclipse-che/tomcat/webapps/docs.war] has finished in [629] ms
2018-06-01 08:25:30,399[ost-startStop-1]  [INFO ] [o.a.c.startup.HostConfig 959]        - Deploying web application archive [/home/user/eclipse-che/tomcat/webapps/dashboard.war]
2018-06-01 08:25:30,964[ost-startStop-1]  [INFO ] [o.a.c.startup.HostConfig 1023]       - Deployment of web application archive [/home/user/eclipse-che/tomcat/webapps/dashboard.war] has finished in [565] ms
2018-06-01 08:25:30,965[ost-startStop-1]  [INFO ] [o.a.c.startup.HostConfig 959]        - Deploying web application archive [/home/user/eclipse-che/tomcat/webapps/workspace-loader.war]
2018-06-01 08:25:31,099[e-5-8zmwz-61330]  [WARN ] [unknown.jul.logger 49]               - Error sending ping request: url [http://10.129.1.22:8888], clusterName [WorkspaceLocks], attempts[3]: connect timed out
2018-06-01 08:25:31,185[ost-startStop-1]  [INFO ] [o.a.c.startup.HostConfig 1023]       - Deployment of web application archive [/home/user/eclipse-che/tomcat/webapps/workspace-loader.war] has finished in [219] ms
2018-06-01 08:25:31,195[main]             [INFO ] [o.a.c.http11.Http11NioProtocol 632]  - Starting ProtocolHandler ["http-nio-8080"]
2018-06-01 08:25:31,209[main]             [INFO ] [o.a.catalina.startup.Catalina 683]   - Server startup in 38297 ms
2018-06-01 08:25:33,599[he-5-8zmwz-5441]  [ERROR] [oggingUncaughtExceptionHandler 31]   - Runtime exception caught in thread INT-1,RemoteSubscriptionChannel,che-5-8zmwz-5441. Message: org/jgroups/protocols/pbcast/STABLE$StabilitySendTask
java.lang.NoClassDefFoundError: org/jgroups/protocols/pbcast/STABLE$StabilitySendTask
    at org.jgroups.protocols.pbcast.STABLE.startStabilityTask(STABLE.java:508)
    at org.jgroups.protocols.pbcast.STABLE.sendStabilityMessage(STABLE.java:717)
    at org.jgroups.protocols.pbcast.STABLE.handleStableMessage(STABLE.java:584)
    at org.jgroups.protocols.pbcast.STABLE.handleUpEvent(STABLE.java:251)
    at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:237)
    at org.jgroups.protocols.UNICAST.up(UNICAST.java:402)
    at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
    at org.jgroups.protocols.BARRIER.up(BARRIER.java:142)
    at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
    at org.jgroups.protocols.FD.up(FD.java:260)
    at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:325)
    at org.jgroups.protocols.MERGE2.up(MERGE2.java:237)
    at org.jgroups.protocols.Discovery.up(Discovery.java:296)
    at org.jgroups.protocols.TP.passMessageUp(TP.java:1658)
    at org.jgroups.protocols.TP$3.run(TP.java:1592)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:748)
2018-06-01 08:25:54,888[he-5-8zmwz-3406]  [ERROR] [oggingUncaughtExceptionHandler 31]   - Runtime exception caught in thread INT-3,WorkspaceStateCache,che-5-8zmwz-3406. Message: org/jgroups/protocols/pbcast/STABLE$StabilitySendTask
java.lang.NoClassDefFoundError: org/jgroups/protocols/pbcast/STABLE$StabilitySendTask
    at org.jgroups.protocols.pbcast.STABLE.startStabilityTask(STABLE.java:508)
    at org.jgroups.protocols.pbcast.STABLE.sendStabilityMessage(STABLE.java:717)
    at org.jgroups.protocols.pbcast.STABLE.handleStableMessage(STABLE.java:584)
    at org.jgroups.protocols.pbcast.STABLE.handleUpEvent(STABLE.java:251)
    at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:237)
    at org.jgroups.protocols.UNICAST.up(UNICAST.java:402)
    at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
    at org.jgroups.protocols.BARRIER.up(BARRIER.java:142)
    at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
    at org.jgroups.protocols.FD.up(FD.java:260)
    at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:325)
    at org.jgroups.protocols.MERGE2.up(MERGE2.java:237)
    at org.jgroups.protocols.Discovery.up(Discovery.java:296)
    at org.jgroups.protocols.TP.passMessageUp(TP.java:1658)
    at org.jgroups.protocols.TP$3.run(TP.java:1592)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:748)
2018-06-01 08:25:57,102[he-5-8zmwz-3406]  [ERROR] [oggingUncaughtExceptionHandler 31]   - Runtime exception caught in thread INT-1,WorkspaceStateCache,che-5-8zmwz-3406. Message: org/jgroups/protocols/pbcast/STABLE$StabilitySendTask
java.lang.NoClassDefFoundError: org/jgroups/protocols/pbcast/STABLE$StabilitySendTask
    at org.jgroups.protocols.pbcast.STABLE.startStabilityTask(STABLE.java:508)
    at org.jgroups.protocols.pbcast.STABLE.sendStabilityMessage(STABLE.java:717)
    at org.jgroups.protocols.pbcast.STABLE.handleStableMessage(STABLE.java:584)
    at org.jgroups.protocols.pbcast.STABLE.handleUpEvent(STABLE.java:251)
    at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:237)
    at org.jgroups.protocols.UNICAST.up(UNICAST.java:402)
    at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
    at org.jgroups.protocols.BARRIER.up(BARRIER.java:142)
    at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
    at org.jgroups.protocols.FD.up(FD.java:260)
    at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:325)
    at org.jgroups.protocols.MERGE2.up(MERGE2.java:237)
    at org.jgroups.protocols.Discovery.up(Discovery.java:296)
    at org.jgroups.protocols.TP.passMessageUp(TP.java:1658)
    at org.jgroups.protocols.TP$3.run(TP.java:1592)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:748)
2018-06-01 08:25:59,543[he-5-8zmwz-5441]  [ERROR] [oggingUncaughtExceptionHandler 31]   - Runtime exception caught in thread INT-3,RemoteSubscriptionChannel,che-5-8zmwz-5441. Message: org/jgroups/protocols/pbcast/STABLE$StabilitySendTask
java.lang.NoClassDefFoundError: org/jgroups/protocols/pbcast/STABLE$StabilitySendTask
    at org.jgroups.protocols.pbcast.STABLE.startStabilityTask(STABLE.java:508)
    at org.jgroups.protocols.pbcast.STABLE.sendStabilityMessage(STABLE.java:717)
    at org.jgroups.protocols.pbcast.STABLE.handleStableMessage(STABLE.java:584)
    at org.jgroups.protocols.pbcast.STABLE.handleUpEvent(STABLE.java:251)
    at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:237)
    at org.jgroups.protocols.UNICAST.up(UNICAST.java:402)
    at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
    at org.jgroups.protocols.BARRIER.up(BARRIER.java:142)
    at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
    at org.jgroups.protocols.FD.up(FD.java:260)
    at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:325)
    at org.jgroups.protocols.MERGE2.up(MERGE2.java:237)
    at org.jgroups.protocols.Discovery.up(Discovery.java:296)
    at org.jgroups.protocols.TP.passMessageUp(TP.java:1658)
    at org.jgroups.protocols.TP$3.run(TP.java:1592)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:748)
2018-06-01 08:26:01,237[he-5-8zmwz-5441]  [ERROR] [oggingUncaughtExceptionHandler 31]   - Runtime exception caught in thread INT-2,RemoteSubscriptionChannel,che-5-8zmwz-5441. Message: org/jgroups/protocols/pbcast/STABLE$StabilitySendTask
java.lang.NoClassDefFoundError: org/jgroups/protocols/pbcast/STABLE$StabilitySendTask
    at org.jgroups.protocols.pbcast.STABLE.startStabilityTask(STABLE.java:508)
    at org.jgroups.protocols.pbcast.STABLE.sendStabilityMessage(STABLE.java:717)
    at org.jgroups.protocols.pbcast.STABLE.handleStableMessage(STABLE.java:584)
    at org.jgroups.protocols.pbcast.STABLE.handleUpEvent(STABLE.java:251)
    at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:237)
    at org.jgroups.protocols.UNICAST.up(UNICAST.java:402)
    at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
    at org.jgroups.protocols.BARRIER.up(BARRIER.java:142)
    at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
    at org.jgroups.protocols.FD.up(FD.java:260)
    at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:325)
    at org.jgroups.protocols.MERGE2.up(MERGE2.java:237)
    at org.jgroups.protocols.Discovery.up(Discovery.java:296)
    at org.jgroups.protocols.TP.passMessageUp(TP.java:1658)
    at org.jgroups.protocols.TP$3.run(TP.java:1592)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:748)

Received SIGTERM
Stopping Che server running on localhost:8080
2018-06-01 08:26:06,782[main]             [INFO ] [o.a.c.core.StandardServer 524]       - A valid shutdown command was received via the shutdown port. Stopping the Server instance.
2018-06-01 08:26:06,783[main]             [INFO ] [o.a.c.http11.Http11NioProtocol 658]  - Pausing ProtocolHandler ["http-nio-8080"]
2018-06-01 08:26:06,835[main]             [INFO ] [o.a.c.core.StandardService 477]      - Stopping service [Catalina]
2018-06-01 08:26:06,973[main]             [INFO ] [o.a.c.http11.Http11NioProtocol 687]  - Stopping ProtocolHandler ["http-nio-8080"]
2018-06-01 08:26:06,978[main]             [INFO ] [o.a.c.http11.Http11NioProtocol 707]  - Destroying ProtocolHandler ["http-nio-8080"]
2018-06-01 08:26:10,874[e-5-8zmwz-61330]  [ERROR] [oggingUncaughtExceptionHandler 31]   - Runtime exception caught in thread INT-2,WorkspaceLocks,che-5-8zmwz-61330. Message: org/jgroups/protocols/pbcast/STABLE$StabilitySendTask
java.lang.NoClassDefFoundError: org/jgroups/protocols/pbcast/STABLE$StabilitySendTask
    at org.jgroups.protocols.pbcast.STABLE.startStabilityTask(STABLE.java:508)
    at org.jgroups.protocols.pbcast.STABLE.sendStabilityMessage(STABLE.java:717)
    at org.jgroups.protocols.pbcast.STABLE.handleStableMessage(STABLE.java:584)
    at org.jgroups.protocols.pbcast.STABLE.handleUpEvent(STABLE.java:251)
    at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:237)
    at org.jgroups.protocols.UNICAST.up(UNICAST.java:402)
    at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
    at org.jgroups.protocols.BARRIER.up(BARRIER.java:142)
    at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
    at org.jgroups.protocols.FD.up(FD.java:260)
    at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:325)
    at org.jgroups.protocols.MERGE2.up(MERGE2.java:237)
    at org.jgroups.protocols.Discovery.up(Discovery.java:296)
    at org.jgroups.protocols.TP.passMessageUp(TP.java:1658)
    at org.jgroups.protocols.TP$3.run(TP.java:1592)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:748)
2018-06-01 08:26:35,863[he-5-8zmwz-5441]  [ERROR] [oggingUncaughtExceptionHandler 31]   - Runtime exception caught in thread INT-5,RemoteSubscriptionChannel,che-5-8zmwz-5441. Message: org/jgroups/protocols/pbcast/STABLE$StabilitySendTask
java.lang.NoClassDefFoundError: org/jgroups/protocols/pbcast/STABLE$StabilitySendTask
    at org.jgroups.protocols.pbcast.STABLE.startStabilityTask(STABLE.java:508)
    at org.jgroups.protocols.pbcast.STABLE.sendStabilityMessage(STABLE.java:717)
    at org.jgroups.protocols.pbcast.STABLE.handleStableMessage(STABLE.java:584)
    at org.jgroups.protocols.pbcast.STABLE.handleUpEvent(STABLE.java:251)
    at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:237)
    at org.jgroups.protocols.UNICAST.up(UNICAST.java:402)
    at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
    at org.jgroups.protocols.BARRIER.up(BARRIER.java:142)
    at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
    at org.jgroups.protocols.FD.up(FD.java:260)
    at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:325)
    at org.jgroups.protocols.MERGE2.up(MERGE2.java:237)
    at org.jgroups.protocols.Discovery.up(Discovery.java:296)
    at org.jgroups.protocols.TP.passMessageUp(TP.java:1658)
    at org.jgroups.protocols.TP$3.run(TP.java:1592)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:748)
2018-06-01 08:26:36,127[e-5-8zmwz-31160]  [ERROR] [oggingUncaughtExceptionHandler 31]   - Runtime exception caught in thread INT-1,EclipseLinkCommandChannel,che-5-8zmwz-31160. Message: org/jgroups/protocols/pbcast/STABLE$StabilitySendTask
java.lang.NoClassDefFoundError: org/jgroups/protocols/pbcast/STABLE$StabilitySendTask
    at org.jgroups.protocols.pbcast.STABLE.startStabilityTask(STABLE.java:508)
    at org.jgroups.protocols.pbcast.STABLE.sendStabilityMessage(STABLE.java:717)
    at org.jgroups.protocols.pbcast.STABLE.handleStableMessage(STABLE.java:584)
    at org.jgroups.protocols.pbcast.STABLE.handleUpEvent(STABLE.java:251)
    at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:237)
    at org.jgroups.protocols.UNICAST.up(UNICAST.java:402)
    at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
    at org.jgroups.protocols.BARRIER.up(BARRIER.java:142)
    at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
    at org.jgroups.protocols.FD.up(FD.java:260)
    at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:325)
    at org.jgroups.protocols.MERGE2.up(MERGE2.java:237)
    at org.jgroups.protocols.Discovery.up(Discovery.java:296)
    at org.jgroups.protocols.TP.passMessageUp(TP.java:1658)
    at org.jgroups.protocols.TP$3.run(TP.java:1592)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:748)
2018-06-01 08:27:20,643[he-5-8zmwz-3406]  [ERROR] [oggingUncaughtExceptionHandler 31]   - Runtime exception caught in thread INT-5,WorkspaceStateCache,che-5-8zmwz-3406. Message: org/jgroups/protocols/pbcast/STABLE$StabilitySendTask
java.lang.NoClassDefFoundError: org/jgroups/protocols/pbcast/STABLE$StabilitySendTask
    at org.jgroups.protocols.pbcast.STABLE.startStabilityTask(STABLE.java:508)
    at org.jgroups.protocols.pbcast.STABLE.sendStabilityMessage(STABLE.java:717)
    at org.jgroups.protocols.pbcast.STABLE.handleStableMessage(STABLE.java:584)
    at org.jgroups.protocols.pbcast.STABLE.handleUpEvent(STABLE.java:251)
    at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:237)
    at org.jgroups.protocols.UNICAST.up(UNICAST.java:402)
    at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
    at org.jgroups.protocols.BARRIER.up(BARRIER.java:142)
    at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
    at org.jgroups.protocols.FD.up(FD.java:260)
    at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:325)
    at org.jgroups.protocols.MERGE2.up(MERGE2.java:237)
    at org.jgroups.protocols.Discovery.up(Discovery.java:296)
    at org.jgroups.protocols.TP.passMessageUp(TP.java:1658)
    at org.jgroups.protocols.TP$3.run(TP.java:1592)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:748)
2018-06-01 08:27:30,399[e-5-8zmwz-31160]  [ERROR] [oggingUncaughtExceptionHandler 31]   - Runtime exception caught in thread INT-2,EclipseLinkCommandChannel,che-5-8zmwz-31160. Message: org/jgroups/protocols/pbcast/STABLE$StabilitySendTask
java.lang.NoClassDefFoundError: org/jgroups/protocols/pbcast/STABLE$StabilitySendTask
    at org.jgroups.protocols.pbcast.STABLE.startStabilityTask(STABLE.java:508)
    at org.jgroups.protocols.pbcast.STABLE.sendStabilityMessage(STABLE.java:717)
    at org.jgroups.protocols.pbcast.STABLE.handleStableMessage(STABLE.java:584)
    at org.jgroups.protocols.pbcast.STABLE.handleUpEvent(STABLE.java:251)
    at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:237)
    at org.jgroups.protocols.UNICAST.up(UNICAST.java:402)
    at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
    at org.jgroups.protocols.BARRIER.up(BARRIER.java:142)
    at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
    at org.jgroups.protocols.FD.up(FD.java:260)
    at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:325)
    at org.jgroups.protocols.MERGE2.up(MERGE2.java:237)
    at org.jgroups.protocols.Discovery.up(Discovery.java:296)
    at org.jgroups.protocols.TP.passMessageUp(TP.java:1658)
    at org.jgroups.protocols.TP$3.run(TP.java:1592)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:748)
2018-06-01 08:27:36,474[e-5-8zmwz-61330]  [ERROR] [oggingUncaughtExceptionHandler 31]   - Runtime exception caught in thread INT-3,WorkspaceLocks,che-5-8zmwz-61330. Message: org/jgroups/protocols/pbcast/STABLE$StabilitySendTask
java.lang.NoClassDefFoundError: org/jgroups/protocols/pbcast/STABLE$StabilitySendTask
    at org.jgroups.protocols.pbcast.STABLE.startStabilityTask(STABLE.java:508)
    at org.jgroups.protocols.pbcast.STABLE.sendStabilityMessage(STABLE.java:717)
    at org.jgroups.protocols.pbcast.STABLE.handleStableMessage(STABLE.java:584)
    at org.jgroups.protocols.pbcast.STABLE.handleUpEvent(STABLE.java:251)
    at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:237)
    at org.jgroups.protocols.UNICAST.up(UNICAST.java:402)
    at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
    at org.jgroups.protocols.BARRIER.up(BARRIER.java:142)
    at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
    at org.jgroups.protocols.FD.up(FD.java:260)
    at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:325)
    at org.jgroups.protocols.MERGE2.up(MERGE2.java:237)
    at org.jgroups.protocols.Discovery.up(Discovery.java:296)
    at org.jgroups.protocols.TP.passMessageUp(TP.java:1658)
    at org.jgroups.protocols.TP$3.run(TP.java:1592)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:748)
2018-06-01 08:28:07,547[he-5-8zmwz-5441]  [ERROR] [oggingUncaughtExceptionHandler 31]   - Runtime exception caught in thread INT-6,RemoteSubscriptionChannel,che-5-8zmwz-5441. Message: org/jgroups/protocols/pbcast/STABLE$StabilitySendTask
java.lang.NoClassDefFoundError: org/jgroups/protocols/pbcast/STABLE$StabilitySendTask
    at org.jgroups.protocols.pbcast.STABLE.startStabilityTask(STABLE.java:508)
    at org.jgroups.protocols.pbcast.STABLE.sendStabilityMessage(STABLE.java:717)
    at org.jgroups.protocols.pbcast.STABLE.handleStableMessage(STABLE.java:584)
    at org.jgroups.protocols.pbcast.STABLE.handleUpEvent(STABLE.java:251)
    at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:237)
    at org.jgroups.protocols.UNICAST.up(UNICAST.java:402)
    at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
    at org.jgroups.protocols.BARRIER.up(BARRIER.java:142)
    at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
    at org.jgroups.protocols.FD.up(FD.java:260)
    at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:325)
    at org.jgroups.protocols.MERGE2.up(MERGE2.java:237)
    at org.jgroups.protocols.Discovery.up(Discovery.java:296)
    at org.jgroups.protocols.TP.passMessageUp(TP.java:1658)
    at org.jgroups.protocols.TP$3.run(TP.java:1592)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:748)
2018-06-01 08:28:29,254[he-5-8zmwz-3406]  [ERROR] [oggingUncaughtExceptionHandler 31]   - Runtime exception caught in thread INT-4,WorkspaceStateCache,che-5-8zmwz-3406. Message: org/jgroups/protocols/pbcast/STABLE$StabilitySendTask
java.lang.NoClassDefFoundError: org/jgroups/protocols/pbcast/STABLE$StabilitySendTask
    at org.jgroups.protocols.pbcast.STABLE.startStabilityTask(STABLE.java:508)
    at org.jgroups.protocols.pbcast.STABLE.sendStabilityMessage(STABLE.java:717)
    at org.jgroups.protocols.pbcast.STABLE.handleStableMessage(STABLE.java:584)
    at org.jgroups.protocols.pbcast.STABLE.handleUpEvent(STABLE.java:251)
    at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:237)
    at org.jgroups.protocols.UNICAST.up(UNICAST.java:402)
    at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
    at org.jgroups.protocols.BARRIER.up(BARRIER.java:142)
    at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
    at org.jgroups.protocols.FD.up(FD.java:260)
    at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:325)
    at org.jgroups.protocols.MERGE2.up(MERGE2.java:237)
    at org.jgroups.protocols.Discovery.up(Discovery.java:296)
    at org.jgroups.protocols.TP.passMessageUp(TP.java:1658)
    at org.jgroups.protocols.TP$3.run(TP.java:1592)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:748)
2018-06-01 08:28:33,391[he-5-8zmwz-5441]  [ERROR] [oggingUncaughtExceptionHandler 31]   - Runtime exception caught in thread INT-4,RemoteSubscriptionChannel,che-5-8zmwz-5441. Message: org/jgroups/protocols/pbcast/STABLE$StabilitySendTask
java.lang.NoClassDefFoundError: org/jgroups/protocols/pbcast/STABLE$StabilitySendTask
    at org.jgroups.protocols.pbcast.STABLE.startStabilityTask(STABLE.java:508)
    at org.jgroups.protocols.pbcast.STABLE.sendStabilityMessage(STABLE.java:717)
    at org.jgroups.protocols.pbcast.STABLE.handleStableMessage(STABLE.java:584)
    at org.jgroups.protocols.pbcast.STABLE.handleUpEvent(STABLE.java:251)
    at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:237)
    at org.jgroups.protocols.UNICAST.up(UNICAST.java:402)
    at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
    at org.jgroups.protocols.BARRIER.up(BARRIER.java:142)
    at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
    at org.jgroups.protocols.FD.up(FD.java:260)
    at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:325)
    at org.jgroups.protocols.MERGE2.up(MERGE2.java:237)
    at org.jgroups.protocols.Discovery.up(Discovery.java:296)
    at org.jgroups.protocols.TP.passMessageUp(TP.java:1658)
    at org.jgroups.protocols.TP$3.run(TP.java:1592)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:748)
2018-06-01 08:28:52,666[e-5-8zmwz-31160]  [ERROR] [oggingUncaughtExceptionHandler 31]   - Runtime exception caught in thread INT-3,EclipseLinkCommandChannel,che-5-8zmwz-31160. Message: org/jgroups/protocols/pbcast/STABLE$StabilitySendTask
java.lang.NoClassDefFoundError: org/jgroups/protocols/pbcast/STABLE$StabilitySendTask
    at org.jgroups.protocols.pbcast.STABLE.startStabilityTask(STABLE.java:508)
    at org.jgroups.protocols.pbcast.STABLE.sendStabilityMessage(STABLE.java:717)
    at org.jgroups.protocols.pbcast.STABLE.handleStableMessage(STABLE.java:584)
    at org.jgroups.protocols.pbcast.STABLE.handleUpEvent(STABLE.java:251)
    at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:237)
    at org.jgroups.protocols.UNICAST.up(UNICAST.java:402)
    at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
    at org.jgroups.protocols.BARRIER.up(BARRIER.java:142)
    at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
    at org.jgroups.protocols.FD.up(FD.java:260)
    at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:325)
    at org.jgroups.protocols.MERGE2.up(MERGE2.java:237)
    at org.jgroups.protocols.Discovery.up(Discovery.java:296)
    at org.jgroups.protocols.TP.passMessageUp(TP.java:1658)
    at org.jgroups.protocols.TP$3.run(TP.java:1592)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:748)
2018-06-01 08:28:55,506[e-5-8zmwz-61330]  [ERROR] [oggingUncaughtExceptionHandler 31]   - Runtime exception caught in thread INT-1,WorkspaceLocks,che-5-8zmwz-61330. Message: org/jgroups/protocols/pbcast/STABLE$StabilitySendTask
java.lang.NoClassDefFoundError: org/jgroups/protocols/pbcast/STABLE$StabilitySendTask
    at org.jgroups.protocols.pbcast.STABLE.startStabilityTask(STABLE.java:508)
    at org.jgroups.protocols.pbcast.STABLE.sendStabilityMessage(STABLE.java:717)
    at org.jgroups.protocols.pbcast.STABLE.handleStableMessage(STABLE.java:584)
    at org.jgroups.protocols.pbcast.STABLE.handleUpEvent(STABLE.java:251)
    at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:237)
    at org.jgroups.protocols.UNICAST.up(UNICAST.java:402)
    at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
    at org.jgroups.protocols.BARRIER.up(BARRIER.java:142)
    at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
    at org.jgroups.protocols.FD.up(FD.java:260)
    at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:325)
    at org.jgroups.protocols.MERGE2.up(MERGE2.java:237)
    at org.jgroups.protocols.Discovery.up(Discovery.java:296)
    at org.jgroups.protocols.TP.passMessageUp(TP.java:1658)
    at org.jgroups.protocols.TP$3.run(TP.java:1592)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:748)
2018-06-01 08:29:32,670[he-5-8zmwz-3406]  [ERROR] [oggingUncaughtExceptionHandler 31]   - Runtime exception caught in thread INT-7,WorkspaceStateCache,che-5-8zmwz-3406. Message: org/jgroups/protocols/pbcast/STABLE$StabilitySendTask
java.lang.NoClassDefFoundError: org/jgroups/protocols/pbcast/STABLE$StabilitySendTask
    at org.jgroups.protocols.pbcast.STABLE.startStabilityTask(STABLE.java:508)
    at org.jgroups.protocols.pbcast.STABLE.sendStabilityMessage(STABLE.java:717)
    at org.jgroups.protocols.pbcast.STABLE.handleStableMessage(STABLE.java:584)
    at org.jgroups.protocols.pbcast.STABLE.handleUpEvent(STABLE.java:251)
    at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:237)
    at org.jgroups.protocols.UNICAST.up(UNICAST.java:402)
    at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
    at org.jgroups.protocols.BARRIER.up(BARRIER.java:142)
    at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
    at org.jgroups.protocols.FD.up(FD.java:260)
    at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:325)
    at org.jgroups.protocols.MERGE2.up(MERGE2.java:237)
    at org.jgroups.protocols.Discovery.up(Discovery.java:296)
    at org.jgroups.protocols.TP.passMessageUp(TP.java:1658)
    at org.jgroups.protocols.TP$3.run(TP.java:1592)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:748)
2018-06-01 08:29:46,090[e-5-8zmwz-61330]  [ERROR] [oggingUncaughtExceptionHandler 31]   - Runtime exception caught in thread INT-5,WorkspaceLocks,che-5-8zmwz-61330. Message: org/jgroups/protocols/pbcast/STABLE$StabilitySendTask
java.lang.NoClassDefFoundError: org/jgroups/protocols/pbcast/STABLE$StabilitySendTask
    at org.jgroups.protocols.pbcast.STABLE.startStabilityTask(STABLE.java:508)
    at org.jgroups.protocols.pbcast.STABLE.sendStabilityMessage(STABLE.java:717)
    at org.jgroups.protocols.pbcast.STABLE.handleStableMessage(STABLE.java:584)
    at org.jgroups.protocols.pbcast.STABLE.handleUpEvent(STABLE.java:251)
    at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:237)
    at org.jgroups.protocols.UNICAST.up(UNICAST.java:402)
    at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
    at org.jgroups.protocols.BARRIER.up(BARRIER.java:142)
    at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
    at org.jgroups.protocols.FD.up(FD.java:260)
    at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:325)
    at org.jgroups.protocols.MERGE2.up(MERGE2.java:237)
    at org.jgroups.protocols.Discovery.up(Discovery.java:296)
    at org.jgroups.protocols.TP.passMessageUp(TP.java:1658)
    at org.jgroups.protocols.TP$3.run(TP.java:1592)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:748)
2018-06-01 08:29:54,201[he-5-8zmwz-5441]  [ERROR] [oggingUncaughtExceptionHandler 31]   - Runtime exception caught in thread INT-8,RemoteSubscriptionChannel,che-5-8zmwz-5441. Message: org/jgroups/protocols/pbcast/STABLE$StabilitySendTask
java.lang.NoClassDefFoundError: org/jgroups/protocols/pbcast/STABLE$StabilitySendTask
    at org.jgroups.protocols.pbcast.STABLE.startStabilityTask(STABLE.java:508)
    at org.jgroups.protocols.pbcast.STABLE.sendStabilityMessage(STABLE.java:717)
    at org.jgroups.protocols.pbcast.STABLE.handleStableMessage(STABLE.java:584)
    at org.jgroups.protocols.pbcast.STABLE.handleUpEvent(STABLE.java:251)
    at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:237)
    at org.jgroups.protocols.UNICAST.up(UNICAST.java:402)
    at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
    at org.jgroups.protocols.BARRIER.up(BARRIER.java:142)
    at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
    at org.jgroups.protocols.FD.up(FD.java:260)
    at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:325)
    at org.jgroups.protocols.MERGE2.up(MERGE2.java:237)
    at org.jgroups.protocols.Discovery.up(Discovery.java:296)
    at org.jgroups.protocols.TP.passMessageUp(TP.java:1658)
    at org.jgroups.protocols.TP$3.run(TP.java:1592)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:748)
2018-06-01 08:29:55,531[e-5-8zmwz-31160]  [ERROR] [oggingUncaughtExceptionHandler 31]   - Runtime exception caught in thread INT-4,EclipseLinkCommandChannel,che-5-8zmwz-31160. Message: org/jgroups/protocols/pbcast/STABLE$StabilitySendTask
java.lang.NoClassDefFoundError: org/jgroups/protocols/pbcast/STABLE$StabilitySendTask
    at org.jgroups.protocols.pbcast.STABLE.startStabilityTask(STABLE.java:508)
    at org.jgroups.protocols.pbcast.STABLE.sendStabilityMessage(STABLE.java:717)
    at org.jgroups.protocols.pbcast.STABLE.handleStableMessage(STABLE.java:584)
    at org.jgroups.protocols.pbcast.STABLE.handleUpEvent(STABLE.java:251)
    at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:237)
    at org.jgroups.protocols.UNICAST.up(UNICAST.java:402)
    at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:649)
    at org.jgroups.protocols.BARRIER.up(BARRIER.java:142)
    at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
    at org.jgroups.protocols.FD.up(FD.java:260)
    at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:325)
    at org.jgroups.protocols.MERGE2.up(MERGE2.java:237)
    at org.jgroups.protocols.Discovery.up(Discovery.java:296)
    at org.jgroups.protocols.TP.passMessageUp(TP.java:1658)
    at org.jgroups.protocols.TP$3.run(TP.java:1592)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:748)

thanks for your help...

mrmStack commented 6 years ago

ghost commented 6 years ago

@mrmStack this is not enough. A the beginning of a log and start of exception thread is what helps.

I have already provided instruction on where to find Che pod logs.

mrmStack commented 6 years ago

Yes, I have followed your instructions, but openshift says me that the logs are not avaiable...I dont understand what happening now.

mrmStack commented 6 years ago

I believe that the problem is the permisions of the user admin openshift...or some thing like this...

ghost commented 6 years ago

mrmStack commented 6 years ago

I am following the same steps, but the logs are unavaiable....I'm so sorry @eivantsov but I think to deploy eclipse che in docker swarm...because in openshift doesn't work...

ghost commented 6 years ago

@mrmStack sorry about that. Logs are there, you just need to figure out how to use my suggestions to get them

mrmStack commented 6 years ago

Only show me this logs..

[root@che-astro templates]# oc logs --loglevel=7 dc/che
I0601 13:57:21.790104   59386 loader.go:357] Config loaded from file /root/.kube/config
I0601 13:57:21.832084   59386 round_trippers.go:414] GET https://che-astro.pic.es:8443/apis/apps.openshift.io/v1/namespaces/eclipse-che/deploymentconfigs/che
I0601 13:57:21.832145   59386 round_trippers.go:421] Request Headers:
I0601 13:57:21.832167   59386 round_trippers.go:424]     User-Agent: oc/v1.9.1+a0ce1bc657 (linux/amd64) kubernetes/a0ce1bc
I0601 13:57:21.832186   59386 round_trippers.go:424]     Accept: application/json, */*
I0601 13:57:21.857618   59386 round_trippers.go:439] Response Status: 200 OK in 25 milliseconds
I0601 13:57:21.889011   59386 round_trippers.go:414] GET https://che-astro.pic.es:8443/apis/apps.openshift.io/v1/namespaces/eclipse-che/deploymentconfigs/che/log
I0601 13:57:21.889224   59386 round_trippers.go:421] Request Headers:
I0601 13:57:21.889484   59386 round_trippers.go:424]     Accept: application/json, */*
I0601 13:57:21.889675   59386 round_trippers.go:424]     User-Agent: oc/v1.9.1+a0ce1bc657 (linux/amd64) kubernetes/a0ce1bc
I0601 13:57:21.910557   59386 round_trippers.go:439] Response Status: 200 OK in 20 milliseconds
--> Scaling che-1 to 1
error: update acceptor rejected che-1: pods for rc 'eclipse-che/che-1' took longer than 600 seconds to become available

I0601 14:01:12.663386   60881 round_trippers.go:417] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: oc/v1.9.1+a0ce1bc657 (linux/amd64) kubernetes/a0ce1bc" https://che-astro.pic.es:8443/api/v1/namespaces/eclipse-che/pods/che-2-7ppcj/log
I0601 14:01:12.690794   60881 round_trippers.go:436] GET https://che-astro.pic.es:8443/api/v1/namespaces/eclipse-che/pods/che-2-7ppcj/log 400 Bad Request in 27 milliseconds
I0601 14:01:12.690872   60881 round_trippers.go:442] Response Headers:
I0601 14:01:12.690893   60881 round_trippers.go:445]     Content-Type: application/json
I0601 14:01:12.690928   60881 round_trippers.go:445]     Content-Length: 194
I0601 14:01:12.690958   60881 round_trippers.go:445]     Date: Fri, 01 Jun 2018 12:01:12 GMT
I0601 14:01:12.690977   60881 round_trippers.go:445]     Cache-Control: no-store
I0601 14:01:12.691032   60881 request.go:874] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container \"che\" in pod \"che-2-7ppcj\" is waiting to start: ContainerCreating","reason":"BadRequest","code":400}
I0601 14:01:12.691516   60881 helpers.go:201] server response object: [{
  "metadata": {},
  "status": "Failure",
  "message": "container \"che\" in pod \"che-2-7ppcj\" is waiting to start: ContainerCreating",
  "reason": "BadRequest",
  "code": 400
}]
F0601 14:01:12.692396   60881 helpers.go:119] Error from server (BadRequest): container "che" in pod "che-2-7ppcj" is waiting to start: ContainerCreating

ghost commented 6 years ago

@mrmStack I have run out of ideas on how to help you give me Che pod logs just after it started. I recorded an animated gif to make it obvious but you still doing things they way you do.

Without Che server logs I won't be able to say why Che server crashed (nobody can)

mrmStack commented 6 years ago

Hi @eivantsov,

Exists anyway to look the logs via terminal (client oc)? because i don't understand why, but in my interface doesn't appear the logs, and I have followed the same steps of you in the animation.

thanks regards

mrmStack commented 6 years ago

It seems that's working now, I create 4 pv volumes and I run the deploy_che.sh that's inside the repository of che with the corresponents enviorment variables. The buils shows me the same error, and I have deleted the claim volume and I have build only the che-server template again, and it's working now....

Do you have any idea why?

Events:
  Type    Reason                 Age   From                        Message
  ----    ------                 ----  ----                        -------
  Normal  Scheduled              30m   default-scheduler           Successfully assigned che-1-f675r to che-node01.pic.es
  Normal  SuccessfulMountVolume  30m   kubelet, che-node01.pic.es  MountVolume.SetUp succeeded for volume "che-token-hvsmk"
  Normal  Pulling                30m   kubelet, che-node01.pic.es  pulling image "docker.io/eclipse/che-server:nightly"
  Normal  Pulled                 30m   kubelet, che-node01.pic.es  Successfully pulled image "docker.io/eclipse/che-server:nightly"
  Normal  Created                30m   kubelet, che-node01.pic.es  Created container
  Normal  Started                30m   kubelet, che-node01.pic.es  Started container

mrmStack commented 6 years ago

I will close the issue, if you hay any idea why the deploy mounts the volume if I have deleted the claim volume that template create first, open again

thanks for all @eivantsov .

eclipse-che / che

timeout expired waiting for volumes to attach/mount for pod "eclipse-che"/"postgres-1-56ffs". list of unattached/unmounted volumes=[postgres-data] #9880

Should I modify some fields of che-server template?