search

eclipse-che / che

Kubernetes based Cloud Development Environments for Enterprise Teams
http://eclipse.org/che
Eclipse Public License 2.0
6.96k stars 1.19k forks source link

Failed to open Eclipse Che Dashboard on minikube v1.25.1 with DevWorkspace enabled #20984

Closed tolusha closed 2 years ago

tolusha commented 2 years ago

Describe the bug

Failed to open

Che version

next (development version)

Steps to reproduce

chectl update next chectl server:deploy --platform minikube chectl dashboard:open

Dashboard is not opened. Che server logs contains stacktrace.

Expected behavior

Dashboard is opened

Runtime

minikube

Screenshots

No response

Installation method

chectl/next

Environment

Linux

Eclipse Che Logs

2022-01-05 14:42:16,013[io-8080-exec-10]  [ERROR] [c.a.c.r.RuntimeExceptionMapper 47]   - Internal Server Error occurred, error time: 2022-01-05 14:42:16
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://10.96.0.1/api/v1/namespaces/admin-che/secrets/workspace-credentials-secret. Message: Unauthorized! Configured service account doesn't have access. Service account may have been revoked. Unauthorized.
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:686)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:623)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:565)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:526)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:493)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:475)
    at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleGet(BaseOperation.java:807)
    at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:188)
    at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:155)
    at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:88)
    at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.configurator.CredentialsSecretConfigurator.configure(CredentialsSecretConfigurator.java:43)
    at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesNamespaceFactory.configureNamespace(KubernetesNamespaceFactory.java:570)
    at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesNamespaceFactory.getOrCreate(KubernetesNamespaceFactory.java:334)
    at org.eclipse.che.workspace.infrastructure.kubernetes.provision.NamespaceProvisioner.provision(NamespaceProvisioner.java:42)
    at org.eclipse.che.workspace.infrastructure.kubernetes.api.server.KubernetesNamespaceService.provision(KubernetesNamespaceService.java:95)


### Additional context

_No response_
max-cx commented 2 years ago

For me, this happened with Che 7.41.1 on minikube v1.24.0, Kubernetes v1.22.3, Docker 20.10.8, chectl 7.41.1, kubectl (client:v1.23.1/server:v1.22.3).

tolusha commented 2 years ago

7.41.1 works fine

tolusha commented 2 years ago

@sparkoo Could you have a look pls. It should work out of box but instead we have the similar problem like this https://github.com/eclipse/che/issues/21049

sparkoo commented 2 years ago

hmm I've quickly looked at this, I can reproduce, but I don't know what's wrong. Token looks ok, permissions looks ok. I can't get the secret even if I try to use token with curl. My bet is something with kubernetes apiserver configuration with dex, because it looks like kubernetes does not know the user from the token. Maybe something new in latest minikube versions?

tolusha commented 2 years ago

Which version minikube did you use for testing? I've tried with minikube version: v1.22.0 and got the same result che-serer logs:

2022-01-28 12:53:36,911[nio-8080-exec-7]  [ERROR] [c.a.c.r.RuntimeExceptionMapper 47]   - Internal Server Error occurred, error time: 2022-01-28 12:53:36
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://10.96.0.1/api/v1/namespaces/admin-che/secrets/workspace-credentials-secret. Message: Unauthorized! Configured service account doesn't have access. Service account may have been revoked. Unauthorized.
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:686)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:623)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:565)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:526)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:493)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:475)
    at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleGet(BaseOperation.java:807)
    at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:188)
    at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:155)
    at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:88)
    at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.configurator.CredentialsSecretConfigurator.configure(CredentialsSecretConfigurator.java:43)
    at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesNamespaceFactory.configureNamespace(KubernetesNamespaceFactory.java:570)
    at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesNamespaceFactory.getOrCreate(KubernetesNamespaceFactory.java:334)
    at org.eclipse.che.workspace.infrastructure.kubernetes.provision.NamespaceProvisioner.provision(NamespaceProvisioner.java:42)
    at org.eclipse.che.workspace.infrastructure.kubernetes.api.server.KubernetesNamespaceService.provision(KubernetesNamespaceService.java:95)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.base/java.lang.reflect.Method.invoke(Unknown Source)

Untitled

sparkoo commented 2 years ago

I have v1.25.0

I would suspect this patch https://github.com/eclipse-che/che-operator/pull/1291. Can you test before and after? Or test stable version if it works ok? I think it's something with mismatching oidc configuration. client_id, secret, claim or something like that does not match kubernetes and Che configurations

tolusha commented 2 years ago

Can't reproduce after switching to minikube v1.24.0 and cleanin up old resources

l0rd commented 2 years ago

@tolusha I am reproducing it on a fresh minikube v1.25.1. What do you mean with "cleaning up old resources"?

tolusha commented 2 years ago

It works well for v1.24.0 Yes, v1.25.1 is still an issue.

"cleaning up old resources"

I mean rm -rf ./minikube but not sure if it really helped. But yes, switching to minikube v1.24.0 fixed issue.

I will reopen to adopt to v1.25.x

l0rd commented 2 years ago

Moving to 7.44 as 7.43 has been released.

barrygear commented 2 years ago

I've been having this issue with clusters created in AWS EKS. Verified that the Dex installation is correct. It creates the namespace with the user name but then fails while checking for the Secret. I've given ClusterAdmin to all forms of claims from Dex and its clearly not using any of them. aws eks 1.21, dex 2.30.0, che 7.43.0, using che operator, and dev-workspace.

slieer commented 2 years ago

lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.3 LTS Release: 20.04 Codename: focal

kubectl version Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.3", GitCommit:"816c97ab8cff8a1c72eccca1026f7820e93e0d25", GitTreeState:"clean", BuildDate:"2022-01-25T21:25:17Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.1", GitCommit:"632ed300f2c34f6d6d15ca4cef3d3c7073412212", GitTreeState:"clean", BuildDate:"2021-08-19T15:39:34Z", GoVersion:"go1.16.7", Compiler:"gc", Platform:"linux/amd64"} minikube version minikube version: v1.25.1 commit: 3e64b11ed75e56e4898ea85f96b2e4af0301f43d chectl version chectl/7.43.0 linux-x64 node-v16.13.2

chectl server:status Eclipse Che Version : 7.43.0 Eclipse Che Url : https://192.168.49.2.nip.io/dashboard/

image

tolusha commented 2 years ago

We have to wait minikube v1.26.0 to test. Multiple issues which prevent deployment Eclipse Che on minikube v1.25.x

https://github.com/kubernetes/minikube/issues/13543 https://github.com/kubernetes/minikube/issues/13552 https://github.com/kubernetes/minikube/issues/13519 https://github.com/kubernetes/minikube/issues/13503 https://github.com/kubernetes/minikube/issues/13625

amol-m-deshpande commented 2 years ago

@tolusha I have been trying the eclipse che on minikube. Installation is done

  ✔ Retrieving Che self-signed CA certificate...OK
  ✔ Prepare post installation output...done
  ✔ Show important messages
    ✔ Eclipse Che 'next' has been successfully deployed.
    ✔ Documentation             : https://www.eclipse.org/che/docs/
    ✔ -------------------------------------------------------------------------------
    ✔ Users Dashboard           : https://192.168.49.2.nip.io/dashboard/
    ✔ -------------------------------------------------------------------------------
    ✔ Plug-in Registry          : https://192.168.49.2.nip.io/plugin-registry/v3/
    ✔ Devfile Registry          : https://192.168.49.2.nip.io/devfile-registry/
    ✔ -------------------------------------------------------------------------------
    ✔ Dex user credentials      : che@eclipse.org:admin
    ✔ Dex user credentials      : user1@che:password
    ✔ Dex user credentials      : user2@che:password
    ✔ Dex user credentials      : user3@che:password
    ✔ Dex user credentials      : user4@che:password
    ✔ Dex user credentials      : user5@che:password
    ✔ -------------------------------------------------------------------------------
Command server:deploy has completed successfully in 09:12.

When I go to http://192.168.49.2.nip.io/dashboard/ on chrome, nothing is loading up for minikube version: 1.24.0 and Eclipse Che Version: next

tolusha commented 2 years ago

yes, v1.24.0 works fine. The issue with v1.25.x

amol-m-deshpande commented 2 years ago

@tolusha For minikube v1.24.0, the output shows deploy has completed successfully, but when I go to this https://192.168.49.2.nip.io/dashboard/ The dashboard doesn't load up, I get the blank screen

tolusha commented 2 years ago

hmm. It works fine on my side Pls check che serverlogs kubectl logs deploy/che -n eclipse-che any stacktraces ?

amol-m-deshpande commented 2 years ago

@tolusha here are the logs, unable to load the dashboard

Using embedded assembly in /home/user/eclipse-che.
Found a custom cert. Adding it to java trust store /home/user/cacerts based on /opt/java/openjdk/lib/security/cacerts
Trust this certificate? [no]:  Certificate was added to keystore
Certificate was added to keystore
Imported 1 certificates from /public-certs/dex-ca.ca.crt
NOTE: Picked up JDK_JAVA_OPTIONS:  --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
17-Feb-2022 12:48:03.728 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version name:   Apache Tomcat/10.0.14
17-Feb-2022 12:48:03.732 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built:          Dec 2 2021 22:01:36 UTC
17-Feb-2022 12:48:03.733 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version number: 10.0.14.0
17-Feb-2022 12:48:03.733 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name:               Linux
17-Feb-2022 12:48:03.734 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version:            5.10.76-linuxkit
17-Feb-2022 12:48:03.734 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture:          amd64
17-Feb-2022 12:48:03.734 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home:             /opt/java/openjdk
17-Feb-2022 12:48:03.735 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version:           11.0.11+9
17-Feb-2022 12:48:03.735 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor:            AdoptOpenJDK
17-Feb-2022 12:48:03.750 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE:         /home/user/eclipse-che/tomcat
17-Feb-2022 12:48:03.752 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME:         /home/user/eclipse-che/tomcat
17-Feb-2022 12:48:03.787 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.lang=ALL-UNNAMED
17-Feb-2022 12:48:03.787 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.io=ALL-UNNAMED
17-Feb-2022 12:48:03.787 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.util=ALL-UNNAMED
17-Feb-2022 12:48:03.787 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.util.concurrent=ALL-UNNAMED
17-Feb-2022 12:48:03.787 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
17-Feb-2022 12:48:03.787 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.config.file=/home/user/eclipse-che/tomcat/conf/logging.properties
17-Feb-2022 12:48:03.787 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
17-Feb-2022 12:48:03.787 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:MaxRAMPercentage=85.0
17-Feb-2022 12:48:03.787 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dche.docker.network=bridge
17-Feb-2022 12:48:03.787 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djavax.net.ssl.trustStore=/home/user/cacerts
17-Feb-2022 12:48:03.787 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djavax.net.ssl.trustStorePassword=changeit
17-Feb-2022 12:48:03.787 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dport.http=8080
17-Feb-2022 12:48:03.788 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dche.home=/home/user/eclipse-che
17-Feb-2022 12:48:03.789 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dche.logs.dir=/logs/
17-Feb-2022 12:48:03.789 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dche.logs.level=INFO
17-Feb-2022 12:48:03.789 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djuli-logback.configurationFile=file:/home/user/eclipse-che/tomcat/conf/tomcat-logger.xml
17-Feb-2022 12:48:03.789 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djdk.tls.ephemeralDHKeySize=2048
17-Feb-2022 12:48:03.790 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
17-Feb-2022 12:48:03.790 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dorg.apache.catalina.security.SecurityListener.UMASK=0022
17-Feb-2022 12:48:03.790 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dche.local.conf.dir=/home/user/eclipse-che/tomcat/conf/
17-Feb-2022 12:48:03.790 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dignore.endorsed.dirs=
17-Feb-2022 12:48:03.791 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.base=/home/user/eclipse-che/tomcat
17-Feb-2022 12:48:03.791 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.home=/home/user/eclipse-che/tomcat
17-Feb-2022 12:48:03.791 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.io.tmpdir=/home/user/eclipse-che/tomcat/temp
17-Feb-2022 12:48:04.831 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8080"]
17-Feb-2022 12:48:04.936 INFO [main] org.apache.catalina.startup.Catalina.load Server initialization in [1901] milliseconds
17-Feb-2022 12:48:05.050 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
17-Feb-2022 12:48:05.051 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: [Apache Tomcat/10.0.14]
17-Feb-2022 12:48:05.253 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/home/user/eclipse-che/tomcat/webapps/ROOT.war]
17-Feb-2022 12:48:11.864 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/home/user/eclipse-che/tomcat/webapps/ROOT.war] has finished in [6,610] ms
17-Feb-2022 12:48:11.866 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/home/user/eclipse-che/tomcat/webapps/api.war]
2022-02-17 12:48:19,748[main]             [INFO ] [.e.c.c.d.JNDIDataSourceFactory 63]   - This=org.eclipse.che.core.db.postgresql.PostgreSQLJndiDataSourceFactory@24a10870 obj=ResourceRef[className=javax.sql.DataSource,factoryClassLocation=null,factoryClassName=org.apache.naming.factory.ResourceFactory,{type=scope,content=Shareable},{type=auth,content=Container},{type=singleton,content=true},{type=factory,content=org.eclipse.che.core.db.postgresql.PostgreSQLJndiDataSourceFactory}] name=che Context=org.apache.naming.NamingContext@70bc4941 environment={}
2022-02-17 12:48:21,748[main]             [INFO ] [.e.c.a.d.WsMasterServletModule 52]   - Running in native-user mode ...
2022-02-17 12:48:30,261[main]             [INFO ] [o.j.p.kubernetes.KUBE_PING 131]      - namespace eclipse-che set; clustering enabled

-------------------------------------------------------------------
GMS: address=che-5cd766576-w5xxr-11380, cluster=RemoteSubscriptionChannel, physical address=172.17.0.16:7800
-------------------------------------------------------------------
2022-02-17 12:48:33,840[main]             [INFO ] [o.jgroups.protocols.pbcast.GMS 125]  - che-5cd766576-w5xxr-11380: no members discovered after 3474 ms: creating cluster as coordinator
2022-02-17 12:48:33,902[main]             [INFO ] [o.j.p.kubernetes.KUBE_PING 131]      - namespace eclipse-che set; clustering enabled

-------------------------------------------------------------------
GMS: address=che-5cd766576-w5xxr-61836, cluster=WorkspaceLocks, physical address=172.17.0.16:7801
-------------------------------------------------------------------
2022-02-17 12:48:37,144[main]             [INFO ] [o.jgroups.protocols.pbcast.GMS 125]  - che-5cd766576-w5xxr-61836: no members discovered after 3205 ms: creating cluster as coordinator
2022-02-17 12:48:37,157[main]             [INFO ] [o.j.p.kubernetes.KUBE_PING 131]      - namespace eclipse-che set; clustering enabled

-------------------------------------------------------------------
GMS: address=che-5cd766576-w5xxr-7602, cluster=WorkspaceStateCache, physical address=172.17.0.16:7802
-------------------------------------------------------------------
2022-02-17 12:48:40,250[main]             [INFO ] [o.jgroups.protocols.pbcast.GMS 125]  - che-5cd766576-w5xxr-7602: no members discovered after 3027 ms: creating cluster as coordinator
2022-02-17 12:48:40,757[main]             [INFO ] [o.f.c.i.d.DbSupportFactory 44]       - Database: jdbc:postgresql://postgres:5432/dbche (PostgreSQL 13.3)
2022-02-17 12:48:40,846[main]             [INFO ] [o.f.c.i.util.VersionPrinter 44]      - Flyway 4.2.0 by Boxfuse
2022-02-17 12:48:40,855[main]             [INFO ] [o.f.c.i.d.DbSupportFactory 44]       - Database: jdbc:postgresql://postgres:5432/dbche (PostgreSQL 13.3)
2022-02-17 12:48:40,960[main]             [INFO ] [i.f.CustomSqlMigrationResolver 158]  - Searching for SQL scripts in locations [classpath:che-schema]
2022-02-17 12:48:41,140[main]             [INFO ] [o.f.c.i.command.DbValidate 44]       - Successfully validated 63 migrations (execution time 00:00.184s)
2022-02-17 12:48:41,223[main]             [INFO ] [o.f.c.i.m.MetaDataTableImpl 44]      - Creating Metadata table: "public"."schema_version"
2022-02-17 12:48:41,321[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Current version of schema "public": << Empty Schema >>
2022-02-17 12:48:41,521[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 5.0.0.8.1 - 1__init.sql
2022-02-17 12:48:41,907[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 5.0.0.9.1 - 1__add_index_on_workspace_temporary.sql
2022-02-17 12:48:41,931[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 5.0.0.9.2 - 2__update_local_links_in_environments.sql
2022-02-17 12:48:42,022[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 5.2.0.1 - 1__increase_project_attributes_values_length.sql
2022-02-17 12:48:42,108[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 5.4.0.1 - 1__drop_user_to_account_relation.sql
2022-02-17 12:48:42,144[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 5.4.0.2 - 2__create_missed_account_indexes.sql
2022-02-17 12:48:42,215[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 5.6.0.1 - 1__add_exec_agent_where_terminal_agent_is_present.sql
2022-02-17 12:48:42,307[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 5.7.0.1 - 1__add_factory.sql
2022-02-17 12:48:42,509[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 5.7.0.2 - 2__remove_match_policy.sql
2022-02-17 12:48:42,535[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 5.8.0.1 - 1__add_foreigh_key_indexes.sql
2022-02-17 12:48:42,680[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 5.11.0.1 - 1__optimize_user_search.sql
2022-02-17 12:48:42,731[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 5.19.0.0.1 - 0.1__add_permissions.sql
2022-02-17 12:48:42,924[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 5.19.0.0.2 - 0.2__add_resources.sql
2022-02-17 12:48:43,018[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 5.19.0.0.3 - 0.3__add_organization.sql
2022-02-17 12:48:43,108[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 6.0.0.1 - 1__add_path_to_serverconf.sql
2022-02-17 12:48:43,124[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 6.0.0.2 - 2__rename_agents_to_installers.sql
2022-02-17 12:48:43,144[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 6.0.0.3 - 3__add_installer.sql
2022-02-17 12:48:43,171[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 6.0.0.3.1 - 3.1__remove_old_recipe_permissions.sql
2022-02-17 12:48:43,181[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 6.0.0.4 - 4__remove_old_recipe.sql
2022-02-17 12:48:43,194[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 6.0.0.5 - 5__add_machine_env.sql
2022-02-17 12:48:43,205[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 6.0.0.6 - 6__remove_snapshots.sql
2022-02-17 12:48:43,217[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 6.0.0.7 - 7__add_machine_volumes.sql
2022-02-17 12:48:43,236[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 6.0.0.8 - 8__add_serverconf_attributes.sql
2022-02-17 12:48:43,269[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 6.0.0.9 - 9__increase_externalmachine_env_value_length.sql
2022-02-17 12:48:43,288[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 6.0.0.10 - 10__move_dockerimage_recipe_location_to_content.sql
2022-02-17 12:48:43,313[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 6.0.0.11 - 11__increase_workspace_attributes_values_length.sql
2022-02-17 12:48:43,324[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 6.0.0.12 - 12__remove_stack_sources.sql
2022-02-17 12:48:43,330[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 6.3.0.1 - 1__add_fk_indexes.sql
2022-02-17 12:48:43,344[main]             [WARN ] [o.f.c.i.dbsupport.JdbcTemplate 48]   - DB: identifier "che_index_factory_on_projects_loaded_action_value_action_entity_id" will be truncated to "che_index_factory_on_projects_loaded_action_value_action_entity" (SQL State: 42622 - Error Code: 0)
2022-02-17 12:48:43,350[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 6.3.0.1.1 - 1.1__add_fk_indexes.sql
2022-02-17 12:48:43,370[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 6.4.0.1 - 1__add_workspace_expirations.sql
2022-02-17 12:48:43,383[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 6.4.0.2 - 2__add_signature_key.sql
2022-02-17 12:48:43,401[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 6.4.0.3 - 3__add_k8s_runtimes.sql
2022-02-17 12:48:43,453[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 6.10.0.1 - 1__add_workspace_cfg_attributes.sql
2022-02-17 12:48:43,475[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 6.10.0.2 - 2__change_signature_key_pair_id.sql
2022-02-17 12:48:43,490[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 6.11.0.1 - 1__add_signature_key_constraints.sql
2022-02-17 12:48:43,507[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 6.12.0.1 - 1__rename_project_attributes_values_field.sql
2022-02-17 12:48:43,523[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 6.15.0.1 - 1__remove_not_null_constraint_from_env_name_fields.sql
2022-02-17 12:48:43,542[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 6.15.0.2 - 2__add_commands_to_k8s_runtime.sql
2022-02-17 12:48:43,582[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 6.16.0.1 - 1__increase_workspace_config_attributes_values_length.sql
2022-02-17 12:48:43,593[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 6.16.0.2 - 2__create_workspace_activity_table.sql
2022-02-17 12:48:43,619[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 6.16.0.3 - 3__bootstrap_ws_activity_data.sql
2022-02-17 12:48:43,634[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 6.17.0.1 - 1__convert_enums_to_strings.sql
2022-02-17 12:48:43,657[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 7.0.0.4.0.1 - 1__add_devfile.sql
2022-02-17 12:48:43,773[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 7.0.0.5.0.1 - 1__devfile_command_reference.sql
2022-02-17 12:48:43,792[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 7.0.0.6.0.1 - 1__add_devfile_component_prefs.sql
2022-02-17 12:48:43,824[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 7.0.0.7.0.1 - 1__add_registry_url_to_devfile_component.sql
2022-02-17 12:48:43,838[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 7.0.0.8.0.2.0.1 - 1__devfile_metadata.sql
2022-02-17 12:48:43,846[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 7.0.0.8.0.2.0.2 - 2__devfile_make_some_fields_optional.sql
2022-02-17 12:48:43,855[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 7.1.0.1 - 1__change_devfile_component_preferences_type.sql
2022-02-17 12:48:43,866[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 7.1.0.1.1 - 1.1__remove_stack_permissions.sql
2022-02-17 12:48:43,930[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 7.1.0.2 - 2__remove_stacks.sql
2022-02-17 12:48:43,946[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 7.2.0.1 - 1__remove_installers.sql
2022-02-17 12:48:43,964[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 7.4.0.1 - 1__add_devfile_source_sparse_checkout_dir.sql
2022-02-17 12:48:43,976[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 7.4.0.2 - 2__add_preview_url_to_devfile_command.sql
2022-02-17 12:48:43,995[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 7.6.0.1 - 1__drop_che_workspace_expiration.sql
2022-02-17 12:48:44,009[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 7.10.0.1 - 1__add_devfile_plugin_editor_component_cpu_limit_request.sql
2022-02-17 12:48:44,020[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 7.10.0.2 - 2__add_devfile_plugin_editor_component_ram_request.sql
2022-02-17 12:48:44,029[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 7.11.0.1 - 1__update_inconsistent_stopped_workspace_activities.sql
2022-02-17 12:48:44,044[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 7.16.0.1 - 1__add_devfile_component_automount_workspace_secrets.sql
2022-02-17 12:48:44,058[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 7.20.0.1 - 1__userdevfile.sql
2022-02-17 12:48:44,093[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 7.20.0.1.1 - 1.1__add_userdevfile_permissions.sql
2022-02-17 12:48:44,131[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 7.21.0.1 - 1__remove_installers.sql
2022-02-17 12:48:44,145[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "public" to version 7.26.0.1 - 1__remove_factory_button_and_image.sql
2022-02-17 12:48:44,164[main]             [INFO ] [o.f.c.i.command.DbMigrate 44]        - Successfully applied 63 migrations to schema "public" (execution time 00:02.942s).
2022-02-17 12:48:46,713[main]             [INFO ] [o.j.p.kubernetes.KUBE_PING 131]      - namespace eclipse-che set; clustering enabled

-------------------------------------------------------------------
GMS: address=che-5cd766576-w5xxr-13566, cluster=EclipseLinkCommandChannel, physical address=172.17.0.16:7803
-------------------------------------------------------------------
2022-02-17 12:48:49,768[main]             [INFO ] [o.jgroups.protocols.pbcast.GMS 125]  - che-5cd766576-w5xxr-13566: no members discovered after 3040 ms: creating cluster as coordinator
2022-02-17 12:48:49,823[main]             [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 182]   - Configured factories for environments: '[kubernetes, no-environment]'
2022-02-17 12:48:49,824[main]             [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 183]   - Registered infrastructure 'kubernetes'
2022-02-17 12:48:49,860[main]             [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 694]   - Infrastructure is tracking 0 active runtimes that need to be stopped
2022-02-17 12:48:49,921[main]             [INFO ] [o.e.c.m.oidc.OIDCInfoProvider 71]    - Retrieving OpenId configuration from endpoint: http://dex.dex:5556/.well-known/openid-configuration
2022-02-17 12:48:50,049[main]             [INFO ] [o.e.c.m.oidc.OIDCInfoProvider 80]    - openid configuration = {issuer=https://dex.192.168.49.2.nip.io, authorization_endpoint=https://dex.192.168.49.2.nip.io/auth, token_endpoint=https://dex.192.168.49.2.nip.io/token, jwks_uri=https://dex.192.168.49.2.nip.io/keys, userinfo_endpoint=https://dex.192.168.49.2.nip.io/userinfo, device_authorization_endpoint=https://dex.192.168.49.2.nip.io/device/code, grant_types_supported=[authorization_code, refresh_token, urn:ietf:params:oauth:grant-type:device_code], response_types_supported=[code], subject_types_supported=[public], id_token_signing_alg_values_supported=[RS256], code_challenge_methods_supported=[S256, plain], scopes_supported=[openid, email, groups, profile, offline_access], token_endpoint_auth_methods_supported=[client_secret_basic, client_secret_post], claims_supported=[iss, sub, aud, iat, exp, email, email_verified, locale, name, preferred_username, at_hash]}
2022-02-17 12:48:50,142[main]             [INFO ] [o.e.c.a.c.u.ApiInfoLogInformer 36]   - Eclipse Che Api Core: Build info '7.43.0' scmRevision '9a0a73c1500f1398796e157bbb4fdafd19c51c7c' implementationVersion '7.43.0'
2022-02-17 12:48:50,228[main]             [INFO ] [o.e.c.c.metrics.MetricsServer 46]    - Metrics server started at port 8087 successfully 
17-Feb-2022 12:48:50.821 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/home/user/eclipse-che/tomcat/webapps/api.war] has finished in [38,955] ms
17-Feb-2022 12:48:50.822 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/home/user/eclipse-che/tomcat/webapps/swagger.war]
17-Feb-2022 12:48:51.138 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/home/user/eclipse-che/tomcat/webapps/swagger.war] has finished in [316] ms
17-Feb-2022 12:48:51.147 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"]
17-Feb-2022 12:48:51.210 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [46294] milliseconds