search

eclipse-che / che

Kubernetes based Cloud Development Environments for Enterprise Teams
http://eclipse.org/che
Eclipse Public License 2.0
6.99k stars 1.19k forks source link

Can't get CHE running on Rancher K8S Cluster #18885

Closed Jonas18175 closed 3 years ago

Jonas18175 commented 3 years ago

Summary

I try to setup Eclipse CHE on a Rancher Kubernetes Cluster. I used for this the helm chart with some modifications. It's all starting and i can login in dashboard, but I can't create any new workspaces. I tried some tipps from older issues but nothing helped. How can I solve this?

Errorlog from "CHE" Container: 2021-01-25 19:14:21,228[aceSharedPool-1] [ERROR] [o.e.c.c.t.TracingInterceptor 63] - Plugins installation process timed out org.eclipse.che.api.workspace.server.spi.InfrastructureException: Plugins installation process timed out at org.eclipse.che.workspace.infrastructure.kubernetes.wsplugins.brokerphases.WaitBrokerResult.execute(WaitBrokerResult.java:75) at org.eclipse.che.workspace.infrastructure.kubernetes.wsplugins.brokerphases.DeployBroker.execute(DeployBroker.java:129) at org.eclipse.che.workspace.infrastructure.kubernetes.wsplugins.brokerphases.PrepareStorage.execute(PrepareStorage.java:76) at org.eclipse.che.workspace.infrastructure.kubernetes.wsplugins.brokerphases.ListenBrokerEvents.execute(ListenBrokerEvents.java:63) at org.eclipse.che.workspace.infrastructure.kubernetes.wsplugins.PluginBrokerManager.getTooling(PluginBrokerManager.java:129) at org.eclipse.che.core.tracing.TracingInterceptor.invoke(TracingInterceptor.java:61) at org.eclipse.che.workspace.infrastructure.kubernetes.wsplugins.SidecarToolingProvisioner.provision(SidecarToolingProvisioner.java:90) at org.eclipse.che.core.tracing.TracingInterceptor.invoke(TracingInterceptor.java:61) at org.eclipse.che.workspace.infrastructure.kubernetes.KubernetesInternalRuntime.provisionWorkspace(KubernetesInternalRuntime.java:288) at org.eclipse.che.workspace.infrastructure.kubernetes.KubernetesInternalRuntime.internalStart(KubernetesInternalRuntime.java:204) at org.eclipse.che.api.workspace.server.spi.InternalRuntime.start(InternalRuntime.java:141) at org.eclipse.che.core.tracing.TracingInterceptor.invoke(TracingInterceptor.java:61) at org.eclipse.che.api.workspace.server.WorkspaceRuntimes$StartRuntimeTask.run(WorkspaceRuntimes.java:969) at org.eclipse.che.commons.lang.concurrent.CopyThreadLocalRunnable.run(CopyThreadLocalRunnable.java:38) at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(Unknown Source) at io.opentracing.contrib.concurrent.TracedRunnable.run(TracedRunnable.java:30) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) 2021-01-25 19:14:21,239[aceSharedPool-1] [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 995] - Workspace 'admin:dotnet-oy8fw' with id 'workspacexde6yczug65tvm69' start failed 2021-01-25 19:14:22,303[ajor-endpoint-0] [ERROR] [.e.c.a.c.j.c.RequestDispatcher 62] - No corresponding to method 'unsubscribe' handler is registered 2021-01-25 19:25:27,088[aceSharedPool-2] [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 1106] - Runtime 'workspace9edw06uph3i0cj9h:default:b07e3a58-ed50-4a6e-be17-fcf49ff8b242' is stopping abnormally. Reason: Workspace is not started in time. Trying interrupt runtime start

Relevant information

tolusha commented 3 years ago

@Jonas18175 What is the version of Eclipse Che in question ?

Jonas18175 commented 3 years ago

It's the docker image che-server:7.24.2 - the helm chart is from master branch - cloned yesterday

tolusha commented 3 years ago

I wouldn't mess master helmcharts and 7.24.2. Could you check if stetting global.useInternalClusterSVCNames to false fixes the issue?

Jonas18175 commented 3 years ago

Sorry - it used the latest nighty (i tried a other with chectl installer with operator that hat the 7.24.2)

I retried again with helm but che dosen' t start anymore - i deleted the namespace for a fresh deployment. I get following stacktrace:

Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at java.base/sun.security.validator.PKIXValidator.doBuild(Unknown Source)
    at java.base/sun.security.validator.PKIXValidator.engineValidate(Unknown Source)
    at java.base/sun.security.validator.Validator.validate(Unknown Source)
    at java.base/sun.security.ssl.X509TrustManagerImpl.validate(Unknown Source)
    at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(Unknown Source)
    at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(Unknown Source)
    ... 96 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(Unknown Source)
    at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(Unknown Source)
    at java.base/java.security.cert.CertPathBuilder.build(Unknown Source)
    ... 102 more
52 errors
    at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:543)
    at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:186)
    at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:109)
    at com.google.inject.Guice.createInjector(Guice.java:87)
    at org.everrest.guice.servlet.EverrestGuiceContextListener.getInjector(EverrestGuiceContextListener.java:141)
    at com.google.inject.servlet.GuiceServletContextListener.contextInitialized(GuiceServletContextListener.java:45)
    at org.everrest.guice.servlet.EverrestGuiceContextListener.contextInitialized(EverrestGuiceContextListener.java:86)
    at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4689)
    at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5155)
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
    at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:743)
    at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:719)
    at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:705)
    at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:970)
    at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1840)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
    at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.base/java.lang.Thread.run(Unknown Source)
26-Jan-2021 13:47:02.050 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [api] registered the JDBC driver [org.postgresql.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
26-Jan-2021 13:47:02.053 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [api] appears to have started a thread named [Annotated-scheduler-0] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.8/jdk.internal.misc.Unsafe.park(Native Method)
 java.base@11.0.8/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
 java.base@11.0.8/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
 java.base@11.0.8/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
 java.base@11.0.8/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
 java.base@11.0.8/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
 java.base@11.0.8/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
 java.base@11.0.8/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
 java.base@11.0.8/java.lang.Thread.run(Unknown Source)
26-Jan-2021 13:47:02.053 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [api] appears to have started a thread named [ServerProbesTimeouts] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.8/java.lang.Object.wait(Native Method)
 java.base@11.0.8/java.lang.Object.wait(Unknown Source)
 java.base@11.0.8/java.util.TimerThread.mainLoop(Unknown Source)
 java.base@11.0.8/java.util.TimerThread.run(Unknown Source)
26-Jan-2021 13:47:02.054 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [api] appears to have started a thread named [Timer runner-3,RemoteSubscriptionChannel,che-bffc54888-tzcgx-50847] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.8/jdk.internal.misc.Unsafe.park(Native Method)
 java.base@11.0.8/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
 java.base@11.0.8/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
 java.base@11.0.8/java.util.concurrent.DelayQueue.take(Unknown Source)
 java.base@11.0.8/java.util.concurrent.DelayQueue.take(Unknown Source)
 org.jgroups.util.TimeScheduler3.run(TimeScheduler3.java:198)
 java.base@11.0.8/java.lang.Thread.run(Unknown Source)
26-Jan-2021 13:47:02.054 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [api] appears to have started a thread named [UdpDiagHandler-7,RemoteSubscriptionChannel,che-bffc54888-tzcgx-50847] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.8/java.net.PlainDatagramSocketImpl.receive0(Native Method)
 java.base@11.0.8/java.net.AbstractPlainDatagramSocketImpl.receive(Unknown Source)
 java.base@11.0.8/java.net.DatagramSocket.receive(Unknown Source)
 org.jgroups.stack.DiagnosticsHandler.runUDP(DiagnosticsHandler.java:139)
 org.jgroups.stack.DiagnosticsHandler$$Lambda$266/0x00000001006ba840.run(Unknown Source)
 org.jgroups.util.Runner.run(Runner.java:75)
 java.base@11.0.8/java.lang.Thread.run(Unknown Source)
26-Jan-2021 13:47:02.055 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [api] appears to have started a thread named [TQ-Bundler-8,RemoteSubscriptionChannel,che-bffc54888-tzcgx-50847] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.8/jdk.internal.misc.Unsafe.park(Native Method)
 java.base@11.0.8/java.util.concurrent.locks.LockSupport.park(Unknown Source)
 java.base@11.0.8/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
 java.base@11.0.8/java.util.concurrent.ArrayBlockingQueue.take(Unknown Source)
 org.jgroups.protocols.TransferQueueBundler.run(TransferQueueBundler.java:104)
 java.base@11.0.8/java.lang.Thread.run(Unknown Source)
26-Jan-2021 13:47:02.056 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [api] appears to have started a thread named [jgroups-6,RemoteSubscriptionChannel,che-bffc54888-tzcgx-50847] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.8/jdk.internal.misc.Unsafe.park(Native Method)
 java.base@11.0.8/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
 java.base@11.0.8/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
 java.base@11.0.8/java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
 java.base@11.0.8/java.util.concurrent.SynchronousQueue.poll(Unknown Source)
 java.base@11.0.8/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
 java.base@11.0.8/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
 java.base@11.0.8/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
 java.base@11.0.8/java.lang.Thread.run(Unknown Source)
26-Jan-2021 13:47:02.056 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [api] appears to have started a thread named [TcpServer.Acceptor[7800]-2,RemoteSubscriptionChannel,che-bffc54888-tzcgx-50847] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.8/java.net.PlainSocketImpl.socketAccept(Native Method)
 java.base@11.0.8/java.net.AbstractPlainSocketImpl.accept(Unknown Source)
 java.base@11.0.8/java.net.ServerSocket.implAccept(Unknown Source)
 java.base@11.0.8/java.net.ServerSocket.accept(Unknown Source)
 org.jgroups.blocks.cs.TcpServer$Acceptor.run(TcpServer.java:110)
 java.base@11.0.8/java.lang.Thread.run(Unknown Source)
26-Jan-2021 13:47:02.057 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [api] appears to have started a thread named [FD_SOCK acceptor-9,RemoteSubscriptionChannel,che-bffc54888-tzcgx-50847] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.8/java.net.PlainSocketImpl.socketAccept(Native Method)
 java.base@11.0.8/java.net.AbstractPlainSocketImpl.accept(Unknown Source)
 java.base@11.0.8/java.net.ServerSocket.implAccept(Unknown Source)
 java.base@11.0.8/java.net.ServerSocket.accept(Unknown Source)
 org.jgroups.protocols.FD_SOCK$ServerSocketHandler.run(FD_SOCK.java:993)
 java.base@11.0.8/java.lang.Thread.run(Unknown Source)
26-Jan-2021 13:47:02.066 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [api] appears to have started a thread named [Timer runner-3,WorkspaceLocks,che-bffc54888-tzcgx-56905] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.8/jdk.internal.misc.Unsafe.park(Native Method)
 java.base@11.0.8/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
 java.base@11.0.8/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
 java.base@11.0.8/java.util.concurrent.DelayQueue.take(Unknown Source)
 java.base@11.0.8/java.util.concurrent.DelayQueue.take(Unknown Source)
 org.jgroups.util.TimeScheduler3.run(TimeScheduler3.java:198)
 java.base@11.0.8/java.lang.Thread.run(Unknown Source)
26-Jan-2021 13:47:02.066 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [api] appears to have started a thread named [UdpDiagHandler-7,WorkspaceLocks,che-bffc54888-tzcgx-56905] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.8/java.net.PlainDatagramSocketImpl.receive0(Native Method)
 java.base@11.0.8/java.net.AbstractPlainDatagramSocketImpl.receive(Unknown Source)
 java.base@11.0.8/java.net.DatagramSocket.receive(Unknown Source)
 org.jgroups.stack.DiagnosticsHandler.runUDP(DiagnosticsHandler.java:139)
 org.jgroups.stack.DiagnosticsHandler$$Lambda$266/0x00000001006ba840.run(Unknown Source)
 org.jgroups.util.Runner.run(Runner.java:75)
 java.base@11.0.8/java.lang.Thread.run(Unknown Source)
26-Jan-2021 13:47:02.067 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [api] appears to have started a thread named [TQ-Bundler-8,WorkspaceLocks,che-bffc54888-tzcgx-56905] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.8/jdk.internal.misc.Unsafe.park(Native Method)
 java.base@11.0.8/java.util.concurrent.locks.LockSupport.park(Unknown Source)
 java.base@11.0.8/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
 java.base@11.0.8/java.util.concurrent.ArrayBlockingQueue.take(Unknown Source)
 org.jgroups.protocols.TransferQueueBundler.run(TransferQueueBundler.java:104)
 java.base@11.0.8/java.lang.Thread.run(Unknown Source)
26-Jan-2021 13:47:02.067 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [api] appears to have started a thread named [jgroups-6,WorkspaceLocks,che-bffc54888-tzcgx-56905] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.8/jdk.internal.misc.Unsafe.park(Native Method)
 java.base@11.0.8/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
 java.base@11.0.8/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
 java.base@11.0.8/java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
 java.base@11.0.8/java.util.concurrent.SynchronousQueue.poll(Unknown Source)
 java.base@11.0.8/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
 java.base@11.0.8/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
 java.base@11.0.8/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
 java.base@11.0.8/java.lang.Thread.run(Unknown Source)
26-Jan-2021 13:47:02.068 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [api] appears to have started a thread named [TcpServer.Acceptor[7801]-2,WorkspaceLocks,che-bffc54888-tzcgx-56905] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.8/java.net.PlainSocketImpl.socketAccept(Native Method)
 java.base@11.0.8/java.net.AbstractPlainSocketImpl.accept(Unknown Source)
 java.base@11.0.8/java.net.ServerSocket.implAccept(Unknown Source)
 java.base@11.0.8/java.net.ServerSocket.accept(Unknown Source)
 org.jgroups.blocks.cs.TcpServer$Acceptor.run(TcpServer.java:110)
 java.base@11.0.8/java.lang.Thread.run(Unknown Source)
26-Jan-2021 13:47:02.068 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [api] appears to have started a thread named [FD_SOCK acceptor-9,WorkspaceLocks,che-bffc54888-tzcgx-56905] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.8/java.net.PlainSocketImpl.socketAccept(Native Method)
 java.base@11.0.8/java.net.AbstractPlainSocketImpl.accept(Unknown Source)
 java.base@11.0.8/java.net.ServerSocket.implAccept(Unknown Source)
 java.base@11.0.8/java.net.ServerSocket.accept(Unknown Source)
 org.jgroups.protocols.FD_SOCK$ServerSocketHandler.run(FD_SOCK.java:993)
 java.base@11.0.8/java.lang.Thread.run(Unknown Source)
26-Jan-2021 13:47:02.069 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [api] appears to have started a thread named [Timer runner-3,WorkspaceStateCache,che-bffc54888-tzcgx-27693] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.8/jdk.internal.misc.Unsafe.park(Native Method)
 java.base@11.0.8/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
 java.base@11.0.8/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
 java.base@11.0.8/java.util.concurrent.DelayQueue.take(Unknown Source)
 java.base@11.0.8/java.util.concurrent.DelayQueue.take(Unknown Source)
 org.jgroups.util.TimeScheduler3.run(TimeScheduler3.java:198)
 java.base@11.0.8/java.lang.Thread.run(Unknown Source)
26-Jan-2021 13:47:02.069 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [api] appears to have started a thread named [UdpDiagHandler-7,WorkspaceStateCache,che-bffc54888-tzcgx-27693] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.8/java.net.PlainDatagramSocketImpl.receive0(Native Method)
 java.base@11.0.8/java.net.AbstractPlainDatagramSocketImpl.receive(Unknown Source)
 java.base@11.0.8/java.net.DatagramSocket.receive(Unknown Source)
 org.jgroups.stack.DiagnosticsHandler.runUDP(DiagnosticsHandler.java:139)
 org.jgroups.stack.DiagnosticsHandler$$Lambda$266/0x00000001006ba840.run(Unknown Source)
 org.jgroups.util.Runner.run(Runner.java:75)
 java.base@11.0.8/java.lang.Thread.run(Unknown Source)
26-Jan-2021 13:47:02.072 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [api] appears to have started a thread named [TQ-Bundler-8,WorkspaceStateCache,che-bffc54888-tzcgx-27693] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.8/jdk.internal.misc.Unsafe.park(Native Method)
 java.base@11.0.8/java.util.concurrent.locks.LockSupport.park(Unknown Source)
 java.base@11.0.8/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
 java.base@11.0.8/java.util.concurrent.ArrayBlockingQueue.take(Unknown Source)
 org.jgroups.protocols.TransferQueueBundler.run(TransferQueueBundler.java:104)
 java.base@11.0.8/java.lang.Thread.run(Unknown Source)
26-Jan-2021 13:47:02.075 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [api] appears to have started a thread named [jgroups-6,WorkspaceStateCache,che-bffc54888-tzcgx-27693] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.8/jdk.internal.misc.Unsafe.park(Native Method)
 java.base@11.0.8/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
 java.base@11.0.8/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
 java.base@11.0.8/java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
 java.base@11.0.8/java.util.concurrent.SynchronousQueue.poll(Unknown Source)
 java.base@11.0.8/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
 java.base@11.0.8/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
 java.base@11.0.8/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
 java.base@11.0.8/java.lang.Thread.run(Unknown Source)
26-Jan-2021 13:47:02.075 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [api] appears to have started a thread named [TcpServer.Acceptor[7802]-2,WorkspaceStateCache,che-bffc54888-tzcgx-27693] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.8/java.net.PlainSocketImpl.socketAccept(Native Method)
 java.base@11.0.8/java.net.AbstractPlainSocketImpl.accept(Unknown Source)
 java.base@11.0.8/java.net.ServerSocket.implAccept(Unknown Source)
 java.base@11.0.8/java.net.ServerSocket.accept(Unknown Source)
 org.jgroups.blocks.cs.TcpServer$Acceptor.run(TcpServer.java:110)
 java.base@11.0.8/java.lang.Thread.run(Unknown Source)
26-Jan-2021 13:47:02.076 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [api] appears to have started a thread named [FD_SOCK acceptor-9,WorkspaceStateCache,che-bffc54888-tzcgx-27693] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.8/java.net.PlainSocketImpl.socketAccept(Native Method)
 java.base@11.0.8/java.net.AbstractPlainSocketImpl.accept(Unknown Source)
 java.base@11.0.8/java.net.ServerSocket.implAccept(Unknown Source)
 java.base@11.0.8/java.net.ServerSocket.accept(Unknown Source)
 org.jgroups.protocols.FD_SOCK$ServerSocketHandler.run(FD_SOCK.java:993)
 java.base@11.0.8/java.lang.Thread.run(Unknown Source)
26-Jan-2021 13:47:02.077 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [api] appears to have started a thread named [Timer runner-3,EclipseLinkCommandChannel,che-bffc54888-tzcgx-22986] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.8/jdk.internal.misc.Unsafe.park(Native Method)
 java.base@11.0.8/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
 java.base@11.0.8/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
 java.base@11.0.8/java.util.concurrent.DelayQueue.take(Unknown Source)
 java.base@11.0.8/java.util.concurrent.DelayQueue.take(Unknown Source)
 org.jgroups.util.TimeScheduler3.run(TimeScheduler3.java:198)
 java.base@11.0.8/java.lang.Thread.run(Unknown Source)
26-Jan-2021 13:47:02.077 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [api] appears to have started a thread named [UdpDiagHandler-7,EclipseLinkCommandChannel,che-bffc54888-tzcgx-22986] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.8/java.net.PlainDatagramSocketImpl.receive0(Native Method)
 java.base@11.0.8/java.net.AbstractPlainDatagramSocketImpl.receive(Unknown Source)
 java.base@11.0.8/java.net.DatagramSocket.receive(Unknown Source)
 org.jgroups.stack.DiagnosticsHandler.runUDP(DiagnosticsHandler.java:139)
 org.jgroups.stack.DiagnosticsHandler$$Lambda$266/0x00000001006ba840.run(Unknown Source)
 org.jgroups.util.Runner.run(Runner.java:75)
 java.base@11.0.8/java.lang.Thread.run(Unknown Source)
26-Jan-2021 13:47:02.078 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [api] appears to have started a thread named [TQ-Bundler-8,EclipseLinkCommandChannel,che-bffc54888-tzcgx-22986] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.8/jdk.internal.misc.Unsafe.park(Native Method)
 java.base@11.0.8/java.util.concurrent.locks.LockSupport.park(Unknown Source)
 java.base@11.0.8/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
 java.base@11.0.8/java.util.concurrent.ArrayBlockingQueue.take(Unknown Source)
 org.jgroups.protocols.TransferQueueBundler.run(TransferQueueBundler.java:104)
 java.base@11.0.8/java.lang.Thread.run(Unknown Source)
26-Jan-2021 13:47:02.078 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [api] appears to have started a thread named [jgroups-6,EclipseLinkCommandChannel,che-bffc54888-tzcgx-22986] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.8/jdk.internal.misc.Unsafe.park(Native Method)
 java.base@11.0.8/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
 java.base@11.0.8/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
 java.base@11.0.8/java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
 java.base@11.0.8/java.util.concurrent.SynchronousQueue.poll(Unknown Source)
 java.base@11.0.8/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
 java.base@11.0.8/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
 java.base@11.0.8/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
 java.base@11.0.8/java.lang.Thread.run(Unknown Source)
26-Jan-2021 13:47:02.079 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [api] appears to have started a thread named [TcpServer.Acceptor[7803]-2,EclipseLinkCommandChannel,che-bffc54888-tzcgx-22986] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.8/java.net.PlainSocketImpl.socketAccept(Native Method)
 java.base@11.0.8/java.net.AbstractPlainSocketImpl.accept(Unknown Source)
 java.base@11.0.8/java.net.ServerSocket.implAccept(Unknown Source)
 java.base@11.0.8/java.net.ServerSocket.accept(Unknown Source)
 org.jgroups.blocks.cs.TcpServer$Acceptor.run(TcpServer.java:110)
 java.base@11.0.8/java.lang.Thread.run(Unknown Source)
26-Jan-2021 13:47:02.079 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [api] appears to have started a thread named [FD_SOCK acceptor-9,EclipseLinkCommandChannel,che-bffc54888-tzcgx-22986] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.8/java.net.PlainSocketImpl.socketAccept(Native Method)
 java.base@11.0.8/java.net.AbstractPlainSocketImpl.accept(Unknown Source)
 java.base@11.0.8/java.net.ServerSocket.implAccept(Unknown Source)
 java.base@11.0.8/java.net.ServerSocket.accept(Unknown Source)
 org.jgroups.protocols.FD_SOCK$ServerSocketHandler.run(FD_SOCK.java:993)
 java.base@11.0.8/java.lang.Thread.run(Unknown Source)
26-Jan-2021 13:47:02.095 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/home/user/eclipse-che/tomcat/webapps/api.war] has finished in [34,439] ms
26-Jan-2021 13:47:02.097 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/home/user/eclipse-che/tomcat/webapps/dashboard]
26-Jan-2021 13:47:02.139 INFO [localhost-startStop-1] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
26-Jan-2021 13:47:02.143 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/home/user/eclipse-che/tomcat/webapps/dashboard] has finished in [46] ms
26-Jan-2021 13:47:02.143 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/home/user/eclipse-che/tomcat/webapps/workspace-loader]
26-Jan-2021 13:47:02.163 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/home/user/eclipse-che/tomcat/webapps/workspace-loader] has finished in [20] ms
26-Jan-2021 13:47:02.168 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"]
26-Jan-2021 13:47:02.181 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 37915 ms
26-Jan-2021 13:47:02.544 INFO [jgroups-6,WorkspaceLocks,che-bffc54888-tzcgx-56905] org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading Illegal access: this web application instance has been stopped already. Could not load [org.jgroups.util.Headers]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
    java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [org.jgroups.util.Headers]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
        at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1378)
        at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1366)
        at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1218)
        at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1180)
        at org.jgroups.Message.putHeader(Message.java:466)
        at org.jgroups.protocols.pbcast.STABLE.sendStableMessage(STABLE.java:667)
        at org.jgroups.protocols.pbcast.STABLE$StableTask.run(STABLE.java:832)
        at org.jgroups.util.TimeScheduler3$Task.run(TimeScheduler3.java:328)
        at org.jgroups.util.TimeScheduler3$RecurringTask.run(TimeScheduler3.java:362)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
26-Jan-2021 13:47:02.548 INFO [jgroups-6,WorkspaceLocks,che-bffc54888-tzcgx-56905] org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading Illegal access: this web application instance has been stopped already. Could not load [ch.qos.logback.core.status.WarnStatus]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
    java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [ch.qos.logback.core.status.WarnStatus]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
        at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1378)
        at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1366)
        at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1218)
        at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1180)
        at ch.qos.logback.classic.LoggerContext.noAppenderDefinedWarning(LoggerContext.java:186)
        at ch.qos.logback.classic.Logger.callAppenders(Logger.java:264)
        at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:421)
        at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:383)
        at ch.qos.logback.classic.Logger.error(Logger.java:538)
        at org.jgroups.logging.Slf4jLogImpl.error(Slf4jLogImpl.java:104)
        at org.jgroups.util.TimeScheduler3$Task.run(TimeScheduler3.java:331)
        at org.jgroups.util.TimeScheduler3$RecurringTask.run(TimeScheduler3.java:362)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
Received SIGTERM
Stopping Che server running on localhost:8080

The problem is i have a custom wildcard certificate and i like to use it. With operator i tried to insert it with secret, the website works, but when i try to start a workspace i get a certificate error from websocket in the workspace start process container. (I have inserted the custom cert chain and verified that the bundle is available.

I thought i had more success with the helm files ...

Jonas18175 commented 3 years ago

Any help here?

che-bot commented 3 years ago

Issues go stale after 180 days of inactivity. lifecycle/stale issues rot after an additional 7 days of inactivity and eventually close.

Mark the issue as fresh with /remove-lifecycle stale in a new comment.

If this issue is safe to close now please do so.

Moderators: Add lifecycle/frozen label to avoid stale mode.

gorshkov-leonid commented 3 years ago

I have a similar problem with the same stack. All I have done - moved from 7.32.1 to 7.34.1. CA certificate specified in serverTrustStoreConfigMapName is the same as earlier. It is active.
To install to k8s I use che-operator. I know about useInternalClusterSVCNames, and as I understand - it is false by default in 7.34.1. How I can look deeper?

PS. Rolled back to 7.32.1 and can see the same situation. I don’t understand anything. It had been worked earlier. But now after enabling debug(Configuring server logging) I found lines in container of Che

[DEBUG] [.h.HttpConnectionServerChecker 116]  - Failed to establish http connection to check server 'theia:https://serverul9q6ui6-jwtproxy-server-4402.a.my.cloud/'. Cause: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
2021-08-12 18:56:19,685[ServersChecker]

Really there is no necessary CA certificate:

keytool -v -list -keystore /opt/java/openjdk/lib/security/cacerts
Enter keystore password:  changeit

... as here:

keytool -v -list -keystore /home/user/cacerts

But this certificate is filled in ConfigMap with the name specified in serverTrustStoreConfigMapName as well as in ca-certs-merged

Then server:logs --chenamespace=che-new --directory=.. Opened ./che-xx-xx/che.log(and keycloak.log after that):

Imported 0 certificates from /public-certs/che-additional-certificates.ca.crt

Checked /public-certs/che-additional-certificates.ca.crt - is OK, in PEM format(checked it with tool) Actually is linked from another folder /public-certs/..2021_08_12_18_52_32.594885990/che-additional-certificates.ca.crt, but it is valid certificate in PEM format

gorshkov-leonid commented 3 years ago

@tolusha maybe you know something about the possible reason?

gorshkov-leonid commented 3 years ago

@tolusha It seems I've found a bug. All works if a certificate has \n at the end, But in my case, I was using a certificate that ends with "...END CERTIFICATE-----\<EOF>"