search

OpenUnison / openunison-k8s-login-oidc

Kubernetes login portal for both kubectl and the dashboard using OpenID Connect. Use groups from your assertion in RBAC policies to control access to your cluster. Supports impersonation and OpenID Connect integration with your API server.
https://www.tremolosecurity.com/kubernetes/
Apache License 2.0
12 stars 5 forks source link

kubectl exec|port-forward fails when using api impersonation #26

Closed dkulchinsky closed 3 years ago

dkulchinsky commented 3 years ago

Hey 👋

We've encountered an issue with the api impersonation mode, it seems that kubectl exec & port-forward do not work (otherwise things are working as expected).

❯ kubectl port-forward -n vault vault-0 8200:8200

error: error upgrading connection: unable to upgrade connection: <html><head><title>Error</title></head><body>Internal Server Error</body></html>
❯ kubectl exec -n vault vault-0 -- bash
Defaulting container name to vault-init.
Use 'kubectl describe pod/vault-0 -n vault' to see all of the containers in this pod.

error: unable to upgrade connection: <html><head><title>Error</title></head><body>Internal Server Error</body></html>

on the orchestra pod I see the following error:

[openunison-orchestra-74f5ffbf54-vlqcj] [2020-11-27 19:24:50,715][XNIO-1 task-14] ERROR ConfigSys - Could not process request
[openunison-orchestra-74f5ffbf54-vlqcj] javax.net.ssl.SSLException: Socket closed

followed by a pretty long exception:

Full exception

``` [openunison-orchestra-74f5ffbf54-vlqcj] [2020-11-27 19:24:50,715][XNIO-1 task-14] ERROR ConfigSys - Could not process request [openunison-orchestra-74f5ffbf54-vlqcj] javax.net.ssl.SSLException: Socket closed [openunison-orchestra-74f5ffbf54-vlqcj] at sun.security.ssl.Alert.createSSLException(Alert.java:127) ~[?:1.8.0_275] [openunison-orchestra-74f5ffbf54-vlqcj] at sun.security.ssl.TransportContext.fatal(TransportContext.java:324) ~[?:1.8.0_275] [openunison-orchestra-74f5ffbf54-vlqcj] at sun.security.ssl.TransportContext.fatal(TransportContext.java:267) ~[?:1.8.0_275] [openunison-orchestra-74f5ffbf54-vlqcj] at sun.security.ssl.TransportContext.fatal(TransportContext.java:262) ~[?:1.8.0_275] [openunison-orchestra-74f5ffbf54-vlqcj] at sun.security.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1303) ~[?:1.8.0_275] [openunison-orchestra-74f5ffbf54-vlqcj] at sun.security.ssl.SSLSocketImpl.access$300(SSLSocketImpl.java:72) ~[?:1.8.0_275] [openunison-orchestra-74f5ffbf54-vlqcj] at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:831) ~[?:1.8.0_275] [openunison-orchestra-74f5ffbf54-vlqcj] at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) ~[httpcore-4.4.13.jar:4.4.13] [openunison-orchestra-74f5ffbf54-vlqcj] at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) ~[httpcore-4.4.13.jar:4.4.13] [openunison-orchestra-74f5ffbf54-vlqcj] at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280) ~[httpcore-4.4.13.jar:4.4.13] [openunison-orchestra-74f5ffbf54-vlqcj] at org.apache.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:261) ~[httpcore-4.4.13.jar:4.4.13] [openunison-orchestra-74f5ffbf54-vlqcj] at org.apache.http.impl.io.ChunkedInputStream.nextChunk(ChunkedInputStream.java:222) ~[httpcore-4.4.13.jar:4.4.13] [openunison-orchestra-74f5ffbf54-vlqcj] at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:183) ~[httpcore-4.4.13.jar:4.4.13] [openunison-orchestra-74f5ffbf54-vlqcj] at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:135) ~[httpclient-4.5.12.jar:4.5.12] [openunison-orchestra-74f5ffbf54-vlqcj] at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:148) ~[httpclient-4.5.12.jar:4.5.12] [openunison-orchestra-74f5ffbf54-vlqcj] at com.tremolosecurity.proxy.ConfigSys.procData(ConfigSys.java:463) ~[unison-server-core-1.0.20.jar:?] [openunison-orchestra-74f5ffbf54-vlqcj] at com.tremolosecurity.proxy.ConfigSys.doConfig(ConfigSys.java:332) [unison-server-core-1.0.20.jar:?] [openunison-orchestra-74f5ffbf54-vlqcj] at com.tremolosecurity.embedd.NextEmbSys.nextSys(NextEmbSys.java:93) [unison-server-core-1.0.20.jar:?] [openunison-orchestra-74f5ffbf54-vlqcj] at com.tremolosecurity.filter.UnisonServletFilter.doFilter(UnisonServletFilter.java:290) [unison-server-core-1.0.20.jar:?] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) [undertow-core-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) [undertow-core-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) [undertow-core-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:269) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:78) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:133) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:130) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:249) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:78) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:99) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:370) [undertow-core-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) [undertow-core-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [jboss-threads-3.1.0.Final.jar:3.1.0.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:2019) [jboss-threads-3.1.0.Final.jar:3.1.0.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1558) [jboss-threads-3.1.0.Final.jar:3.1.0.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1449) [jboss-threads-3.1.0.Final.jar:3.1.0.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275] [openunison-orchestra-74f5ffbf54-vlqcj] Caused by: java.net.SocketException: Socket closed [openunison-orchestra-74f5ffbf54-vlqcj] at java.net.SocketInputStream.socketRead0(Native Method) ~[?:1.8.0_275] [openunison-orchestra-74f5ffbf54-vlqcj] at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) ~[?:1.8.0_275] [openunison-orchestra-74f5ffbf54-vlqcj] at java.net.SocketInputStream.read(SocketInputStream.java:171) ~[?:1.8.0_275] [openunison-orchestra-74f5ffbf54-vlqcj] at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_275] [openunison-orchestra-74f5ffbf54-vlqcj] at sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:457) ~[?:1.8.0_275] [openunison-orchestra-74f5ffbf54-vlqcj] at sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:68) ~[?:1.8.0_275] [openunison-orchestra-74f5ffbf54-vlqcj] at sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1095) ~[?:1.8.0_275] [openunison-orchestra-74f5ffbf54-vlqcj] at sun.security.ssl.SSLSocketImpl.access$200(SSLSocketImpl.java:72) ~[?:1.8.0_275] [openunison-orchestra-74f5ffbf54-vlqcj] at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:815) ~[?:1.8.0_275] [openunison-orchestra-74f5ffbf54-vlqcj] ... 45 more [openunison-orchestra-74f5ffbf54-vlqcj] [2020-11-27 19:24:50,721][XNIO-1 task-14] ERROR UnisonServletFilter - Could not process request [openunison-orchestra-74f5ffbf54-vlqcj] java.lang.IllegalStateException: UT010019: Response already commited [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.spec.ServletOutputStreamImpl.resetBuffer(ServletOutputStreamImpl.java:739) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.spec.HttpServletResponseImpl.resetBuffer(HttpServletResponseImpl.java:550) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImpl(RequestDispatcherImpl.java:169) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImplSetup(RequestDispatcherImpl.java:149) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.spec.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:111) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at com.tremolosecurity.proxy.ConfigSys.doConfig(ConfigSys.java:379) ~[unison-server-core-1.0.20.jar:?] [openunison-orchestra-74f5ffbf54-vlqcj] at com.tremolosecurity.embedd.NextEmbSys.nextSys(NextEmbSys.java:93) ~[unison-server-core-1.0.20.jar:?] [openunison-orchestra-74f5ffbf54-vlqcj] at com.tremolosecurity.filter.UnisonServletFilter.doFilter(UnisonServletFilter.java:290) [unison-server-core-1.0.20.jar:?] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) [undertow-core-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) [undertow-core-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) [undertow-core-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:269) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:78) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:133) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:130) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:249) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:78) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:99) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:370) [undertow-core-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) [undertow-core-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [jboss-threads-3.1.0.Final.jar:3.1.0.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:2019) [jboss-threads-3.1.0.Final.jar:3.1.0.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1558) [jboss-threads-3.1.0.Final.jar:3.1.0.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1449) [jboss-threads-3.1.0.Final.jar:3.1.0.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275] [openunison-orchestra-74f5ffbf54-vlqcj] [2020-11-27 19:24:50,719][XNIO-1 task-10] ERROR ConfigSys - Could not process request [openunison-orchestra-74f5ffbf54-vlqcj] javax.servlet.ServletException: javax.net.ssl.SSLException: Socket closed [openunison-orchestra-74f5ffbf54-vlqcj] at com.tremolosecurity.proxy.ProxySys.doPush(ProxySys.java:184) ~[unison-server-core-1.0.20.jar:?] [openunison-orchestra-74f5ffbf54-vlqcj] at com.tremolosecurity.embedd.NextEmbSys.nextSys(NextEmbSys.java:139) ~[unison-server-core-1.0.20.jar:?] [openunison-orchestra-74f5ffbf54-vlqcj] at com.tremolosecurity.proxy.auth.AuthMgrSys.doAuthMgr(AuthMgrSys.java:135) ~[unison-server-core-1.0.20.jar:?] [openunison-orchestra-74f5ffbf54-vlqcj] at com.tremolosecurity.embedd.NextEmbSys.nextSys(NextEmbSys.java:126) ~[unison-server-core-1.0.20.jar:?] [openunison-orchestra-74f5ffbf54-vlqcj] at com.tremolosecurity.proxy.auth.AzSys.doAz(AzSys.java:139) ~[unison-sdk-1.0.20.jar:?] [openunison-orchestra-74f5ffbf54-vlqcj] at com.tremolosecurity.embedd.NextEmbSys.nextSys(NextEmbSys.java:111) ~[unison-server-core-1.0.20.jar:?] [openunison-orchestra-74f5ffbf54-vlqcj] at com.tremolosecurity.proxy.auth.sys.AuthManagerImpl.finishSuccessfulLogin(AuthManagerImpl.java:731) ~[unison-server-core-1.0.20.jar:?] [openunison-orchestra-74f5ffbf54-vlqcj] at com.tremolosecurity.proxy.auth.sys.AuthManagerImpl.execAuth(AuthManagerImpl.java:453) ~[unison-server-core-1.0.20.jar:?] [openunison-orchestra-74f5ffbf54-vlqcj] at com.tremolosecurity.proxy.auth.sys.AuthManagerImpl.nextAuth(AuthManagerImpl.java:125) ~[unison-server-core-1.0.20.jar:?] [openunison-orchestra-74f5ffbf54-vlqcj] at com.tremolosecurity.proxy.auth.sys.AuthManagerImpl.nextAuth(AuthManagerImpl.java:83) ~[unison-server-core-1.0.20.jar:?] [openunison-orchestra-74f5ffbf54-vlqcj] at com.tremolosecurity.proxy.auth.oauth2.OAuth2JWT.processToken(OAuth2JWT.java:260) ~[unison-server-core-1.0.20.jar:?] [openunison-orchestra-74f5ffbf54-vlqcj] at com.tremolosecurity.proxy.auth.oauth2.OAuth2Bearer.doGet(OAuth2Bearer.java:110) ~[unison-sdk-1.0.20.jar:?] [openunison-orchestra-74f5ffbf54-vlqcj] at com.tremolosecurity.proxy.auth.oauth2.OAuth2Bearer.doPost(OAuth2Bearer.java:145) ~[unison-sdk-1.0.20.jar:?] [openunison-orchestra-74f5ffbf54-vlqcj] at com.tremolosecurity.proxy.auth.sys.AuthManagerImpl.execAuth(AuthManagerImpl.java:412) ~[unison-server-core-1.0.20.jar:?] [openunison-orchestra-74f5ffbf54-vlqcj] at com.tremolosecurity.proxy.auth.sys.AuthManagerImpl.nextAuth(AuthManagerImpl.java:125) ~[unison-server-core-1.0.20.jar:?] [openunison-orchestra-74f5ffbf54-vlqcj] at com.tremolosecurity.proxy.auth.AuthSys.doAuth(AuthSys.java:159) ~[unison-server-core-1.0.20.jar:?] [openunison-orchestra-74f5ffbf54-vlqcj] at com.tremolosecurity.embedd.NextEmbSys.nextSys(NextEmbSys.java:105) ~[unison-server-core-1.0.20.jar:?] [openunison-orchestra-74f5ffbf54-vlqcj] at com.tremolosecurity.proxy.ConfigSys.doConfig(ConfigSys.java:293) [unison-server-core-1.0.20.jar:?] [openunison-orchestra-74f5ffbf54-vlqcj] at com.tremolosecurity.embedd.NextEmbSys.nextSys(NextEmbSys.java:93) [unison-server-core-1.0.20.jar:?] [openunison-orchestra-74f5ffbf54-vlqcj] at com.tremolosecurity.filter.UnisonServletFilter.doFilter(UnisonServletFilter.java:290) [unison-server-core-1.0.20.jar:?] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) [undertow-core-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) [undertow-core-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) [undertow-core-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:269) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:78) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:133) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:130) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:249) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:78) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:99) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:370) [undertow-core-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) [undertow-core-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [jboss-threads-3.1.0.Final.jar:3.1.0.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:2019) [jboss-threads-3.1.0.Final.jar:3.1.0.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1558) [jboss-threads-3.1.0.Final.jar:3.1.0.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1449) [jboss-threads-3.1.0.Final.jar:3.1.0.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275] [openunison-orchestra-74f5ffbf54-vlqcj] Caused by: javax.net.ssl.SSLException: Socket closed [openunison-orchestra-74f5ffbf54-vlqcj] at sun.security.ssl.Alert.createSSLException(Alert.java:127) ~[?:1.8.0_275] [openunison-orchestra-74f5ffbf54-vlqcj] at sun.security.ssl.TransportContext.fatal(TransportContext.java:324) ~[?:1.8.0_275] [openunison-orchestra-74f5ffbf54-vlqcj] at sun.security.ssl.TransportContext.fatal(TransportContext.java:267) ~[?:1.8.0_275] [openunison-orchestra-74f5ffbf54-vlqcj] at sun.security.ssl.TransportContext.fatal(TransportContext.java:262) ~[?:1.8.0_275] [openunison-orchestra-74f5ffbf54-vlqcj] at sun.security.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1303) ~[?:1.8.0_275] [openunison-orchestra-74f5ffbf54-vlqcj] at sun.security.ssl.SSLSocketImpl.access$300(SSLSocketImpl.java:72) ~[?:1.8.0_275] [openunison-orchestra-74f5ffbf54-vlqcj] at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:831) ~[?:1.8.0_275] [openunison-orchestra-74f5ffbf54-vlqcj] at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) ~[httpcore-4.4.13.jar:4.4.13] [openunison-orchestra-74f5ffbf54-vlqcj] at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) ~[httpcore-4.4.13.jar:4.4.13] [openunison-orchestra-74f5ffbf54-vlqcj] at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280) ~[httpcore-4.4.13.jar:4.4.13] [openunison-orchestra-74f5ffbf54-vlqcj] at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) ~[httpclient-4.5.12.jar:4.5.12] [openunison-orchestra-74f5ffbf54-vlqcj] at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) ~[httpclient-4.5.12.jar:4.5.12] [openunison-orchestra-74f5ffbf54-vlqcj] at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) ~[httpcore-4.4.13.jar:4.4.13] [openunison-orchestra-74f5ffbf54-vlqcj] at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) ~[httpcore-4.4.13.jar:4.4.13] [openunison-orchestra-74f5ffbf54-vlqcj] at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157) ~[httpclient-4.5.12.jar:4.5.12] [openunison-orchestra-74f5ffbf54-vlqcj] at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) ~[httpcore-4.4.13.jar:4.4.13] [openunison-orchestra-74f5ffbf54-vlqcj] at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) ~[httpcore-4.4.13.jar:4.4.13] [openunison-orchestra-74f5ffbf54-vlqcj] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) ~[httpclient-4.5.12.jar:4.5.12] [openunison-orchestra-74f5ffbf54-vlqcj] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.12.jar:4.5.12] [openunison-orchestra-74f5ffbf54-vlqcj] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.12.jar:4.5.12] [openunison-orchestra-74f5ffbf54-vlqcj] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.12.jar:4.5.12] [openunison-orchestra-74f5ffbf54-vlqcj] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.12.jar:4.5.12] [openunison-orchestra-74f5ffbf54-vlqcj] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.12.jar:4.5.12] [openunison-orchestra-74f5ffbf54-vlqcj] at com.tremolosecurity.proxy.postProcess.PushRequestProcess.postProcess(PushRequestProcess.java:193) ~[unison-server-core-1.0.20.jar:?] [openunison-orchestra-74f5ffbf54-vlqcj] at com.tremolosecurity.proxy.filter.HttpFilterChainImpl.nextFilter(HttpFilterChainImpl.java:92) ~[unison-server-core-1.0.20.jar:?] [openunison-orchestra-74f5ffbf54-vlqcj] at com.tremolosecurity.proxy.filters.K8sInjectImpersonation.doFilter(K8sInjectImpersonation.java:92) ~[unison-applications-k8s-1.0.20.jar:?] [openunison-orchestra-74f5ffbf54-vlqcj] at com.tremolosecurity.proxy.filter.HttpFilterChainImpl.nextFilter(HttpFilterChainImpl.java:86) ~[unison-server-core-1.0.20.jar:?] [openunison-orchestra-74f5ffbf54-vlqcj] at com.tremolosecurity.proxy.ProxySys.doPush(ProxySys.java:181) ~[unison-server-core-1.0.20.jar:?] [openunison-orchestra-74f5ffbf54-vlqcj] ... 52 more [openunison-orchestra-74f5ffbf54-vlqcj] Caused by: java.net.SocketException: Socket closed [openunison-orchestra-74f5ffbf54-vlqcj] at java.net.SocketInputStream.socketRead0(Native Method) ~[?:1.8.0_275] [openunison-orchestra-74f5ffbf54-vlqcj] at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) ~[?:1.8.0_275] [openunison-orchestra-74f5ffbf54-vlqcj] at java.net.SocketInputStream.read(SocketInputStream.java:171) ~[?:1.8.0_275] [openunison-orchestra-74f5ffbf54-vlqcj] at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_275] [openunison-orchestra-74f5ffbf54-vlqcj] at sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:457) ~[?:1.8.0_275] [openunison-orchestra-74f5ffbf54-vlqcj] at sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:68) ~[?:1.8.0_275] [openunison-orchestra-74f5ffbf54-vlqcj] at sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1095) ~[?:1.8.0_275] [openunison-orchestra-74f5ffbf54-vlqcj] at sun.security.ssl.SSLSocketImpl.access$200(SSLSocketImpl.java:72) ~[?:1.8.0_275] [openunison-orchestra-74f5ffbf54-vlqcj] at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:815) ~[?:1.8.0_275] [openunison-orchestra-74f5ffbf54-vlqcj] at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) ~[httpcore-4.4.13.jar:4.4.13] [openunison-orchestra-74f5ffbf54-vlqcj] at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) ~[httpcore-4.4.13.jar:4.4.13] [openunison-orchestra-74f5ffbf54-vlqcj] at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280) ~[httpcore-4.4.13.jar:4.4.13] [openunison-orchestra-74f5ffbf54-vlqcj] at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) ~[httpclient-4.5.12.jar:4.5.12] [openunison-orchestra-74f5ffbf54-vlqcj] at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) ~[httpclient-4.5.12.jar:4.5.12] [openunison-orchestra-74f5ffbf54-vlqcj] at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) ~[httpcore-4.4.13.jar:4.4.13] [openunison-orchestra-74f5ffbf54-vlqcj] at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) ~[httpcore-4.4.13.jar:4.4.13] [openunison-orchestra-74f5ffbf54-vlqcj] at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157) ~[httpclient-4.5.12.jar:4.5.12] [openunison-orchestra-74f5ffbf54-vlqcj] at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) ~[httpcore-4.4.13.jar:4.4.13] [openunison-orchestra-74f5ffbf54-vlqcj] at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) ~[httpcore-4.4.13.jar:4.4.13] [openunison-orchestra-74f5ffbf54-vlqcj] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) ~[httpclient-4.5.12.jar:4.5.12] [openunison-orchestra-74f5ffbf54-vlqcj] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.12.jar:4.5.12] [openunison-orchestra-74f5ffbf54-vlqcj] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.12.jar:4.5.12] [openunison-orchestra-74f5ffbf54-vlqcj] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.12.jar:4.5.12] [openunison-orchestra-74f5ffbf54-vlqcj] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.12.jar:4.5.12] [openunison-orchestra-74f5ffbf54-vlqcj] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.12.jar:4.5.12] [openunison-orchestra-74f5ffbf54-vlqcj] at com.tremolosecurity.proxy.postProcess.PushRequestProcess.postProcess(PushRequestProcess.java:193) ~[unison-server-core-1.0.20.jar:?] [openunison-orchestra-74f5ffbf54-vlqcj] at com.tremolosecurity.proxy.filter.HttpFilterChainImpl.nextFilter(HttpFilterChainImpl.java:92) ~[unison-server-core-1.0.20.jar:?] [openunison-orchestra-74f5ffbf54-vlqcj] at com.tremolosecurity.proxy.filters.K8sInjectImpersonation.doFilter(K8sInjectImpersonation.java:92) ~[unison-applications-k8s-1.0.20.jar:?] [openunison-orchestra-74f5ffbf54-vlqcj] at com.tremolosecurity.proxy.filter.HttpFilterChainImpl.nextFilter(HttpFilterChainImpl.java:86) ~[unison-server-core-1.0.20.jar:?] [openunison-orchestra-74f5ffbf54-vlqcj] at com.tremolosecurity.proxy.ProxySys.doPush(ProxySys.java:181) ~[unison-server-core-1.0.20.jar:?] [openunison-orchestra-74f5ffbf54-vlqcj] ... 52 more [openunison-orchestra-74f5ffbf54-vlqcj] [2020-11-27 19:24:50,723][XNIO-1 task-14] ERROR request - UT005023: Exception handling request to /api/v1/namespaces/openunison/pods/openunison-orchestra-74f5ffbf54-vlqcj/log [openunison-orchestra-74f5ffbf54-vlqcj] java.lang.IllegalStateException: UT010019: Response already commited [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.spec.ServletOutputStreamImpl.resetBuffer(ServletOutputStreamImpl.java:739) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.spec.HttpServletResponseImpl.resetBuffer(HttpServletResponseImpl.java:550) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImpl(RequestDispatcherImpl.java:169) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImplSetup(RequestDispatcherImpl.java:149) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.spec.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:111) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at com.tremolosecurity.filter.UnisonServletFilter.doFilter(UnisonServletFilter.java:299) ~[unison-server-core-1.0.20.jar:?] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) ~[undertow-core-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) ~[undertow-core-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) ~[undertow-core-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) ~[undertow-core-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) ~[undertow-core-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) ~[undertow-core-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:269) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:78) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:133) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:130) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:249) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:78) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:99) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:370) [undertow-core-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) [undertow-core-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [jboss-threads-3.1.0.Final.jar:3.1.0.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:2019) [jboss-threads-3.1.0.Final.jar:3.1.0.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1558) [jboss-threads-3.1.0.Final.jar:3.1.0.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1449) [jboss-threads-3.1.0.Final.jar:3.1.0.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275] [openunison-orchestra-74f5ffbf54-vlqcj] [2020-11-27 19:24:50,726][XNIO-1 task-10] ERROR UnisonServletFilter - Could not process request [openunison-orchestra-74f5ffbf54-vlqcj] org.apache.jasper.JasperException: JBWEB004038: An exception occurred processing JSP page /auth/forms/error.jsp at line 23 [openunison-orchestra-74f5ffbf54-vlqcj] [openunison-orchestra-74f5ffbf54-vlqcj] 23 [openunison-orchestra-74f5ffbf54-vlqcj] [openunison-orchestra-74f5ffbf54-vlqcj] 20: [openunison-orchestra-74f5ffbf54-vlqcj] 21: <% [openunison-orchestra-74f5ffbf54-vlqcj] 22: [openunison-orchestra-74f5ffbf54-vlqcj] 23: RequestHolder reqHolder = ((AuthController) session.getAttribute(ProxyConstants.AUTH_CTL)).getHolder(); [openunison-orchestra-74f5ffbf54-vlqcj] 24: String authURL = "/auth/forms/"; [openunison-orchestra-74f5ffbf54-vlqcj] 25: [openunison-orchestra-74f5ffbf54-vlqcj] 26: if (reqHolder != null) { [openunison-orchestra-74f5ffbf54-vlqcj] [openunison-orchestra-74f5ffbf54-vlqcj] [openunison-orchestra-74f5ffbf54-vlqcj] [openunison-orchestra-74f5ffbf54-vlqcj] [openunison-orchestra-74f5ffbf54-vlqcj] Stacktrace: [openunison-orchestra-74f5ffbf54-vlqcj] at org.apache.jasper.servlet.JspServletWrapper.handleJspException(JspServletWrapper.java:578) ~[jastow-2.1.0.Final.jar:2.1.0.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:476) ~[jastow-2.1.0.Final.jar:2.1.0.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:403) ~[jastow-2.1.0.Final.jar:2.1.0.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:347) ~[jastow-2.1.0.Final.jar:2.1.0.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at javax.servlet.http.HttpServlet.service(HttpServlet.java:590) ~[jakarta.servlet-api-4.0.3.jar:4.0.3] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:81) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) ~[undertow-core-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) ~[undertow-core-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:251) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchToPath(ServletInitialHandler.java:186) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImpl(RequestDispatcherImpl.java:227) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImplSetup(RequestDispatcherImpl.java:149) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.spec.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:111) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at com.tremolosecurity.proxy.ConfigSys.doConfig(ConfigSys.java:379) ~[unison-server-core-1.0.20.jar:?] [openunison-orchestra-74f5ffbf54-vlqcj] at com.tremolosecurity.embedd.NextEmbSys.nextSys(NextEmbSys.java:93) ~[unison-server-core-1.0.20.jar:?] [openunison-orchestra-74f5ffbf54-vlqcj] at com.tremolosecurity.filter.UnisonServletFilter.doFilter(UnisonServletFilter.java:290) [unison-server-core-1.0.20.jar:?] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) [undertow-core-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) [undertow-core-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) [undertow-core-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:269) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:78) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:133) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:130) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:249) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:78) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:99) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:370) [undertow-core-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) [undertow-core-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [jboss-threads-3.1.0.Final.jar:3.1.0.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:2019) [jboss-threads-3.1.0.Final.jar:3.1.0.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1558) [jboss-threads-3.1.0.Final.jar:3.1.0.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1449) [jboss-threads-3.1.0.Final.jar:3.1.0.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275] [openunison-orchestra-74f5ffbf54-vlqcj] Caused by: java.lang.NullPointerException [openunison-orchestra-74f5ffbf54-vlqcj] at org.apache.jsp.auth.forms.error_jsp._jspService(error_jsp.java:139) ~[?:?] [openunison-orchestra-74f5ffbf54-vlqcj] at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) ~[jastow-2.1.0.Final.jar:2.1.0.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at javax.servlet.http.HttpServlet.service(HttpServlet.java:590) ~[jakarta.servlet-api-4.0.3.jar:4.0.3] [openunison-orchestra-74f5ffbf54-vlqcj] at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:438) ~[jastow-2.1.0.Final.jar:2.1.0.Final] [openunison-orchestra-74f5ffbf54-vlqcj] ... 52 more [openunison-orchestra-74f5ffbf54-vlqcj] [2020-11-27 19:24:50,729][XNIO-1 task-10] ERROR request - UT005023: Exception handling request to /api/v1/namespaces/vault/pods/vault-0/portforward [openunison-orchestra-74f5ffbf54-vlqcj] org.apache.jasper.JasperException: JBWEB004038: An exception occurred processing JSP page /auth/forms/error.jsp at line 23 [openunison-orchestra-74f5ffbf54-vlqcj] [openunison-orchestra-74f5ffbf54-vlqcj] 23 [openunison-orchestra-74f5ffbf54-vlqcj] [openunison-orchestra-74f5ffbf54-vlqcj] 20: [openunison-orchestra-74f5ffbf54-vlqcj] 21: <% [openunison-orchestra-74f5ffbf54-vlqcj] 22: [openunison-orchestra-74f5ffbf54-vlqcj] 23: RequestHolder reqHolder = ((AuthController) session.getAttribute(ProxyConstants.AUTH_CTL)).getHolder(); [openunison-orchestra-74f5ffbf54-vlqcj] 24: String authURL = "/auth/forms/"; [openunison-orchestra-74f5ffbf54-vlqcj] 25: [openunison-orchestra-74f5ffbf54-vlqcj] 26: if (reqHolder != null) { [openunison-orchestra-74f5ffbf54-vlqcj] [openunison-orchestra-74f5ffbf54-vlqcj] [openunison-orchestra-74f5ffbf54-vlqcj] [openunison-orchestra-74f5ffbf54-vlqcj] [openunison-orchestra-74f5ffbf54-vlqcj] Stacktrace: [openunison-orchestra-74f5ffbf54-vlqcj] at org.apache.jasper.servlet.JspServletWrapper.handleJspException(JspServletWrapper.java:578) ~[jastow-2.1.0.Final.jar:2.1.0.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:476) ~[jastow-2.1.0.Final.jar:2.1.0.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:403) ~[jastow-2.1.0.Final.jar:2.1.0.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:347) ~[jastow-2.1.0.Final.jar:2.1.0.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at javax.servlet.http.HttpServlet.service(HttpServlet.java:590) ~[jakarta.servlet-api-4.0.3.jar:4.0.3] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:81) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) ~[undertow-core-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) ~[undertow-core-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:251) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchToPath(ServletInitialHandler.java:186) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImpl(RequestDispatcherImpl.java:227) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImplSetup(RequestDispatcherImpl.java:149) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.spec.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:111) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at com.tremolosecurity.filter.UnisonServletFilter.doFilter(UnisonServletFilter.java:299) ~[unison-server-core-1.0.20.jar:?] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) ~[undertow-core-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) ~[undertow-core-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) ~[undertow-core-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) ~[undertow-core-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) ~[undertow-core-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) ~[undertow-core-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:269) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:78) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:133) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:130) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:249) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:78) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:99) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:370) [undertow-core-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) [undertow-core-2.1.3.Final.jar:2.1.3.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [jboss-threads-3.1.0.Final.jar:3.1.0.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:2019) [jboss-threads-3.1.0.Final.jar:3.1.0.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1558) [jboss-threads-3.1.0.Final.jar:3.1.0.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1449) [jboss-threads-3.1.0.Final.jar:3.1.0.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275] [openunison-orchestra-74f5ffbf54-vlqcj] Caused by: java.lang.NullPointerException [openunison-orchestra-74f5ffbf54-vlqcj] at org.apache.jsp.auth.forms.error_jsp._jspService(error_jsp.java:139) ~[?:?] [openunison-orchestra-74f5ffbf54-vlqcj] at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) ~[jastow-2.1.0.Final.jar:2.1.0.Final] [openunison-orchestra-74f5ffbf54-vlqcj] at javax.servlet.http.HttpServlet.service(HttpServlet.java:590) ~[jakarta.servlet-api-4.0.3.jar:4.0.3] [openunison-orchestra-74f5ffbf54-vlqcj] at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:438) ~[jastow-2.1.0.Final.jar:2.1.0.Final] [openunison-orchestra-74f5ffbf54-vlqcj] ... 50 more ```

mlbiam commented 3 years ago

This is a known issue. Undertow, the web server OpenUnison runs in, does not support SPDY (nor does most any modern web server). Switching kubectl from SPDY to WebSockets or Http2 has been on the book with Kubernetes for a few years now but has not yet been implemented. We started implementing websockets support in kubectl, but have been stalled in completing it (https://github.com/kubernetes/kubernetes/issues/89163).

As a workaround with kubectl exec, the terminal in the dashboard works well, the integrated terminal uses websockets. For kubectl portforward, we don't have a workaround other then integrating those applications where you can with an authenticating reverse proxy or directly with an Oidc provider.

dkulchinsky commented 3 years ago

Thanks @mlbiam for the prompt response and details! and thank you for the effort to fix this!

Do you plan on working on this? do you think there's a reasonable chance to see this implemented any time soon?

mlbiam commented 3 years ago

hoping to get back to this in January.

dkulchinsky commented 3 years ago

Thanks again @mlbiam

Are the required changes are only on kubectl side? or server side changes required as well?

mlbiam commented 3 years ago

It's all client side. The issue is that kubectl uses SPDY, which has been deprecated by Google since 2018 I think. Very little outside of kubernetes (and really Golang) still supports the SPDY protocol. The API server has support WebSockets for years, but no one took the time to add support to kubectl. We started building in support to kubectl to make it easier to use OpenUnison but found that few users really found it to be an issue. We do still plan to finish the work to get kubectl running with WebSockets. The main issue is that websockets and spdy are very different protocols

There has been talk of using HTTP2 instead of WebSockets, but the API server doesn't support HTTP2 yet.

dkulchinsky commented 3 years ago

Thanks @mlbiam, I do hope to see this getting implemented.

librannk commented 3 years ago

Hi @mlbiam , I am facing the same issue while using EKS I would have to use impersonation, any workaround with EKS for kubectl exec apart from the dashboard? ..

mlbiam commented 3 years ago

not at this time, no. we did look at creating plugins to handle instead of trying to build into kubectl but it turned out to be nearly as much work.

firoshaq commented 3 years ago

Hi @mlbiam

We are facing similar issues while trying to exec into a pod using openunsion with API impersonation. 

error: unable to upgrade connection: <html><head><title>Error</title></head><body>Internal Server Error</body></html>

Is there any update to this issue?

dkulchinsky commented 3 years ago

@firoshaq, the issue is with kubectl that still uses SPDY for exec & port-forward, there's effort to move kubectl to WebSocket and it's tracked here https://github.com/kubernetes/kubernetes/issues/89163, so that would be the best place to track progress on this.

EDIT: and looks like there's some progress on this per this comment https://github.com/kubernetes/kubernetes/issues/89163#issuecomment-787488477 🤞🏼

mlbiam commented 3 years ago

We're looking at ways to work around this issue. Even assuming we can get exec and port forward to work via websockets in the client-go SDK, it will take a while for clients to catch up. What we're likely going to be doing in the next few weeks is adding support for a go based reverse proxy, such as oidc-proxy, that will deploy and be integrated automatically by the operator. So it will be transparent to users but will support SPDY (assuming your network infrastructure still supports it)

dkulchinsky commented 3 years ago

Thanks for this update @mlbiam 👍🏼 sounds interesting! and consider us in for any early-adopter/beta testing 😄

fcrespofastly commented 3 years ago

Awesome news!! Thanks for looking into this @mlbiam

mlbiam commented 3 years ago

Got an initial prototype working. To get it working:

  1. Download https://github.com/mlbiam/helm-charts/tree/oidc-proxy and checkout the oidc-proxy branch
  2. update your operator helm upgrade openunison /path/to/helm-charts/openunison-operator -n openunison --set image=docker.io/tremolosecurity/betas:operator-oidc-proxy
  3. Add the following to your values.yaml:
impersonation:
  use_jetstack: true
  jetstack_oidc_proxy_image: quay.io/jetstack/kube-oidc-proxy:v0.3.0
  1. Upgrade orchestra - helm upgrade orchestra /path/tol/helm-charts/openunison-k8s-login-oidc --namespace openunison -f /path/to/values.yaml

Once done running, you should have an openunison-orchestra pod and a kube-oidc-proxy-orchestra pod.

NOTE: if you enabled TokenRequest in your values.yaml by setting services.enable_tokenrequest to true,it needs to be set back to false, jetstack doesn't support yet support the TokenRequest API. Please add your voice here to make sure we can get it submitted - https://github.com/jetstack/kube-oidc-proxy/issues/190

From here we're going to be adding NetworkPolicy support, memory and cpu requests, add support for nodeselector, etc. Also looking at auditing support.

dkulchinsky commented 3 years ago

😮 wow! will try this ASAP, also adding our 👍🏼 on the linked issue.

great news @mlbiam!

dkulchinsky commented 3 years ago

@mlbiam quick question about ou-tls-certificate, we do not generate this secret since we're using a certificate from cert-manager (i.e. network.createIngressCertificate == false).

Looking at the oidc-proxy deployment manifest, it looks like it expects the cert, key & ca from this secret.

Any suggestion how to go about this in our setup?

mlbiam commented 3 years ago

@dkulchinsky for the first pass. the plan is to make it so the configuration is consistent. For now the easiest thing for testing is to just edit the kube-oidc-proxy-orchestra Deployment to remove - --oidc-ca-file=/etc/oidc/oidc-ca.pem from spec.template.spec.containers[0].args

dkulchinsky commented 3 years ago

Thanks @mlbiam, so kube-oidc-proxy should just read the crt and key from the cert-manager generated certificate secret?

mlbiam commented 3 years ago

several updates:

  1. NetworkPolicy generation now supports the oidc proxy too (no change in config)
  2. If you configured nodeSelector in the values.yaml, itis applied to the oidc-proxy (no change in config)
  3. The number of instances of openunison are also applied to the oidc-proxy (no change in config)
  4. you can add resource limits and requests to the impersonation section:
impersonation:
  use_jetstack: true
  jetstack_oidc_proxy_image: quay.io/jetstack/kube-oidc-proxy:v0.3.0
  resources:
    requests:
      memory: 100Mi
      cpu: .25
    limits:
      memory: 200Mi
      cpu: .50
  1. You can skip explicit trust for a certificate if using a commercially signed certificate by setting impersonation.explicit_certificate_trust to false. 
impersonation:
  use_jetstack: true
  jetstack_oidc_proxy_image: quay.io/jetstack/kube-oidc-proxy:v0.3.0
  explicit_certificate_trust: false

If you're using a self signed CA, then the same rules apply as with orchestra and the CA certificate will need to be installed as a tls certificate Secret in the same namespace. You can set the name of the Secret with impersonation.ca_secret_name

dkulchinsky commented 3 years ago

Hey @mlbiam, just rolled out the changes to our sandbox cluster and both exec & port-forward works flawlessly! 👏🏼 👏🏼 👏🏼

The only thing we noticed is that the connection is dropped pretty quickly if it's unused, and we see the following error:

E0323 23:39:44.810888   76039 portforward.go:233] lost connection to pod
dkulchinsky commented 3 years ago

Was looking into these connection resets again, noticed that when streaming logs (kubectl logs -f) if there's no data received (similarly for exec & port-forward) for a certain period of time kubectl bails out:

and I see the following log entry in nginx (ingress controller):

[int-ic-ingress-nginx-controller-79787886dc-sn4lb] 2021/03/24 13:22:01 [error] 13780#13780: *69425356 upstream timed out (110: Operation timed out) while reading upstream, client: 10.253.5.26, server: k8sapi.<domain>, request: "GET /api/v1/namespaces/platform/pods/int-ic-ingress-nginx-controller-79787886dc-2zpj2/log?container=controller&follow=true&sinceSeconds=10 HTTP/2.0", upstream: "https://172.26.6.130:8443/api/v1/namespaces/platform/pods/int-ic-ingress-nginx-controller-79787886dc-2zpj2/log?container=controller&follow=true&sinceSeconds=10", host: "k8sapi.<domain>"

This is a Private GKE cluster (v1.18.16-gke.300), I'll keep looking into it but if you have any ideas/suggestions - please let me know 😄

mlbiam commented 3 years ago

I'm going to try to build a version of the proxy and publish it to our docker registry. It doesn't look like there's been a release for over a year so I wonder if something like this has been fixed or handled already. I don't see anything in their issues that looks like a timeout.

dkulchinsky commented 3 years ago

I'm going to try to build a version of the proxy and publish it to our docker registry. It doesn't look like there's been a release for over a year so I wonder if something like this has been fixed or handled already. I don't see anything in their issues that looks like a timeout.

Thanks @mlbiam, the read timeout appears to be in the nginx ingress controller, I wonder if it's an nginx timeout that triggers here, so I'll try to see if this is something that can be possibly tuned on that end, will report back if I discover something useful.

mlbiam commented 3 years ago

Do you get the same issue when using openunison as the reverse proxy?

dkulchinsky commented 3 years ago

Do you get the same issue when using openunison as the reverse proxy?

was just going to report on that 😄 I ran kubectl logs -f on another cluster with orchestra as the proxy and if there are no logs streamed for ~60 seconds it bails out with:

❯ time k logs node-local-dns-2wbvm -f -n kube-system
<some output>
error: stream error: stream ID 3; INTERNAL_ERROR
kubectl logs node-local-dns-2wbvm -f  0.12s user 0.04s system 0% cpu 1:00.38 total

the above is a very "quiet" process, it doesn't print logs that often, if I "tail" a more chatty service it's fine, so looks like it only happens when there's no incoming stream for >60s

the same error appears in the nginx ingress controller:

[int-ic-ingress-nginx-controller-79787886dc-fr7r8] 2021/03/24 15:59:51 [error] 22821#22821: *69671698 upstream timed out (110: Operation timed out) while reading upstream, client: 10.253.3.45, server: k8sapi.<domain>, request: "GET /api/v1/namespaces/kube-system/pods/node-local-dns-2wbvm/log?follow=true HTTP/2.0", upstream: "https://172.22.3.34:8443/api/v1/namespaces/kube-system/pods/node-local-dns-2wbvm/log?follow=true", host: "k8sapi.<domain>"
dkulchinsky commented 3 years ago

@mlbiam, I believe I figured it out.

nginx's default for proxy-read-timeout is 60s, I added nginx.ingress.kubernetes.io/proxy-read-timeout: 900 annotation to the Ingress and now it's solid 👍🏼

dkulchinsky commented 3 years ago

@mlbiam question about the Network Policies, both the allow-from-ingress & allow-from-prometheus use namespaceSelector, however the labels in the values file seem to suggest these should use podSelector instead?

edit in fact, they should probably have both? since the from pods are all in different namespaces.

edit 2 made it work by specifying both the namespaceSelector and podSelector with the appropriate labels, I'm a bit confused about the apiserver ingress rule (which seems to be disabled by default), can you clarify why this is needed?

mlbiam commented 3 years ago

Sorry for the delay in answering:

@mlbiam question about the Network Policies, both the allow-from-ingress & allow-from-prometheus use namespaceSelector, however the labels in the values file seem to suggest these should use podSelector instead?

The design was namespace based on the idea that most ingress controllers run in their own namespace but without getting too granular on the definition. It's a balancing act between simplifying the NetworkPolicy configuration and providing all the flexibility needed. If more granularity is needed, manual policies can be created.

I'm a bit confused about the apiserver ingress rule (which seems to be disabled by default), can you clarify why this is needed?

This is to support our validating webhook used for complex configurations. Its not in place for the login portals yet. In the near future, all of OpenUnison's configuration will be available as custom resources instead of having to customize OpenUnison by editing XML and rebuilding the container. We started this process with the provisioning side of our tools and quickly discovered that JSON schema wasn't verbose or flexible enough to validate our workflow configuration so we built a validating webhook to verify more complex CustomResource configuration objects. In order to use this with NetworkPolicies enabled we needed to add a way to configure it through the helm chart since there's no standard labels on the kube-system namespace and NetworkPolicy doesn't let you explicitly name a namespace.

dkulchinsky commented 3 years ago

Thanks for the extra info @mlbiam!

We've deployed this to few more non-production clusters and so far haven't encountered any issues, so we're super happy how this works 👍🏼 👏🏼

Sorry for the delay in answering:

@mlbiam question about the Network Policies, both the allow-from-ingress & allow-from-prometheus use namespaceSelector, however the labels in the values file seem to suggest these should use podSelector instead?

The design was namespace based on the idea that most ingress controllers run in their own namespace but without getting too granular on the definition. It's a balancing act between simplifying the NetworkPolicy configuration and providing all the flexibility needed. If more granularity is needed, manual policies can be created.

Understood 👍🏼 what threw me off was the label that was used in the chart (app.kubernetes.io/name: ingress-nginx) that suggest this is a deployment/pod label, in any case, I tweaked this to use both namespace and pod selectors and works well.

I'm a bit confused about the apiserver ingress rule (which seems to be disabled by default), can you clarify why this is needed?

This is to support our validating webhook used for complex configurations. Its not in place for the login portals yet. In the near future, all of OpenUnison's configuration will be available as custom resources instead of having to customize OpenUnison by editing XML and rebuilding the container. We started this process with the provisioning side of our tools and quickly discovered that JSON schema wasn't verbose or flexible enough to validate our workflow configuration so we built a validating webhook to verify more complex CustomResource configuration objects. In order to use this with NetworkPolicies enabled we needed to add a way to configure it through the helm chart since there's no standard labels on the kube-system namespace and NetworkPolicy doesn't let you explicitly name a namespace.

Got it 👍🏼 thanks!

mlbiam commented 3 years ago

feature is now published - https://www.tremolosecurity.com/post/jetstack-oidc-proxy-integrated-into-openunison

dkulchinsky commented 3 years ago

feature is now published - tremolosecurity.com/post/jetstack-oidc-proxy-integrated-into-openunison

Thank you @mlbiam! we're very grateful for this work and the project in general. Bravo!

dkulchinsky commented 3 years ago

@mlbiam looks like the chart changes were not merged yet or I missed something?

giotab commented 3 years ago

Also confused about the helm-charts repository not updated. Any ETA on that?

mlbiam commented 3 years ago

Looks like I didn't merge into the main repo. I'll get that done now. But the helm chart repo should be updated. Should be 1.0.8. Looks like it's updated on artifacthub

luka5 commented 3 years ago

Thanks @mlbiam https://github.com/OpenUnison/helm-charts/pull/18