search

OpenUnison / openunison-k8s-login-oidc

Kubernetes login portal for both kubectl and the dashboard using OpenID Connect. Use groups from your assertion in RBAC policies to control access to your cluster. Supports impersonation and OpenID Connect integration with your API server.
https://www.tremolosecurity.com/kubernetes/
Apache License 2.0
12 stars 5 forks source link

Streaming commands (logs and get -w) cuts off after 20~40 seconds #27

Closed dkulchinsky closed 3 years ago

dkulchinsky commented 3 years ago

Hey @mlbiam 👋🏼

We're using openunison in impersonation mode and we noticed that streaming commands such as:

fail after a short while (usually within 20~40 seconds).

when that happens, the orchestra log dump this exception:

[2020-12-08 20:43:06,928][XNIO-1 task-3] INFO  AccessLog - [Error] - apiserver - https://k8sapi.our.domain.net/api/v1/namespaces/vault/pods/secrets-manager-6b764c686-cwdvv/log - sub=<my subid>,ou=oauth2,o=Tremolo - NONE [10.233.118.114] - [null]
[2020-12-08 20:43:06,928][XNIO-1 task-3] ERROR ConfigSys - Could not process request
javax.net.ssl.SSLException: Socket closed

any ideas?

full exception log

``` [2020-12-08 20:43:06,928][XNIO-1 task-3] INFO AccessLog - [Error] - apiserver - https://k8sapi.our.domain.net/api/v1/namespaces/vault/pods/secrets-manager-6b764c686-cwdvv/log - sub=,ou=oauth2,o=Tremolo - NONE [10.233.118.114] - [null] [2020-12-08 20:43:06,928][XNIO-1 task-3] ERROR ConfigSys - Could not process request javax.net.ssl.SSLException: Socket closed at sun.security.ssl.Alert.createSSLException(Alert.java:127) ~[?:1.8.0_275] at sun.security.ssl.TransportContext.fatal(TransportContext.java:324) ~[?:1.8.0_275] at sun.security.ssl.TransportContext.fatal(TransportContext.java:267) ~[?:1.8.0_275] at sun.security.ssl.TransportContext.fatal(TransportContext.java:262) ~[?:1.8.0_275] at sun.security.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1303) ~[?:1.8.0_275] at sun.security.ssl.SSLSocketImpl.access$300(SSLSocketImpl.java:72) ~[?:1.8.0_275] at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:831) ~[?:1.8.0_275] at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) ~[httpcore-4.4.13.jar:4.4.13] at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) ~[httpcore-4.4.13.jar:4.4.13] at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280) ~[httpcore-4.4.13.jar:4.4.13] at org.apache.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:261) ~[httpcore-4.4.13.jar:4.4.13] at org.apache.http.impl.io.ChunkedInputStream.nextChunk(ChunkedInputStream.java:222) ~[httpcore-4.4.13.jar:4.4.13] at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:183) ~[httpcore-4.4.13.jar:4.4.13] at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:135) ~[httpclient-4.5.12.jar:4.5.12] at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:148) ~[httpclient-4.5.12.jar:4.5.12] at com.tremolosecurity.proxy.ConfigSys.procData(ConfigSys.java:463) ~[unison-server-core-1.0.20.jar:?] at com.tremolosecurity.proxy.ConfigSys.doConfig(ConfigSys.java:332) [unison-server-core-1.0.20.jar:?] at com.tremolosecurity.embedd.NextEmbSys.nextSys(NextEmbSys.java:93) [unison-server-core-1.0.20.jar:?] at com.tremolosecurity.filter.UnisonServletFilter.doFilter(UnisonServletFilter.java:290) [unison-server-core-1.0.20.jar:?] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-2.1.3.Final.jar:2.1.3.Final] at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) [undertow-core-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) [undertow-core-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) [undertow-core-2.1.3.Final.jar:2.1.3.Final] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-2.1.3.Final.jar:2.1.3.Final] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:269) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:78) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:133) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:130) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:249) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:78) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:99) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:370) [undertow-core-2.1.3.Final.jar:2.1.3.Final] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) [undertow-core-2.1.3.Final.jar:2.1.3.Final] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [jboss-threads-3.1.0.Final.jar:3.1.0.Final] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:2019) [jboss-threads-3.1.0.Final.jar:3.1.0.Final] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1558) [jboss-threads-3.1.0.Final.jar:3.1.0.Final] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1449) [jboss-threads-3.1.0.Final.jar:3.1.0.Final] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275] Caused by: java.net.SocketException: Socket closed at java.net.SocketInputStream.socketRead0(Native Method) ~[?:1.8.0_275] at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) ~[?:1.8.0_275] at java.net.SocketInputStream.read(SocketInputStream.java:171) ~[?:1.8.0_275] at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_275] at sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:457) ~[?:1.8.0_275] at sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:68) ~[?:1.8.0_275] at sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1095) ~[?:1.8.0_275] at sun.security.ssl.SSLSocketImpl.access$200(SSLSocketImpl.java:72) ~[?:1.8.0_275] at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:815) ~[?:1.8.0_275] ... 45 more [2020-12-08 20:43:06,929][XNIO-1 task-3] ERROR UnisonServletFilter - Could not process request java.lang.IllegalStateException: UT010019: Response already commited at io.undertow.servlet.spec.ServletOutputStreamImpl.resetBuffer(ServletOutputStreamImpl.java:739) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.spec.HttpServletResponseImpl.resetBuffer(HttpServletResponseImpl.java:550) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImpl(RequestDispatcherImpl.java:169) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImplSetup(RequestDispatcherImpl.java:149) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.spec.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:111) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at com.tremolosecurity.proxy.ConfigSys.doConfig(ConfigSys.java:379) ~[unison-server-core-1.0.20.jar:?] at com.tremolosecurity.embedd.NextEmbSys.nextSys(NextEmbSys.java:93) ~[unison-server-core-1.0.20.jar:?] at com.tremolosecurity.filter.UnisonServletFilter.doFilter(UnisonServletFilter.java:290) [unison-server-core-1.0.20.jar:?] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-2.1.3.Final.jar:2.1.3.Final] at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) [undertow-core-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) [undertow-core-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) [undertow-core-2.1.3.Final.jar:2.1.3.Final] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-2.1.3.Final.jar:2.1.3.Final] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:269) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:78) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:133) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:130) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:249) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:78) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:99) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:370) [undertow-core-2.1.3.Final.jar:2.1.3.Final] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) [undertow-core-2.1.3.Final.jar:2.1.3.Final] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [jboss-threads-3.1.0.Final.jar:3.1.0.Final] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:2019) [jboss-threads-3.1.0.Final.jar:3.1.0.Final] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1558) [jboss-threads-3.1.0.Final.jar:3.1.0.Final] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1449) [jboss-threads-3.1.0.Final.jar:3.1.0.Final] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275] [2020-12-08 20:43:06,930][XNIO-1 task-3] ERROR request - UT005023: Exception handling request to /api/v1/namespaces/vault/pods/secrets-manager-6b764c686-cwdvv/log java.lang.IllegalStateException: UT010019: Response already commited at io.undertow.servlet.spec.ServletOutputStreamImpl.resetBuffer(ServletOutputStreamImpl.java:739) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.spec.HttpServletResponseImpl.resetBuffer(HttpServletResponseImpl.java:550) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImpl(RequestDispatcherImpl.java:169) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImplSetup(RequestDispatcherImpl.java:149) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.spec.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:111) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at com.tremolosecurity.filter.UnisonServletFilter.doFilter(UnisonServletFilter.java:299) ~[unison-server-core-1.0.20.jar:?] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) ~[undertow-core-2.1.3.Final.jar:2.1.3.Final] at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) ~[undertow-core-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) ~[undertow-core-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) ~[undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) ~[undertow-core-2.1.3.Final.jar:2.1.3.Final] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) ~[undertow-core-2.1.3.Final.jar:2.1.3.Final] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) ~[undertow-core-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:269) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:78) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:133) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:130) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:249) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:78) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:99) [undertow-servlet-2.1.3.Final.jar:2.1.3.Final] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:370) [undertow-core-2.1.3.Final.jar:2.1.3.Final] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) [undertow-core-2.1.3.Final.jar:2.1.3.Final] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [jboss-threads-3.1.0.Final.jar:3.1.0.Final] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:2019) [jboss-threads-3.1.0.Final.jar:3.1.0.Final] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1558) [jboss-threads-3.1.0.Final.jar:3.1.0.Final] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1449) [jboss-threads-3.1.0.Final.jar:3.1.0.Final] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275] ```

mlbiam commented 3 years ago

i'm pretty sure this is because the token is scoped to a minute. in our next release (end of month) we'll make the token lifetime configurable so you can choose how long you want individual tokens to live for

dkulchinsky commented 3 years ago

ohh, I see! ok, that would make sense.

we did increase session_inactivity_timeout_seconds to 1 hour (3600), and the doc states "The number of seconds of inactivity before the session is terminated, also the length of the refresh token's session" so I assumed that the token's validity will also be 1 hour, but perhaps this refers to something else?

Sorry if this is already written somewhere, just trying to figure out all the cogs in the mechanism here.

mlbiam commented 3 years ago

so I assumed that the token's validity will also be 1 hour, but perhaps this refers to something else?

No. session_inactivity_timeout_seconds is the number of seconds the refresh_token is valid. The id_token its self is scoped to 1 minute

dkulchinsky commented 3 years ago

Ahh, I see 👍🏼 thanks for clarifying.

dkulchinsky commented 3 years ago

Hey @mlbiam 👋🏼 don't mean to bother, just wanted to see if there's any ETA for OU 1.0.21? looks like the rest of the bits are in place.

mlbiam commented 3 years ago

@dkulchinsky working on the release now. But if you're looking to be able to change the time-to-live of the id_token by adding K8S_TOKEN_LIFE_MILLIS to your non_secret_data section of your orchestra OpenUnison object. It defaults to 60000 (one minute) but you can now change it to anything.

dkulchinsky commented 3 years ago

Hey @mlbiam, followed you direction above and applied the change:

❯ k get openunisons.openunison.tremolo.io orchestra -ojsonpath='{.spec.non_secret_data}' | jq ' . | from_entries | .K8S_TOKEN_LIFE_MILLIS'
"900000"

I made sure to use the latest CRDs, operator and orchestra containers.

In the orchestra container I can see the above attribute in /etc/openunison/ou.env:

openunison@openunison-orchestra-844b79dfd7-tkhd2:/$ cat /etc/openunison/ou.env|grep TOKEN_LIFE
K8S_TOKEN_LIFE_MILLIS=900000

I also verified that the container I'm running includes the change to read K8S_TOKEN_LIFE_MILLIS from environment variable:

openunison@openunison-orchestra-844b79dfd7-tkhd2:/$ grep K8S_TOKEN_LIFE_MILLIS /usr/local/openunison/work/webapp/WEB-INF/applications/40-k8sIdP.xml
                          <param name="codeTokenSkewMilis" value="#[K8S_TOKEN_LIFE_MILLIS:60000]"/>
                          <param name="accessTokenTimeToLive" value="#[K8S_TOKEN_LIFE_MILLIS:60000]"/>
                          <param name="codeTokenSkewMilis" value="#[K8S_TOKEN_LIFE_MILLIS:60000]"/>
                          <param name="accessTokenTimeToLive" value="#[K8S_TOKEN_LIFE_MILLIS:60000]"/>

can also be seen in the orchestra log when starting up:

[2021-02-19 16:39:31,002][main] INFO  OpenUnisonOnUndertow - Loading environment file : '/etc/openunison/ou.env'
.
.
.
[2021-02-19 16:39:31,003][main] INFO  OpenUnisonOnUndertow - Adding property : 'K8S_TOKEN_LIFE_MILLIS'
.
.
.

everything otherwise seem to work just fine, but running logs -f still cuts off after 40~60 seconds.

any suggestion how to debug this? or possibly what I might be doing wrong?

error seem to be the same as before:

[openunison-orchestra-844b79dfd7-tkhd2] [2021-02-19 17:22:42,250][XNIO-1 task-11] INFO  AccessLog - [Error] - apiserver - https://k8sapi.<our.domain>/api/v1/namespaces/openunison/pods/openunison-orchestra-844b79dfd7-tkhd2/log - sub=<my sub id>,ou=oauth2,o=Tremolo - NONE [172.26.5.84] - [null]
[openunison-orchestra-844b79dfd7-tkhd2] [2021-02-19 17:22:42,250][XNIO-1 task-11] ERROR ConfigSys - Could not process request
[openunison-orchestra-844b79dfd7-tkhd2] javax.net.ssl.SSLException: Socket closed
dkulchinsky commented 3 years ago

@mlbiam not sure if related, but I see that in 90-k8s-login-cli.xml the values are still hardcoded

https://github.com/OpenUnison/openunison-k8s-login-oidc/blob/6b8c255f2ed0f3fd331d45714a38aa36839998aa/src/main/webapp/WEB-INF/applications/90-k8s-login-cli.xml#L37-L38

EDIT: pretty sure this ☝🏼 is unrelated

mlbiam commented 3 years ago

pretty sure this ☝🏼 is unrelated

correct, that trust is just for the oulogin plugin.

What k8s distro are you using? EKS or something else?

dkulchinsky commented 3 years ago

pretty sure this ☝🏼 is unrelated

correct, that trust is just for the oulogin plugin.

What k8s distro are you using? EKS or something else?

We're using GKE mostly, and also clusters we deploy on-prem using Kubespray, all running v1.18.14

I didn't check the same on our non-GKE cluster, will check this now.

dkulchinsky commented 3 years ago

confirmed, same behaviour on a Kubespray deployed cluster.

mlbiam commented 3 years ago

confirmed, same behaviour on a Kubespray deployed cluster.

do your kubespray clusters also use impersonation?

dkulchinsky commented 3 years ago

confirmed, same behaviour on a Kubespray deployed cluster.

do your kubespray clusters also use impersonation?

Yes, we use impersonation on these clusters too.

mlbiam commented 3 years ago

last question, is the API server talking directly to an API server or to a load balancer?

dkulchinsky commented 3 years ago

last question, is the API server talking directly to an API server or to a load balancer?

mmm, not sure I understood that part. wdym by the API server talking directly to an API server?

dkulchinsky commented 3 years ago

if you're referring to the network path between Pods and API Server, it's a bit different for each K8s architecture:

but I may have misunderstood your question

mlbiam commented 3 years ago

give docker.io/tremolosecurity/betas:oidc-1.0.21 a try as your image. think i got this fixed

dkulchinsky commented 3 years ago

give docker.io/tremolosecurity/betas:oidc-1.0.21 a try as your image. think i got this fixed

Thanks Mark! was out today, will give a try tomorrow and report.

dkulchinsky commented 3 years ago

@mlbiam I just replaced the image on our test cluster (GKE) to the above and looks like it's working 😄 👏🏼 (I had to use the kubectl command from the portal though, since oulogin is not working, details below).

Some issues:

  1. When trying login from CLI using oulogin, I get this (pointed KUBECONFIG to a new file just in case):

      ❯ k oulogin --host=k8sou.<cluster>
  2021/02/25 15:07:14 http: panic serving 127.0.0.1:51863: runtime error: slice bounds out of range [:-1]
  goroutine 20 [running]:
  net/http.(*conn).serve.func1(0xc0001a8fa0)
          /usr/lib/go-1.14/src/net/http/server.go:1772 +0x139
  panic(0x16a6920, 0xc0000366c0)
          /usr/lib/go-1.14/src/runtime/panic.go:975 +0x3e3
  main.byte2string(0x0, 0x0, 0x0, 0xb, 0x1c71f00)
          /home/mlbiam/git-local/kubectl-login/kubectl-login.go:271 +0x16f
  main.(*oidcService).oidcHandleRedirect(0xc0000f81c0, 0x17dd1c0, 0xc0000f8620, 0xc000294100)
          /home/mlbiam/git-local/kubectl-login/kubectl-login.go:179 +0x5d4
  net/http.HandlerFunc.ServeHTTP(0xc0001c4510, 0x17dd1c0, 0xc0000f8620, 0xc000294100)
          /usr/lib/go-1.14/src/net/http/server.go:2012 +0x44
  net/http.(*ServeMux).ServeHTTP(0xc0000a6f80, 0x17dd1c0, 0xc0000f8620, 0xc000294100)
          /usr/lib/go-1.14/src/net/http/server.go:2387 +0x1a5
  net/http.serverHandler.ServeHTTP(0xc0000f82a0, 0x17dd1c0, 0xc0000f8620, 0xc000294100)
          /usr/lib/go-1.14/src/net/http/server.go:2807 +0xa3
  net/http.(*conn).serve(0xc0001a8fa0, 0x17dea40, 0xc0000a7000)
          /usr/lib/go-1.14/src/net/http/server.go:1895 +0x86c
  created by net/http.(*Server).Serve
          /usr/lib/go-1.14/src/net/http/server.go:2933 +0x35c

  2. The portal was throwing some weird errors at first but after a few retries I managed to connect, could see the dashboard and get the kubectl command.

  3. I noticed some errors and warnings in the orchestra logs:

[openunison-orchestra-6bd444644c-b5cj4] context [anonymous] 1:40 attribute k8s_newline_cert isn't defined
[openunison-orchestra-6bd444644c-b5cj4] context [anonymous] 1:769 attribute ou_b64_cert isn't defined
[openunison-orchestra-6bd444644c-b5cj4] context [anonymous] 1:726 attribute ou_b64_cert isn't defined
[openunison-orchestra-6bd444644c-m6d2m] [2021-02-25 20:27:08,494][XNIO-1 task-7] WARN  OAuth2JWT - No audience configuration, all requests will fail
[openunison-orchestra-6bd444644c-m6d2m] [2021-02-25 20:27:08,494][XNIO-1 task-7] WARN  OAuth2JWT - Invalid audience
mlbiam commented 3 years ago

the plugin issue should be fixed now. in 1.0.21 we are being more strict about verifying the audience of JWTs. updated the config to properly checka gainst the audience. Delete the openuison-orchestra pod and once it's back you should be able to login with the plugin again.

dkulchinsky commented 3 years ago

Thanks @mlbiam! just upgraded oulogin and restarted openuison-orchestra pods, everything works and don't see any issues/errors 👍🏼 👍🏼 👍🏼

I did have to remove cookies in the browser for the k8sou... site after rolling our the new Pods, was giving me a 404, works fine after clearing the cookies.

mlbiam commented 3 years ago

great. 1.0.21 should be rolled out early next week. will close then

dkulchinsky commented 3 years ago

great. 1.0.21 should be rolled out early next week. will close then

wow! great news 🥳

thanks for your efforts @mlbiam and have a great weekend!