search

TremoloSecurity / OpenUnison

Unified Identity Management
Apache License 2.0
72 stars 9 forks source link

webhook requires hostNetwork: true on EKS with Calico CNI #788

Open o-grigorev opened 9 months ago

o-grigorev commented 9 months ago

There is an EKS cluster with the Calico CNI installed. In this environment, the deployment of orchestra-login-portal fails when attempting to deploy anyAuthenticationChain due to the webhook.

Internal error occurred: failed calling webhook "authmechs-openunison.tremolo.io": failed to call webhook: Post "https://openunison-openunison.openunison.svc:443/k8s/webhooks/v1/authmechs?timeout=5s": Address is not allowed

To work around this issue, it is required to patch the openunison deployment by addinghostNetwork: true for the Orchestra (OpenUnison) pods. This network issue is known to occur on EKS with Calico CNI, and you can find more details about it here.

Is it possible to add this parameter into kind: OpenUnison being possible to set it via helm chart.

Thanks

mlbiam commented 9 months ago

I've run into this issue a couple of times and could never figure out the issue. Didn't realize it had to do with an alternate CNI. Thanks! Added the flag network.enableHostNetwork to the values.yaml. When true, it sets the hostNetwork to true in OpenUnison's Deployment

o-grigorev commented 9 months ago

Hello @mlbiam , I figured out that Pod DNS policy should be changed as well, without that I got an issue with DNS resolution

[XNIO-1 task-2] ERROR ConfigSys - Could not process request
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison jakarta.servlet.ServletException: Could not execute request
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at com.tremolosecurity.proxy.ProxySys.doURI(ProxySys.java:112) ~[unison-server-core-1.0.37.jar:?]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at com.tremolosecurity.embedd.NextEmbSys.nextSys(NextEmbSys.java:141) ~[unison-server-core-1.0.37.jar:?]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at com.tremolosecurity.proxy.auth.AuthMgrSys.doAuthMgr(AuthMgrSys.java:138) ~[unison-server-core-1.0.37.jar:?]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at com.tremolosecurity.embedd.NextEmbSys.nextSys(NextEmbSys.java:126) ~[unison-server-core-1.0.37.jar:?]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at com.tremolosecurity.proxy.auth.AzSys.doAz(AzSys.java:139) ~[unison-sdk-1.0.37.jar:?]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at com.tremolosecurity.embedd.NextEmbSys.nextSys(NextEmbSys.java:111) ~[unison-server-core-1.0.37.jar:?]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at com.tremolosecurity.proxy.auth.AuthSys.doAuth(AuthSys.java:140) ~[unison-server-core-1.0.37.jar:?]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at com.tremolosecurity.embedd.NextEmbSys.nextSys(NextEmbSys.java:105) ~[unison-server-core-1.0.37.jar:?]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at com.tremolosecurity.proxy.ConfigSys.doConfig(ConfigSys.java:296) [unison-server-core-1.0.37.jar:?]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at com.tremolosecurity.embedd.NextEmbSys.nextSys(NextEmbSys.java:93) [unison-server-core-1.0.37.jar:?]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at com.tremolosecurity.filter.UnisonServletFilter.doFilter(UnisonServletFilter.java:299) [unison-server-core-1.0.37.jar:?]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:67) [undertow-servlet-2.3.7.Final.jar:2.3.7.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-2.3.7.Final.jar:2.3.7.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) [undertow-servlet-2.3.7.Final.jar:2.3.7.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) [undertow-servlet-2.3.7.Final.jar:2.3.7.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68) [undertow-servlet-2.3.7.Final.jar:2.3.7.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) [undertow-servlet-2.3.7.Final.jar:2.3.7.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68) [undertow-servlet-2.3.7.Final.jar:2.3.7.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:117) [undertow-servlet-2.3.7.Final.jar:2.3.7.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) [undertow-servlet-2.3.7.Final.jar:2.3.7.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-2.3.7.Final.jar:2.3.7.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) [undertow-core-2.3.7.Final.jar:2.3.7.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) [undertow-servlet-2.3.7.Final.jar:2.3.7.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) [undertow-core-2.3.7.Final.jar:2.3.7.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) [undertow-servlet-2.3.7.Final.jar:2.3.7.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) [undertow-core-2.3.7.Final.jar:2.3.7.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-2.3.7.Final.jar:2.3.7.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at io.undertow.servlet.handlers.SendErrorPageHandler.handleRequest(SendErrorPageHandler.java:52) [undertow-servlet-2.3.7.Final.jar:2.3.7.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-2.3.7.Final.jar:2.3.7.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:276) [undertow-servlet-2.3.7.Final.jar:2.3.7.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) [undertow-servlet-2.3.7.Final.jar:2.3.7.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:132) [undertow-servlet-2.3.7.Final.jar:2.3.7.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) [undertow-servlet-2.3.7.Final.jar:2.3.7.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) [undertow-servlet-2.3.7.Final.jar:2.3.7.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:256) [undertow-servlet-2.3.7.Final.jar:2.3.7.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:101) [undertow-servlet-2.3.7.Final.jar:2.3.7.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at io.undertow.server.Connectors.executeRootHandler(Connectors.java:393) [undertow-core-2.3.7.Final.jar:2.3.7.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:859) [undertow-core-2.3.7.Final.jar:2.3.7.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [jboss-threads-2.3.6.Final.jar:2.3.6.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982) [jboss-threads-2.3.6.Final.jar:2.3.6.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486) [jboss-threads-2.3.6.Final.jar:2.3.6.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377) [jboss-threads-2.3.6.Final.jar:2.3.6.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1282) [xnio-api-3.8.9.Final.jar:3.8.9.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at java.lang.Thread.run(Thread.java:829) [?:?]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison Caused by: java.net.UnknownHostException: ouhtml-openunison.openunison.svc: Name or service not known
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method) ~[?:?]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:930) ~[?:?]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1543) ~[?:?]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:848) ~[?:?]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at java.net.InetAddress.getAllByName0(InetAddress.java:1533) ~[?:?]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at java.net.InetAddress.getAllByName(InetAddress.java:1386) ~[?:?]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at java.net.InetAddress.getAllByName(InetAddress.java:1307) ~[?:?]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:45) ~[httpclient-4.5.14.jar:4.5.14]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:112) ~[httpclient-4.5.14.jar:4.5.14]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.14.jar:4.5.14]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.14.jar:4.5.14]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.14.jar:4.5.14]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.14.jar:4.5.14]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.14.jar:4.5.14]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.14.jar:4.5.14]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.14.jar:4.5.14]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.14.jar:4.5.14]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at com.tremolosecurity.proxy.postProcess.UriRequestProcess.postProcess(UriRequestProcess.java:127) ~[unison-server-core-1.0.37.jar:?]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at com.tremolosecurity.proxy.filter.HttpFilterChainImpl.nextFilter(HttpFilterChainImpl.java:92) ~[unison-server-core-1.0.37.jar:?]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at com.tremolosecurity.proxy.filters.SetNoCacheHeaders.doFilter(SetNoCacheHeaders.java:25) ~[unison-server-core-1.0.37.jar:?]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at com.tremolosecurity.proxy.filter.HttpFilterChainImpl.nextFilter(HttpFilterChainImpl.java:86) ~[unison-server-core-1.0.37.jar:?]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at com.tremolosecurity.proxy.filters.XForward.doFilter(XForward.java:61) ~[unison-server-core-1.0.37.jar:?]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at com.tremolosecurity.proxy.filter.HttpFilterChainImpl.nextFilter(HttpFilterChainImpl.java:86) ~[unison-server-core-1.0.37.jar:?]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison    at com.tremolosecurity.proxy.ProxySys.doURI(ProxySys.java:97) ~[unison-server-core-1.0.37.jar:?]

The fix is to change Orchestra pod's dnsPolicy to ClusterFirstWithHostNet

spantaleev commented 2 months ago

I'm also hitting this issue on a cluster configured with Kubespray, which uses Calico by default.

Besides the networkenableHostNetwork: true value change and patching /spec/template/spec/dnsPolicy in the openunison-openunison Deployment resource manually, those deploying via ArgoCD may wish to adjust their application like this to avoid it undoing their patch:

 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
   name: openunison
   namespace: argocd
 spec:
   project: default
   ignoreDifferences:
   - group: "admissionregistration.k8s.io"
     kind: "ValidatingWebhookConfiguration"
     jsonPointers:
     - /webhooks/0/clientConfig/caBundle
     - /webhooks/1/clientConfig/caBundle
     - /webhooks/2/clientConfig/caBundle
     - /webhooks/3/clientConfig/caBundle
     - /webhooks/4/clientConfig/caBundle
+
+  # Work around a Calico CNI issue.
+  # See: https://github.com/TremoloSecurity/OpenUnison/issues/788
+  - group: apps
+    kind: Deployment
+    jsonPointers:
+      - /spec/template/spec/dnsPolicy

It'd be great if the Helm chart provided a configuration value for dnsPolicy, so that we won't have to rest to such hacks.