search

opensearch-project / security

🔐 Secure your cluster with TLS, numerous authentication backends, data masking, audit logging as well as role-based access control on indices, documents, and fields
https://opensearch.org/docs/latest/security-plugin/index/
Apache License 2.0
197 stars 275 forks source link

LDAP Connection timed out when not using global catalog port in LDAP config #1682

Open sevenval-admins opened 2 years ago

sevenval-admins commented 2 years ago

Describe the bug Hi everybody, I am not sure if this is a bug but I would like to share with you my experience in order to know if anyone else has had the same problem, how and if they have solved it. I installed opensearch and opensearch-dashboard via helm-chart, rewrote the opensearch configuration to connect my LDAP server to the opensearch instance for authc and authz. Initially I used port 389, I could easily log in as admin, but as soon as I used my LDAP credentials the dashboard (after at least 30 seconds of loading) reported that I had used an incorrect username and password; the logs were full of connection time out errors:

[2022-03-01T12:58:25,697][WARN ][o.l.r.SearchReferralHandler] [opensearch-test-cluster-master-0] Could not follow referral to ldap://ForestDnsZones.example.com/DC=ForestDnsZones,DC=example,DC=com
org.ldaptive.provider.ConnectionException: javax.naming.CommunicationException: ForestDnsZones.example.com:389 [Root exception is java.net.SocketTimeoutException: Connect timed out]
    at org.ldaptive.provider.jndi.JndiConnectionFactory.createInternal(JndiConnectionFactory.java:90) ~[ldaptive-1.2.3.jar:?]
    at org.ldaptive.provider.jndi.JndiConnectionFactory.createInternal(JndiConnectionFactory.java:21) ~[ldaptive-1.2.3.jar:?]
    at org.ldaptive.provider.AbstractProviderConnectionFactory.create(AbstractProviderConnectionFactory.java:84) ~[ldaptive-1.2.3.jar:?]
    at org.ldaptive.DefaultConnectionFactory$DefaultConnection.open(DefaultConnectionFactory.java:267) ~[ldaptive-1.2.3.jar:?]
    at org.ldaptive.referral.AbstractReferralHandler.followReferral(AbstractReferralHandler.java:156) [ldaptive-1.2.3.jar:?]
    at org.ldaptive.referral.AbstractReferralHandler.handle(AbstractReferralHandler.java:221) [ldaptive-1.2.3.jar:?]
    at org.ldaptive.referral.SearchReferralHandler$SearchReferenceHandler.handle(SearchReferralHandler.java:268) [ldaptive-1.2.3.jar:?]
    at org.ldaptive.referral.SearchReferralHandler$SearchReferenceHandler.handle(SearchReferralHandler.java:155) [ldaptive-1.2.3.jar:?]
    at org.ldaptive.AbstractOperation.executeHandlers(AbstractOperation.java:186) [ldaptive-1.2.3.jar:?]
    at org.ldaptive.SearchOperation.readResult(SearchOperation.java:152) [ldaptive-1.2.3.jar:?]
    at org.ldaptive.SearchOperation.executeSearch(SearchOperation.java:104) [ldaptive-1.2.3.jar:?]
    at org.ldaptive.SearchOperation.invoke(SearchOperation.java:85) [ldaptive-1.2.3.jar:?]
    at org.ldaptive.SearchOperation.invoke(SearchOperation.java:15) [ldaptive-1.2.3.jar:?]
    at org.ldaptive.AbstractOperation.execute(AbstractOperation.java:126) [ldaptive-1.2.3.jar:?]
    at com.amazon.dlic.auth.ldap.util.LdapHelper$1.run(LdapHelper.java:67) [opensearch-security-1.2.4.0.jar:1.2.4.0]
    at com.amazon.dlic.auth.ldap.util.LdapHelper$1.run(LdapHelper.java:56) [opensearch-security-1.2.4.0.jar:1.2.4.0]
    at java.security.AccessController.doPrivileged(AccessController.java:554) [?:?]
    at com.amazon.dlic.auth.ldap.util.LdapHelper.search(LdapHelper.java:56) [opensearch-security-1.2.4.0.jar:1.2.4.0]
    at com.amazon.dlic.auth.ldap.backend.LDAPAuthorizationBackend.resolveNestedRoles(LDAPAuthorizationBackend.java:1021) [opensearch-security-1.2.4.0.jar:1.2.4.0]
    at com.amazon.dlic.auth.ldap.backend.LDAPAuthorizationBackend.fillRoles(LDAPAuthorizationBackend.java:894) [opensearch-security-1.2.4.0.jar:1.2.4.0]
    at org.opensearch.security.auth.BackendRegistry.authz(BackendRegistry.java:635) [opensearch-security-1.2.4.0.jar:1.2.4.0]
    at org.opensearch.security.auth.BackendRegistry.access$000(BackendRegistry.java:82) [opensearch-security-1.2.4.0.jar:1.2.4.0]
    at org.opensearch.security.auth.BackendRegistry$9.call(BackendRegistry.java:675) [opensearch-security-1.2.4.0.jar:1.2.4.0]
    at org.opensearch.security.auth.BackendRegistry$9.call(BackendRegistry.java:668) [opensearch-security-1.2.4.0.jar:1.2.4.0]
    at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4875) [guava-25.1-jre.jar:?]
    at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) [guava-25.1-jre.jar:?]
    at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2276) [guava-25.1-jre.jar:?]
    at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2154) [guava-25.1-jre.jar:?]
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2044) [guava-25.1-jre.jar:?]
    at com.google.common.cache.LocalCache.get(LocalCache.java:3951) [guava-25.1-jre.jar:?]
    at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4870) [guava-25.1-jre.jar:?]
    at org.opensearch.security.auth.BackendRegistry.authcz(BackendRegistry.java:668) [opensearch-security-1.2.4.0.jar:1.2.4.0]
    at org.opensearch.security.auth.BackendRegistry.authenticate(BackendRegistry.java:471) [opensearch-security-1.2.4.0.jar:1.2.4.0]
    at org.opensearch.security.filter.SecurityRestFilter.checkAndAuthenticateRequest(SecurityRestFilter.java:188) [opensearch-security-1.2.4.0.jar:1.2.4.0]
    at org.opensearch.security.filter.SecurityRestFilter.access$000(SecurityRestFilter.java:72) [opensearch-security-1.2.4.0.jar:1.2.4.0]
    at org.opensearch.security.filter.SecurityRestFilter$1.handleRequest(SecurityRestFilter.java:123) [opensearch-security-1.2.4.0.jar:1.2.4.0]
    at org.opensearch.rest.RestController.dispatchRequest(RestController.java:306) [opensearch-1.2.4.jar:1.2.4]
    at org.opensearch.rest.RestController.tryAllHandlers(RestController.java:392) [opensearch-1.2.4.jar:1.2.4]
    at org.opensearch.rest.RestController.dispatchRequest(RestController.java:235) [opensearch-1.2.4.jar:1.2.4]
    at org.opensearch.security.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:63) [opensearch-security-1.2.4.0.jar:1.2.4.0]
    at org.opensearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:361) [opensearch-1.2.4.jar:1.2.4]
    at org.opensearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:440) [opensearch-1.2.4.jar:1.2.4]
    at org.opensearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:351) [opensearch-1.2.4.jar:1.2.4]
    at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:55) [transport-netty4-client-1.2.4.jar:1.2.4]
    at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:41) [transport-netty4-client-1.2.4.jar:1.2.4]
    at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
    at org.opensearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:71) [transport-netty4-client-1.2.4.jar:1.2.4]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.72.Final.jar:4.1.72.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.72.Final.jar:4.1.72.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.72.Final.jar:4.1.72.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) [netty-codec-4.1.72.Final.jar:4.1.72.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) [netty-codec-4.1.72.Final.jar:4.1.72.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
    at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.72.Final.jar:4.1.72.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.72.Final.jar:4.1.72.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
    at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1371) [netty-handler-4.1.72.Final.jar:4.1.72.Final]
    at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1234) [netty-handler-4.1.72.Final.jar:4.1.72.Final]
    at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1283) [netty-handler-4.1.72.Final.jar:4.1.72.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507) [netty-codec-4.1.72.Final.jar:4.1.72.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446) [netty-codec-4.1.72.Final.jar:4.1.72.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) [netty-codec-4.1.72.Final.jar:4.1.72.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.72.Final.jar:4.1.72.Final]
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.72.Final.jar:4.1.72.Final]
    at java.lang.Thread.run(Thread.java:832) [?:?]
Caused by: javax.naming.CommunicationException: ForestDnsZones.eample.com:389
    at com.sun.jndi.ldap.Connection.<init>(Connection.java:244) ~[?:?]
    at com.sun.jndi.ldap.LdapClient.<init>(LdapClient.java:137) ~[?:?]
    at com.sun.jndi.ldap.LdapClient.getInstance(LdapClient.java:1616) ~[?:?]
    at com.sun.jndi.ldap.LdapCtx.connect(LdapCtx.java:2847) ~[?:?]
    at com.sun.jndi.ldap.LdapCtx.<init>(LdapCtx.java:348) ~[?:?]
    at com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxFromUrl(LdapCtxFactory.java:225) ~[?:?]
    at com.sun.jndi.ldap.LdapCtxFactory.getUsingURL(LdapCtxFactory.java:189) ~[?:?]
    at com.sun.jndi.ldap.LdapCtxFactory.getUsingURLs(LdapCtxFactory.java:243) ~[?:?]
    at com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxInstance(LdapCtxFactory.java:154) ~[?:?]
    at com.sun.jndi.ldap.LdapCtxFactory.getInitialContext(LdapCtxFactory.java:84) ~[?:?]
    at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:719) ~[?:?]
    at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:305) ~[?:?]
    at javax.naming.InitialContext.init(InitialContext.java:236) ~[?:?]
    at javax.naming.ldap.InitialLdapContext.<init>(InitialLdapContext.java:154) ~[?:?]
    at org.ldaptive.provider.jndi.JndiConnectionFactory.createInternal(JndiConnectionFactory.java:87) ~[ldaptive-1.2.3.jar:?]
    ... 98 more
Caused by: java.net.SocketTimeoutException: Connect timed out
    at sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546) ~[?:?]
    at sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597) ~[?:?]
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:333) ~[?:?]
    at java.net.Socket.connect(Socket.java:648) ~[?:?]
    at com.sun.jndi.ldap.Connection.createSocket(Connection.java:327) ~[?:?]
    at com.sun.jndi.ldap.Connection.<init>(Connection.java:223) ~[?:?]
    at com.sun.jndi.ldap.LdapClient.<init>(LdapClient.java:137) ~[?:?]
    at com.sun.jndi.ldap.LdapClient.getInstance(LdapClient.java:1616) ~[?:?]
    at com.sun.jndi.ldap.LdapCtx.connect(LdapCtx.java:2847) ~[?:?]
    at com.sun.jndi.ldap.LdapCtx.<init>(LdapCtx.java:348) ~[?:?]
    at com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxFromUrl(LdapCtxFactory.java:225) ~[?:?]
    at com.sun.jndi.ldap.LdapCtxFactory.getUsingURL(LdapCtxFactory.java:189) ~[?:?]
    at com.sun.jndi.ldap.LdapCtxFactory.getUsingURLs(LdapCtxFactory.java:243) ~[?:?]
    at com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxInstance(LdapCtxFactory.java:154) ~[?:?]
    at com.sun.jndi.ldap.LdapCtxFactory.getInitialContext(LdapCtxFactory.java:84) ~[?:?]
    at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:719) ~[?:?]
    at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:305) ~[?:?]
    at javax.naming.InitialContext.init(InitialContext.java:236) ~[?:?]
    at javax.naming.ldap.InitialLdapContext.<init>(InitialLdapContext.java:154) ~[?:?]
    at org.ldaptive.provider.jndi.JndiConnectionFactory.createInternal(JndiConnectionFactory.java:87) ~[ldaptive-1.2.3.jar:?]

Finally I solved the problem by changing the LDAP-Server port to the global-catalog port, 3268 and 3269 (fort TLS). I already reported the problem and the findig to guys of the helm-chart repo, but I just thought that was interessant to share my experience with you too. Please feel free to ask any context, details and question.

mmiller1 commented 2 years ago

I've experienced the same issue and wound up at the same solution as you. In our environment, our domain controllers are globally distributed, the domain controller configured can forward requests to any of these DCs for a number of reasons, we were seeing our requests occasionally being forwarded to DCs very geographically distant from ES, which resulted in timeouts. The catalog ports do not participate in this forwarding activity. I would consider this as-designed.

I should also mention it's up to the client application (ES in this case) to honor the forwarding request, many do not, however whatever library ES is using does, this doesn't necessarily seem wrong to me.

peternied commented 2 years ago

[Triage] We need to do a little more digging into this issue to see if this behavior should be changed or not.