SEC-2170: DefaultSpringSecurityContextSource Not Disabling Connection Pooling Properly

[spring-projects-issues]

James Carman (Migrated from SEC-2170) said:

We configured our server using the namespace configuration as follows:

<sec:ldap-server 
  url="${ldap.url}"
  manager-dn="${ldap.managerDn}"
  manager-password="${ldap.managerPassword}"/>

When we ran a load test against the server, we noticed that we had a memory leak. After investigating the cause, it showed that we had over 7,000 threads running with their "target" being a com.sun.jndi.ldap.Connection object. After much investigation, we decided to try turning off pooling to see if this would help:

<bean id="ldapContextSource" class="org.springframework.security.ldap.DefaultSpringSecurityContextSource">
  <constructor-arg value="${ldap.url}" />
  <property name="pooled" value="false" />
  <property name="userDn" value="${ldap.managerDn}" />
  <property name="password" value="${ldap.managerPassword}" />
</bean>

The memory leak went away. Now, the documentation for the DefaultSpringSecurityContextSource says that it will turn off pooling "disable pooling when the DN doesn't match the userDn property." It appears as if it is attempting to disable pooling, but for some reason our connections piled up anyway.

[spring-projects-issues]

Rob Winch said:

Thank you for your report. We have a test that explicitly test the pool property is set properly so I am confident that it gets set. This doesn't mean there isn't an issue though, so I would like to ensure we address this.

The root of the problem seems to be that you have a memory leak. However, with the provided information I am unable to reproduce the issue locally. I have created a sample test, that does not reproduce the issue. Can you please provide additional details?

• If possible the easiest way to ensure we can fix this issue is for you to provide a complete project that reproduces the issue.
• How are you using the DefaultSpringSecurityContextSource? For example, do you use it directly or with an AuthenticationProvider? If you use it directly, are you ensuring to close the connection? If you do not close the connection there will be a memory leak.
• If you are using an AuthenticationProvider, what type of authentication are you performing (i.e. bind or password)?
• Can you provide the additional ldap related configuration?
• What are your ldap connection pool settings?
• Can you provide the logs when enabling debugging (i.e. setting the system property com.sun.jndi.ldap.connect.pool.debug=all)?
• Can you provide a copy of the data that demonstrates the memory leak? Also if it is in a proprietary format please let me know which profiler you are using (i.e. YourKit)

[spring-projects-issues]

James Carman said:

I don't debate that the property is being set. I don't think you're overriding that property properly with that SimpleDirContextAuthenticationStrategy you guys plug in. It doesn't appear to be listening to you. :)

1. We are using it via an AuthenticationProvider and not directly either. We have a few elements defined in our project.
2. BindAuthenticator is used (with user-dn-pattern, group-search-base, and group-search-filter defined).
3. What sort of other information do you want.
4. I do not define any connection pool settings.
5. This is a project for a client, so giving you a log might not be possible.
6. I did a heap dump using jmap and then opened it in Eclipse Memory Analysis Tool to find it. Also, I ran it locally and saw a bunch of persistent threads being created with the Connection as the target. All I did is run it in debug mode and pause the JVM to see the threads.

[spring-projects-issues]

Luke Taylor said:

A few things...

Are you doing anything else involving LDAP in your application? LDAP connection pooling is JVM-wide.

The property you refer to only applies for calls to BindAuthenticator. All other calls will use pooling. You should work out under what identity the bind operation took place for each connection.

If you are using LDAP, then you should configure the JVM's connection pooling, otherwise you can run into problems with timeouts etc.

See http://docs.oracle.com/javase/jndi/tutorial/ldap/connect/config.html

You can dump the current pool statistics using

com.sun.jndi.ldap.LdapPoolManager.showStats(System.out);

which should be useful in monitoring how different operations affect the pool size.

This code has been using this approach for a long time (a decade, give or take :) ), see:

https://github.com/SpringSource/spring-security/blob/1.0.0/core/src/main/java/org/acegisecurity/ldap/DefaultInitialDirContextFactory.java#L218

It now relies on Spring LDAP for most of the equivalent code, but again it's been that way for a long time, and Spring LDAP has been used a lot elsewhere.

So it seems against the odds that something like this would go unnoticed for so long unless it's a corner-case issue. Given that, I think you'll need to reproduce the problem in a self-contained sample (possibly with a slapd configuration to mimic the directory setup).

[spring-projects-issues]

James Carman said:

I was following instructions here:

http://static.springsource.org/spring-security/site/docs/3.2.x/reference/ldap.html

Which doesn't mention anything about pooling at all.

[spring-projects-issues]

James Carman said:

Here's an example application that exhibits the issue. It uses the same LDIF file you guys use in your unit tests. I ran this application using mvn jetty:run. Then, I connected JVisualVM to the running VM, noticed there were no LDAP threads running (their names are "Thread-*"), then accessed the index.jsp page using the following Apache Bench command:

ab -A bob:bobspassword -n 1000 -c 20 http://localhost:8080/index.jsp

After that, JVisualVM is showing a bunch of "live" LDAP threads running.

[spring-projects-issues]

James Carman said:

See my comment and attached example project.

[spring-projects-issues]

Rob Winch said:

Thank you for the sample project.

After authenticating, the sample application is performing a search with DefaultLdapAuthoritiesPopulator which uses the managerDn to authenticate and search for user's authorities. This means that pooling is used for looking up the authorities. You can find an example call stack of the thread being created below [1]. A thread is being created for each Connection and is still runnable (live) so long as the Connection (the Runnable for the Thread) has not reached the end of the stream. In short, this is not a bug in Spring Security. You need to configure LDAP Pooling to limit the number of threads that are created.

In respect to the documentation not mentioning anything about Pooling, I think this is a fair point. The documentation does make references to other documentation, but it is a bit vague. There is explicit mention of this on the JavaDoc of AbstractContextSource, but this is a buried (esp when using the namespace configuration). I have created SEC-2171 to enhance the documentation to include information about pooling. Please feel free to submit a pull request.

[1]

com.sun.jndi.ldap.VersionHelper12$3.run()
java.security.AccessController.doPrivileged(PrivilegedAction)
com.sun.jndi.ldap.VersionHelper12.createThread(Runnable)
com.sun.jndi.ldap.Connection.<init>(LdapClient, String, int, String, int, int, OutputStream)
com.sun.jndi.ldap.LdapClient.<init>(String, int, String, int, int, OutputStream, PoolCallback)
com.sun.jndi.ldap.LdapClientFactory.createPooledConnection(PoolCallback)
com.sun.jndi.ldap.pool.Connections.getOrCreateConnection(PooledConnectionFactory)
com.sun.jndi.ldap.pool.Connections.get(long, PooledConnectionFactory)
com.sun.jndi.ldap.pool.Pool.getPooledConnection(Object, long, PooledConnectionFactory)
com.sun.jndi.ldap.LdapPoolManager.getLdapClient(String, int, String, int, int, OutputStream, int, String, Control[], String, String, Object, Hashtable)
com.sun.jndi.ldap.LdapClient.getInstance(boolean, String, int, String, int, int, OutputStream, int, String, Control[], String, String, Object, Hashtable)
com.sun.jndi.ldap.LdapCtx.connect(boolean)
com.sun.jndi.ldap.LdapCtx.<init>(String, String, int, Hashtable, boolean)
com.sun.jndi.ldap.LdapCtxFactory.getUsingURL(String, Hashtable)
com.sun.jndi.ldap.LdapCtxFactory.getUsingURLs(String[], Hashtable)
com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxInstance(Object, Hashtable)
com.sun.jndi.ldap.LdapCtxFactory.getInitialContext(Hashtable)
javax.naming.spi.NamingManager.getInitialContext(Hashtable)
javax.naming.InitialContext.getDefaultInitCtx()
javax.naming.InitialContext.init(Hashtable)
javax.naming.ldap.InitialLdapContext.<init>(Hashtable, Control[])
org.springframework.ldap.core.support.LdapContextSource.getDirContextInstance(Hashtable)
org.springframework.ldap.core.support.AbstractContextSource.createContext(Hashtable)
org.springframework.ldap.core.support.AbstractContextSource.getContext(String, String)
org.springframework.ldap.core.support.AbstractContextSource.getReadOnlyContext()
org.springframework.ldap.core.LdapTemplate.search(SearchExecutor, NameClassPairCallbackHandler, DirContextProcessor)
org.springframework.ldap.core.LdapTemplate.search(String, String, SearchControls, NameClassPairCallbackHandler, DirContextProcessor)
org.springframework.ldap.core.LdapTemplate.search(String, String, SearchControls, ContextMapper, DirContextProcessor)
org.springframework.ldap.core.LdapTemplate.search(String, String, SearchControls, ContextMapper)
org.springframework.security.ldap.SpringSecurityLdapTemplate.searchForSingleAttributeValues(String, String, Object[], String)
org.springframework.security.ldap.userdetails.DefaultLdapAuthoritiesPopulator.getGroupMembershipRoles(String, String)
org.springframework.security.ldap.userdetails.DefaultLdapAuthoritiesPopulator.getGrantedAuthorities(DirContextOperations, String)
org.springframework.security.ldap.authentication.LdapAuthenticationProvider.loadUserAuthorities(DirContextOperations, String, String)
org.springframework.security.ldap.authentication.AbstractLdapAuthenticationProvider.authenticate(Authentication)
org.springframework.security.authentication.ProviderManager.authenticate(Authentication)<2 recursive calls>
org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(ServletRequest, ServletResponse, FilterChain)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(ServletRequest, ServletResponse)
org.springframework.security.web.authentication.ui.DefaultLoginPageGeneratingFilter.doFilter(ServletRequest, ServletResponse, FilterChain)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(ServletRequest, ServletResponse)
org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(ServletRequest, ServletResponse, FilterChain)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(ServletRequest, ServletResponse)
org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(ServletRequest, ServletResponse, FilterChain)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(ServletRequest, ServletResponse)
org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(ServletRequest, ServletResponse, FilterChain)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(ServletRequest, ServletResponse)
org.springframework.security.web.FilterChainProxy.doFilterInternal(ServletRequest, ServletResponse, FilterChain)
org.springframework.security.web.FilterChainProxy.doFilter(ServletRequest, ServletResponse, FilterChain)
org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(Filter, ServletRequest, ServletResponse, FilterChain)
org.springframework.web.filter.DelegatingFilterProxy.doFilter(ServletRequest, ServletResponse, FilterChain)
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletRequest, ServletResponse)
org.eclipse.jetty.servlet.ServletHandler.doHandle(String, Request, HttpServletRequest, HttpServletResponse)
org.eclipse.jetty.server.handler.ScopedHandler.handle(String, Request, HttpServletRequest, HttpServletResponse)
org.eclipse.jetty.security.SecurityHandler.handle(String, Request, HttpServletRequest, HttpServletResponse)
org.eclipse.jetty.server.session.SessionHandler.doHandle(String, Request, HttpServletRequest, HttpServletResponse)
org.eclipse.jetty.server.handler.ContextHandler.doHandle(String, Request, HttpServletRequest, HttpServletResponse)
org.eclipse.jetty.servlet.ServletHandler.doScope(String, Request, HttpServletRequest, HttpServletResponse)
org.eclipse.jetty.server.session.SessionHandler.doScope(String, Request, HttpServletRequest, HttpServletResponse)
org.eclipse.jetty.server.handler.ContextHandler.doScope(String, Request, HttpServletRequest, HttpServletResponse)
org.eclipse.jetty.server.handler.ScopedHandler.handle(String, Request, HttpServletRequest, HttpServletResponse)
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(String, Request, HttpServletRequest, HttpServletResponse)
org.eclipse.jetty.server.handler.HandlerCollection.handle(String, Request, HttpServletRequest, HttpServletResponse)
org.eclipse.jetty.server.handler.HandlerWrapper.handle(String, Request, HttpServletRequest, HttpServletResponse)
org.eclipse.jetty.server.Server.handle(AbstractHttpConnection)
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest()
org.eclipse.jetty.server.AbstractHttpConnection.headerComplete()
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete()
org.eclipse.jetty.http.HttpParser.parseNext()
org.eclipse.jetty.http.HttpParser.parseAvailable()
org.eclipse.jetty.server.AsyncHttpConnection.handle()
org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle()
org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run()
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(Runnable)
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run()
java.lang.Thread.run()

[spring-projects-issues]

James Carman said:

Sounds good. Once I figure out the pooling settings, I'll try to submit a pull request to help with docs. Thanks for investigating. I don't know how we got to 7000. Tomcat only accepts 250 connections in our config.