Closed spring-projects-issues closed 10 years ago
Rob Winch said:
Thank you for your report. We have a test that explicitly test the pool property is set properly so I am confident that it gets set. This doesn't mean there isn't an issue though, so I would like to ensure we address this.
The root of the problem seems to be that you have a memory leak. However, with the provided information I am unable to reproduce the issue locally. I have created a sample test, that does not reproduce the issue. Can you please provide additional details?
James Carman said:
I don't debate that the property is being set. I don't think you're overriding that property properly with that SimpleDirContextAuthenticationStrategy you guys plug in. It doesn't appear to be listening to you. :)
Luke Taylor said:
A few things...
Are you doing anything else involving LDAP in your application? LDAP connection pooling is JVM-wide.
The property you refer to only applies for calls to BindAuthenticator. All other calls will use pooling. You should work out under what identity the bind operation took place for each connection.
If you are using LDAP, then you should configure the JVM's connection pooling, otherwise you can run into problems with timeouts etc.
See http://docs.oracle.com/javase/jndi/tutorial/ldap/connect/config.html
You can dump the current pool statistics using
com.sun.jndi.ldap.LdapPoolManager.showStats(System.out);
which should be useful in monitoring how different operations affect the pool size.
This code has been using this approach for a long time (a decade, give or take :) ), see:
It now relies on Spring LDAP for most of the equivalent code, but again it's been that way for a long time, and Spring LDAP has been used a lot elsewhere.
So it seems against the odds that something like this would go unnoticed for so long unless it's a corner-case issue. Given that, I think you'll need to reproduce the problem in a self-contained sample (possibly with a slapd configuration to mimic the directory setup).
James Carman said:
I was following instructions here:
http://static.springsource.org/spring-security/site/docs/3.2.x/reference/ldap.html
Which doesn't mention anything about pooling at all.
James Carman said:
Here's an example application that exhibits the issue. It uses the same LDIF file you guys use in your unit tests. I ran this application using mvn jetty:run. Then, I connected JVisualVM to the running VM, noticed there were no LDAP threads running (their names are "Thread-*"), then accessed the index.jsp page using the following Apache Bench command:
ab -A bob:bobspassword -n 1000 -c 20 http://localhost:8080/index.jsp
After that, JVisualVM is showing a bunch of "live" LDAP threads running.
James Carman said:
See my comment and attached example project.
Rob Winch said:
Thank you for the sample project.
After authenticating, the sample application is performing a search with DefaultLdapAuthoritiesPopulator which uses the managerDn to authenticate and search for user's authorities. This means that pooling is used for looking up the authorities. You can find an example call stack of the thread being created below [1]. A thread is being created for each Connection and is still runnable (live) so long as the Connection (the Runnable for the Thread) has not reached the end of the stream. In short, this is not a bug in Spring Security. You need to configure LDAP Pooling to limit the number of threads that are created.
In respect to the documentation not mentioning anything about Pooling, I think this is a fair point. The documentation does make references to other documentation, but it is a bit vague. There is explicit mention of this on the JavaDoc of AbstractContextSource, but this is a buried (esp when using the namespace configuration). I have created SEC-2171 to enhance the documentation to include information about pooling. Please feel free to submit a pull request.
[1]
com.sun.jndi.ldap.VersionHelper12$3.run()
java.security.AccessController.doPrivileged(PrivilegedAction)
com.sun.jndi.ldap.VersionHelper12.createThread(Runnable)
com.sun.jndi.ldap.Connection.<init>(LdapClient, String, int, String, int, int, OutputStream)
com.sun.jndi.ldap.LdapClient.<init>(String, int, String, int, int, OutputStream, PoolCallback)
com.sun.jndi.ldap.LdapClientFactory.createPooledConnection(PoolCallback)
com.sun.jndi.ldap.pool.Connections.getOrCreateConnection(PooledConnectionFactory)
com.sun.jndi.ldap.pool.Connections.get(long, PooledConnectionFactory)
com.sun.jndi.ldap.pool.Pool.getPooledConnection(Object, long, PooledConnectionFactory)
com.sun.jndi.ldap.LdapPoolManager.getLdapClient(String, int, String, int, int, OutputStream, int, String, Control[], String, String, Object, Hashtable)
com.sun.jndi.ldap.LdapClient.getInstance(boolean, String, int, String, int, int, OutputStream, int, String, Control[], String, String, Object, Hashtable)
com.sun.jndi.ldap.LdapCtx.connect(boolean)
com.sun.jndi.ldap.LdapCtx.<init>(String, String, int, Hashtable, boolean)
com.sun.jndi.ldap.LdapCtxFactory.getUsingURL(String, Hashtable)
com.sun.jndi.ldap.LdapCtxFactory.getUsingURLs(String[], Hashtable)
com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxInstance(Object, Hashtable)
com.sun.jndi.ldap.LdapCtxFactory.getInitialContext(Hashtable)
javax.naming.spi.NamingManager.getInitialContext(Hashtable)
javax.naming.InitialContext.getDefaultInitCtx()
javax.naming.InitialContext.init(Hashtable)
javax.naming.ldap.InitialLdapContext.<init>(Hashtable, Control[])
org.springframework.ldap.core.support.LdapContextSource.getDirContextInstance(Hashtable)
org.springframework.ldap.core.support.AbstractContextSource.createContext(Hashtable)
org.springframework.ldap.core.support.AbstractContextSource.getContext(String, String)
org.springframework.ldap.core.support.AbstractContextSource.getReadOnlyContext()
org.springframework.ldap.core.LdapTemplate.search(SearchExecutor, NameClassPairCallbackHandler, DirContextProcessor)
org.springframework.ldap.core.LdapTemplate.search(String, String, SearchControls, NameClassPairCallbackHandler, DirContextProcessor)
org.springframework.ldap.core.LdapTemplate.search(String, String, SearchControls, ContextMapper, DirContextProcessor)
org.springframework.ldap.core.LdapTemplate.search(String, String, SearchControls, ContextMapper)
org.springframework.security.ldap.SpringSecurityLdapTemplate.searchForSingleAttributeValues(String, String, Object[], String)
org.springframework.security.ldap.userdetails.DefaultLdapAuthoritiesPopulator.getGroupMembershipRoles(String, String)
org.springframework.security.ldap.userdetails.DefaultLdapAuthoritiesPopulator.getGrantedAuthorities(DirContextOperations, String)
org.springframework.security.ldap.authentication.LdapAuthenticationProvider.loadUserAuthorities(DirContextOperations, String, String)
org.springframework.security.ldap.authentication.AbstractLdapAuthenticationProvider.authenticate(Authentication)
org.springframework.security.authentication.ProviderManager.authenticate(Authentication)<2 recursive calls>
org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(ServletRequest, ServletResponse, FilterChain)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(ServletRequest, ServletResponse)
org.springframework.security.web.authentication.ui.DefaultLoginPageGeneratingFilter.doFilter(ServletRequest, ServletResponse, FilterChain)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(ServletRequest, ServletResponse)
org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(ServletRequest, ServletResponse, FilterChain)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(ServletRequest, ServletResponse)
org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(ServletRequest, ServletResponse, FilterChain)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(ServletRequest, ServletResponse)
org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(ServletRequest, ServletResponse, FilterChain)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(ServletRequest, ServletResponse)
org.springframework.security.web.FilterChainProxy.doFilterInternal(ServletRequest, ServletResponse, FilterChain)
org.springframework.security.web.FilterChainProxy.doFilter(ServletRequest, ServletResponse, FilterChain)
org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(Filter, ServletRequest, ServletResponse, FilterChain)
org.springframework.web.filter.DelegatingFilterProxy.doFilter(ServletRequest, ServletResponse, FilterChain)
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletRequest, ServletResponse)
org.eclipse.jetty.servlet.ServletHandler.doHandle(String, Request, HttpServletRequest, HttpServletResponse)
org.eclipse.jetty.server.handler.ScopedHandler.handle(String, Request, HttpServletRequest, HttpServletResponse)
org.eclipse.jetty.security.SecurityHandler.handle(String, Request, HttpServletRequest, HttpServletResponse)
org.eclipse.jetty.server.session.SessionHandler.doHandle(String, Request, HttpServletRequest, HttpServletResponse)
org.eclipse.jetty.server.handler.ContextHandler.doHandle(String, Request, HttpServletRequest, HttpServletResponse)
org.eclipse.jetty.servlet.ServletHandler.doScope(String, Request, HttpServletRequest, HttpServletResponse)
org.eclipse.jetty.server.session.SessionHandler.doScope(String, Request, HttpServletRequest, HttpServletResponse)
org.eclipse.jetty.server.handler.ContextHandler.doScope(String, Request, HttpServletRequest, HttpServletResponse)
org.eclipse.jetty.server.handler.ScopedHandler.handle(String, Request, HttpServletRequest, HttpServletResponse)
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(String, Request, HttpServletRequest, HttpServletResponse)
org.eclipse.jetty.server.handler.HandlerCollection.handle(String, Request, HttpServletRequest, HttpServletResponse)
org.eclipse.jetty.server.handler.HandlerWrapper.handle(String, Request, HttpServletRequest, HttpServletResponse)
org.eclipse.jetty.server.Server.handle(AbstractHttpConnection)
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest()
org.eclipse.jetty.server.AbstractHttpConnection.headerComplete()
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete()
org.eclipse.jetty.http.HttpParser.parseNext()
org.eclipse.jetty.http.HttpParser.parseAvailable()
org.eclipse.jetty.server.AsyncHttpConnection.handle()
org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle()
org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run()
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(Runnable)
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run()
java.lang.Thread.run()
James Carman said:
Sounds good. Once I figure out the pooling settings, I'll try to submit a pull request to help with docs. Thanks for investigating. I don't know how we got to 7000. Tomcat only accepts 250 connections in our config.
James Carman (Migrated from SEC-2170) said:
We configured our server using the namespace configuration as follows:
When we ran a load test against the server, we noticed that we had a memory leak. After investigating the cause, it showed that we had over 7,000 threads running with their "target" being a com.sun.jndi.ldap.Connection object. After much investigation, we decided to try turning off pooling to see if this would help:
The memory leak went away. Now, the documentation for the DefaultSpringSecurityContextSource says that it will turn off pooling "disable pooling when the DN doesn't match the userDn property." It appears as if it is attempting to disable pooling, but for some reason our connections piled up anyway.