search

brettwooldridge / HikariCP

光 HikariCP・A solid, high-performance, JDBC connection pool at last.
Apache License 2.0
20k stars 2.93k forks source link

There are 72989 intances of com.zaxxer.hikari.pool.PoolEntry in Heap #1621

Open yourchanges opened 4 years ago

yourchanges commented 4 years ago

ENV:

/ # java -version
openjdk version "1.8.0_252"
OpenJDK Runtime Environment (build 1.8.0_252-b09)
Eclipse OpenJ9 VM (build openj9-0.20.0, JRE 1.8.0 Linux amd64-64-Bit Compressed References 20200416_608 (JIT enabled, AOT enabled)
OpenJ9   - 05fa2d361
OMR      - d4365f371
JCL      - cfa47e66cd5 based on jdk8u252-b09)

mybatis 3.4.5
HikariCP: 3.2.0
mssql-jdbc: 6.4.0-jre
springboot: 2.1.13.RELEASE

our app process takes 10GB memory, and there are many instances related to hikari:

 5          64261       22105784    com.microsoft.sqlserver.jdbc.SQLServerConnection

/ # jmap -histo:live 1 | grep "hikari"
   27          81586        4568816    com.zaxxer.hikari.pool.PoolEntry
   49          73395        1761480    com.zaxxer.hikari.util.FastList
   55          91685        1466960    com.zaxxer.hikari.pool.HikariPool$$Lambda$634.0000000000000000
  909              4            736    com.zaxxer.hikari.HikariDataSource
 1034              3            504    com.zaxxer.hikari.pool.HikariPool
 1200             14            336    com.zaxxer.hikari.pool.HikariPool$$Lambda$1116.0000000000000000
 1495             12            192    com.zaxxer.hikari.util.UtilityElf$DefaultThreadFactory
 1897              3            120    com.zaxxer.hikari.util.ConcurrentBag
 2088              6             96    com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator
 2370              3             72    com.zaxxer.hikari.util.DriverDataSource
 2406              3             72    com.zaxxer.hikari.pool.ProxyLeakTaskFactory
 2435              3             72    com.zaxxer.hikari.pool.HikariPool$HouseKeeper
 2699              1             56    com.zaxxer.hikari.metrics.micrometer.MicrometerMetricsTracker
 3200              1             48    com.zaxxer.hikari.pool.HikariPool$1
 3674              2             32    com.zaxxer.hikari.pool.PoolBase$NopMetricsTrackerDelegate
 4103              1             32    com.zaxxer.hikari.pool.ProxyLeakTask$1
 4884              1             16    com.zaxxer.hikari.metrics.micrometer.MicrometerMetricsTrackerFactory
 5219              1             16    com.zaxxer.hikari.metrics.micrometer.MicrometerMetricsTracker$$Lambda$962.0000000000000000
 5279              1             16    com.zaxxer.hikari.pool.PoolBase$MetricsTrackerDelegate
 5528              1             16    com.zaxxer.hikari.metrics.micrometer.MicrometerMetricsTracker$$Lambda$963.0000000000000000
 5864              1             16    com.zaxxer.hikari.util.ClockSource$NanosecondClockSource
 5912              1             16    com.zaxxer.hikari.metrics.micrometer.MicrometerMetricsTracker$$Lambda$964.0000000000000000
 6226              1             16    com.zaxxer.hikari.metrics.micrometer.MicrometerMetricsTracker$$Lambda$965.0000000000000000
 6628              1             16    com.zaxxer.hikari.metrics.micrometer.MicrometerMetricsTracker$$Lambda$966.0000000000000000
 6938              1             16    com.zaxxer.hikari.metrics.micrometer.MicrometerMetricsTracker$$Lambda$967.0000000000000000
 7162              1             16    com.zaxxer.hikari.util.ConcurrentBag$$Lambda$632.0000000000000000
 7443              1             16    com.zaxxer.hikari.pool.ProxyConnection$ClosedConnection$$Lambda$635.0000000000000000
 7649              1             16    com.zaxxer.hikari.util.SuspendResumeLock$1

Is there a connection leak? What should I do in next step?

JMAP full output:

/ # jmap -histo:live 1 | more
  num   object count     total size    class name

    1        1487136       47588352    [C
    2        1397932       33550368    java.util.Hashtable$Entry
    3         204215       26139520    java.net.SocksSocketImpl
    4        1437818       23005088    java.lang.String
    5          64261       22105784    com.microsoft.sqlserver.jdbc.SQLServerConnection
    6        1272832       20365312    java.lang.Object
    7        1265447       20247152    [B
    8         361953       14478120    sun.misc.Cleaner
    9           3035       12455640    [Ljava.nio.channels.SelectionKey;
   10         208704       11687424    java.nio.HeapByteBuffer
   11         183676       10285856    java.net.SocketInputStream
   12         183664       10285184    java.net.SocketOutputStream
   13         183694        8817312    java.net.Socket
   14         180463        8662224    sun.security.provider.NativeSHA5$SHA384
   15         180371        8657808    sun.security.provider.NativeSHA
   16         265961        6383064    java.util.ArrayList
   17         360946        5775136    sun.security.provider.NativeDigest$DigestCleanerRunnable
   18         102912        5763072    com.microsoft.sqlserver.jdbc.TDSReader
   19          96973        5430488    [Ljava.util.Hashtable$Entry;
   20          73812        5314464    com.microsoft.sqlserver.jdbc.TDSWriter
   21          71559        5152248    com.microsoft.sqlserver.jdbc.SQLServerConnection$LogonCommand
   22         191142        4587408    java.net.InetAddress$InetAddressHolder
   23          63099        4543128    com.microsoft.sqlserver.jdbc.TDSChannel
   24          63087        4542264    [Ljava.sql.Statement;
   25          70377        4504128    java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask
   26         184873        4436952    java.io.FileDescriptor
   27         262144        4194304    org.apache.logging.log4j.core.async.AsyncLoggerConfigDisruptor$Log4jEventWrapper
   28          72989        4087384    com.zaxxer.hikari.pool.PoolEntry
   29          72937        4084472    java.util.LinkedHashMap
   30         254181        4066896    [Ljava.lang.Object;
   31         168772        4050528    java.util.concurrent.ConcurrentHashMap$Node
   32          54006        3888432    [Ljava.util.HashMap$Node;
   33         106451        3406432    java.util.LinkedHashMap$Entry
   34         130261        3126264    java.util.UUID
   35          78055        3122200    java.util.HashMap
   36          96286        3081152    java.util.concurrent.locks.AbstractQueuedSynchronizer$Node
   37         189584        3033344    java.net.Inet4Address
   38          63177        3032496    java.util.Properties
   39          23847        2861640    sun.security.ssl.SSLSessionImpl
   40          32500        2860000    java.lang.reflect.Method
   41          11422        2649904    sun.security.ssl.SSLSocketImpl
   42         110014        2640336    com.microsoft.sqlserver.jdbc.TDSPacket
   43           9824        2593536    [J
   44          29063        2325040    com.sun.crypto.provider.NativeGaloisCounterMode
   45          43103        2068944    [Ljava.lang.String;
   46          63095        2019040    com.microsoft.sqlserver.jdbc.ServerPortPlaceHolder
   47          27578        1764992    com.sun.crypto.provider.CipherCore
   48          67182        1612368    com.microsoft.sqlserver.jdbc.ActivityId
   49          27968        1566208    javax.crypto.Cipher
   50          65115        1562760    com.microsoft.sqlserver.jdbc.SQLCollation
   51          64962        1559088    com.zaxxer.hikari.util.FastList
   52          95886        1534176    java.io.ByteArrayOutputStream
   53          46780        1496960    sun.security.util.MemoryCache
   54          17978        1438240    java.lang.Class
   55          89476        1431616    java.util.concurrent.Executors$RunnableAdapter
   56          83252        1332032    com.zaxxer.hikari.pool.HikariPool$$Lambda$634.0000000000000000
   57          26896        1291008    sun.security.ssl.CipherBox
   58          52456        1258944    java.lang.ref.ReferenceQueue
   59          78477        1255632    java.util.concurrent.ConcurrentLinkedQueue$Node
   60          30984        1239360    sun.security.util.MemoryCache$SoftCacheEntry
   61          73074        1169184    com.microsoft.sqlserver.jdbc.SQLServerConnectionSecurityManager
   62          67582        1081312    java.lang.Long
   63          26650        1066000    java.lang.Class$ReflectRef
   64          65955        1055280    java.util.concurrent.atomic.AtomicInteger
   65          42894        1029456    java.util.HashMap$Node
   66          42878        1029072    com.mongodb.internal.connection.ConcurrentLinkedDeque$Node
   67          63572        1017152    java.util.concurrent.ConcurrentLinkedQueue
   68          31394        1004608    com.sun.crypto.provider.AESCrypt
   69          24726         989040    java.util.Hashtable
   70          17516         840768    java.security.AccessControlContext
   71          11466         825552    com.microsoft.sqlserver.jdbc.TDSChannel$ProxySocket
   72           1586         824720    [Ljava.lang.ref.Reference;
   73          24090         770880    java.security.MessageDigest$Delegate
   74          11457         733248    java.security.SecureRandom
   75          45243         723888    [Ljava.lang.Class;
   76          29952         718848    org.apache.http.message.BufferedHeader
   77          14003         672144    [I
   78          10403         665792    io.netty.buffer.PoolSubpage
   79          27347         656328    com.sun.crypto.provider.AESCipher$General
   80           5411         649320    java.lang.Thread
   81          11448         641088    sun.security.ssl.InputRecord
   82          26635         639240    java.lang.Class$CacheKey
   83          18186         581952    java.util.concurrent.locks.ReentrantLock$NonfairSync
   84          23354         560496    sun.security.ssl.SSLSessionContextImpl
   85            542         559344    [Lio.netty.util.Recycler$DefaultHandle;
   86          23092         554208    sun.security.ssl.EphemeralKeyManager$EphemeralKeyPair
   87          11494         551712    sun.security.ssl.OutputRecord
   88           6714         537120    java.net.URI
   89           6880         495360    [Ljava.lang.ThreadLocal$ThreadLocalMap$Entry;
   90           2807         494032    org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper
   91           3491         474776    [Ljava.util.concurrent.ConcurrentHashMap$Node;
   92          11440         457600    sun.security.ssl.SSLContextImpl$TLSContext
   93          27734         443744    javax.crypto.spec.SecretKeySpec
   94          26951         431216    sun.security.ssl.Authenticator
   95          17189         412536    java.io.ByteArrayInputStream
   96          12389         396448    sun.security.ssl.HandshakeHash
   97          24605         393680    sun.security.ssl.SessionId
   98          24295         388720    org.apache.http.util.CharArrayBuffer
   99          22797         364752    java.util.concurrent.LinkedBlockingQueue$Node
  100           2811         337320    sun.nio.ch.SocketChannelImpl
  101          13829         331896    com.sun.crypto.provider.TlsMasterSecretGenerator$TlsMasterSecretKey
  102           8263         330520    java.util.WeakHashMap$Entry
  103          12697         304728    java.util.LinkedList$Node
  104           5383         301448    java.util.concurrent.ThreadPoolExecutor$Worker
  105           2840         295360    com.mongodb.connection.ServerDescription
  106          17758         284128    java.util.concurrent.atomic.AtomicReference
  107          11468         275232    sun.security.ssl.AppOutputStream
  108          11464         275136    com.microsoft.sqlserver.jdbc.TDSChannel$PermissiveX509TrustManager
  109          11463         275112    sun.security.provider.SecureRandom
  110          11459         275016    com.microsoft.sqlserver.jdbc.TDSChannel$ProxyInputStream
  111          11458         274992    com.microsoft.sqlserver.jdbc.TDSChannel$ProxyOutputStream
  112          11420         274080    sun.security.ssl.AppInputStream
  113          16843         269488    java.util.concurrent.locks.ReentrantLock
  114           4691         262696    java.util.zip.Inflater
  115           6324         252960    org.aspectj.weaver.reflect.ShadowMatchImpl
  116           3650         233600    java.util.concurrent.ConcurrentHashMap
  117          13030         208480    [Ljava.security.cert.X509Certificate;
  118           8605         206520    java.lang.ref.WeakReference
  119           6324         202368    org.aspectj.weaver.patterns.ExposedState
  120           2804         201888    [Ljava.util.WeakHashMap$Entry;
  121          12238         195808    java.util.concurrent.atomic.AtomicLong
  122           4067         195216    sun.nio.cs.UTF_8$Encoder
  123           3031         193984    java.lang.reflect.Field
  124           2419         193520    java.lang.invoke.DirectHandle
  125          11782         188512    java.util.HashSet
  126          11621         185936    [Ljava.security.ProtectionDomain;
  127          11447         183152    [Lsun.security.ssl.EphemeralKeyManager$EphemeralKeyPair;
  128          11441         183056    sun.security.ssl.AbstractTrustManagerWrapper
  129          11439         183024    sun.security.ssl.EphemeralKeyManager
  130           5687         181984    java.math.BigInteger
  131           2810         179840    org.springframework.core.annotation.AnnotationAttributes
  132           5610         179520    java.util.concurrent.Semaphore$NonfairSync
  133           5480         175360    java.lang.OutOfMemoryError
  134          10959         175344    org.apache.ibatis.scripting.xmltags.StaticTextSqlNode
  135           6879         165096    java.lang.ThreadLocal$ThreadLocalMap
  136           2901         162456    sun.nio.ch.SocketAdaptor
  137           2523         161472    java.util.zip.GZIPInputStream
  138            292         161184    io.netty.util.internal.shaded.org.jctools.queues.MpscUnboundedArrayQueue
  139           9887         158192    java.util.HashMap$KeySet
  140           9714         155424    org.springframework.core.MethodClassKey
  141           6430         154320    java.net.InetSocketAddress$InetSocketAddressHolder
  142           1907         152560    org.HdrHistogram.HistogramIterationValue
  143           3167         152016    org.springframework.core.ResolvableType
  144           9238         147808    java.util.Collections$UnmodifiableSet
  145           6135         147240    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject
  146           4512         144384    org.springframework.boot.actuate.trace.http.HttpTrace
  147           4457         142624    java.util.concurrent.CountDownLatch$Sync
  148           4446         142272    java.lang.ref.SoftReference
  149           5320         127680    org.apache.ibatis.scripting.xmltags.IfSqlNode
  150           3874         123968    java.util.HashMap$KeyIterator
  151            117         120744    [[C
  152           1674         120528    java.lang.Class$ReflectCache
  153           3739         119648    org.apache.http.impl.io.ContentLengthInputStream
  154            917         117376    org.HdrHistogram.PercentileIterator
  155           7108         113728    java.util.LinkedHashSet
  156            540         112320    org.springframework.context.annotation.ConfigurationClassBeanDefinitionReader$ConfigurationClassBeanDefinition
  157           2805         112200    java.util.concurrent.LinkedBlockingDeque
  158           2804         112160    sun.nio.ch.SelectionKeyImpl
  159           4645         111480    java.util.LinkedList
  160           6963         111408    java.util.concurrent.atomic.AtomicBoolean
  161            600         110400    com.jiuji.oa.orginfo.areainfo.vo.AreaInfo
  162           4595         110280    java.net.InetAddress$CacheEntry
  163           4482         107568    sun.reflect.generics.tree.SimpleClassTypeSignature
  164           4293         103032    java.time.Instant
  165           4286         102864    org.springframework.boot.actuate.trace.http.HttpTrace$Request
  166            918         102816    org.HdrHistogram.RecordedValuesIterator
  167            156         101088    io.netty.util.internal.shaded.org.jctools.queues.MpscArrayQueue
  168           3145         100640    java.lang.ThreadLocal$ThreadLocalMap$Entry
  169           3132         100224    com.mongodb.connection.ClusterDescription
  170           6220          99520    java.net.InetSocketAddress
  171           4130          99120    java.lang.StringCoding$StringEncoder
  172           6081          97296    java.lang.Thread$ThreadLock
  173           6050          96800    [Ljava.net.InetAddress;
  174           5912          94592    java.lang.J9VMInternals$ClassInitializationLock
  175            470          94000    org.springframework.beans.factory.support.RootBeanDefinition
  176           5793          92688    org.apache.ibatis.scripting.xmltags.MixedSqlNode
  177           5664          90624    java.util.concurrent.Semaphore
  178           1239          89208    java.lang.reflect.Constructor
  179           1827          87696    org.apache.http.message.BasicHttpResponse
  180           5430          86880    java.util.Collections$UnmodifiableRandomAccessList
  181           5364          85824    org.apache.ibatis.scripting.xmltags.ExpressionEvaluator
  182           3533          84792    sun.reflect.annotation.AnnotationInvocationHandler
  183           2648          84736    org.apache.http.HttpHost
  184           5278          84448    org.apache.http.message.BasicHeader
  185           2632          84224    org.apache.http.auth.AuthState
  186           3509          84216    java.util.ArrayList$Itr
  187            460          80960    org.HdrHistogram.AtomicHistogram
  188           2529          80928    org.apache.http.entity.BasicHttpEntity
  189           2474          79168    java.util.TreeMap$Entry
  190           1969          78760    java.lang.invoke.MethodType
  191            578          78608    [Lio.netty.buffer.PoolSubpage;
  192           4902          78432    org.springframework.core.annotation.AnnotationUtils$DefaultValueHolder
  193           3230          77520    org.LatencyUtils.PauseDetector$PauseNotification
  194            461          77448    org.HdrHistogram.Histogram
  195           1209          77376    java.lang.ClassValue$ClassValueMap
  196           1590          76320    java.util.WeakHashMap
  197           4701          75216    java.util.zip.ZStreamRef
  198           4684          74944    java.util.LinkedHashMap$LinkedEntrySet
yourchanges commented 4 years ago

maybe related to #1420

yourchanges commented 4 years ago

we use AbstractRoutingDataSource to manage 4 datasource, here is the ds build code:

 String driverClassName = properties.getDriverClassName();
            String url = properties.getUrl();
            String username = properties.getUsername();
            String password = properties.getPassword();

            DataSourceBuilder factory = DataSourceBuilder.create().driverClassName(driverClassName).url(url)
                    .username(username).password(password).type(dataSourceType);
            DataSource ds = factory.build();
            if (ds instanceof HikariDataSource) {
                HikariDataSource hikariDataSource = (HikariDataSource) ds;
                hikariDataSource.setMaximumPoolSize(properties.getMaxPoolSize() == null
                        ? 100 : properties.getMaxPoolSize());
                hikariDataSource.setMaxLifetime(properties.getMaxLeftTime() == null
                        ? 30000 : properties.getMaxLeftTime());
            }
            return ds;