search

brettwooldridge / HikariCP

光 HikariCP・A solid, high-performance, JDBC connection pool at last.
Apache License 2.0
19.96k stars 2.93k forks source link

Leak detection threshold does not display error #1364

Open simran66 opened 5 years ago

simran66 commented 5 years ago

Environment

HikariCP version: 3.3.1
JDK version     : 1.8.0_111
Database        : MySQL
Driver version  : 5.1

Here's my config values:

2019-04-25 16:58:37 [pool-1-thread-17] DEBUG com.zaxxer.hikari.HikariConfig - Driver class com.mysql.cj.jdbc.Driver found in Thread context class loader sun.misc.Launcher$AppClassLoader@18b4aac2 2019-04-25 16:58:37 [pool-1-thread-17] DEBUG com.zaxxer.hikari.HikariConfig - MySQLConnectionPool - configuration: 2019-04-25 16:58:37 [pool-1-thread-17] DEBUG com.zaxxer.hikari.HikariConfig - allowPoolSuspension.............false 2019-04-25 16:58:37 [pool-1-thread-17] DEBUG com.zaxxer.hikari.HikariConfig - autoCommit......................true 2019-04-25 16:58:37 [pool-1-thread-17] DEBUG com.zaxxer.hikari.HikariConfig - catalog.........................none 2019-04-25 16:58:37 [pool-1-thread-17] DEBUG com.zaxxer.hikari.HikariConfig - connectionInitSql...............none 2019-04-25 16:58:37 [pool-1-thread-17] DEBUG com.zaxxer.hikari.HikariConfig - connectionTestQuery.............none 2019-04-25 16:58:37 [pool-1-thread-17] DEBUG com.zaxxer.hikari.HikariConfig - connectionTimeout...............30000 2019-04-25 16:58:37 [pool-1-thread-17] DEBUG com.zaxxer.hikari.HikariConfig - dataSource......................none 2019-04-25 16:58:37 [pool-1-thread-17] DEBUG com.zaxxer.hikari.HikariConfig - dataSourceClassName.............none 2019-04-25 16:58:37 [pool-1-thread-17] DEBUG com.zaxxer.hikari.HikariConfig - dataSourceJNDI..................none 2019-04-25 16:58:37 [pool-1-thread-17] DEBUG com.zaxxer.hikari.HikariConfig - dataSourceProperties............{autoReconnect=true, password=<masked>} 2019-04-25 16:58:37 [pool-1-thread-17] DEBUG com.zaxxer.hikari.HikariConfig - driverClassName................."com.mysql.cj.jdbc.Driver" 2019-04-25 16:58:37 [pool-1-thread-17] DEBUG com.zaxxer.hikari.HikariConfig - healthCheckProperties...........{} 2019-04-25 16:58:37 [pool-1-thread-17] DEBUG com.zaxxer.hikari.HikariConfig - healthCheckRegistry.............none 2019-04-25 16:58:37 [pool-1-thread-17] DEBUG com.zaxxer.hikari.HikariConfig - idleTimeout.....................600000 2019-04-25 16:58:37 [pool-1-thread-17] DEBUG com.zaxxer.hikari.HikariConfig - initializationFailTimeout.......1 2019-04-25 16:58:37 [pool-1-thread-17] DEBUG com.zaxxer.hikari.HikariConfig - isolateInternalQueries..........false 2019-04-25 16:58:37 [pool-1-thread-17] DEBUG com.zaxxer.hikari.HikariConfig - jdbcUrl.........................jdbc:mysql://localhost:3306/push_notifications 2019-04-25 16:58:37 [pool-1-thread-17] DEBUG com.zaxxer.hikari.HikariConfig - leakDetectionThreshold..........15000 2019-04-25 16:58:37 [pool-1-thread-17] DEBUG com.zaxxer.hikari.HikariConfig - maxLifetime.....................1800000 2019-04-25 16:58:37 [pool-1-thread-17] DEBUG com.zaxxer.hikari.HikariConfig - maximumPoolSize.................40 2019-04-25 16:58:37 [pool-1-thread-17] DEBUG com.zaxxer.hikari.HikariConfig - metricRegistry..................com.codahale.metrics.MetricRegistry@48cf2b5a 2019-04-25 16:58:37 [pool-1-thread-17] DEBUG com.zaxxer.hikari.HikariConfig - metricsTrackerFactory...........none 2019-04-25 16:58:37 [pool-1-thread-17] DEBUG com.zaxxer.hikari.HikariConfig - minimumIdle.....................40 2019-04-25 16:58:37 [pool-1-thread-17] DEBUG com.zaxxer.hikari.HikariConfig - password........................<masked> 2019-04-25 16:58:37 [pool-1-thread-17] DEBUG com.zaxxer.hikari.HikariConfig - poolName........................"MySQLConnectionPool" 2019-04-25 16:58:37 [pool-1-thread-17] DEBUG com.zaxxer.hikari.HikariConfig - readOnly........................false 2019-04-25 16:58:37 [pool-1-thread-17] DEBUG com.zaxxer.hikari.HikariConfig - registerMbeans..................false 2019-04-25 16:58:37 [pool-1-thread-17] DEBUG com.zaxxer.hikari.HikariConfig - scheduledExecutor...............none 2019-04-25 16:58:37 [pool-1-thread-17] DEBUG com.zaxxer.hikari.HikariConfig - schema..........................none 2019-04-25 16:58:37 [pool-1-thread-17] DEBUG com.zaxxer.hikari.HikariConfig - threadFactory...................internal 2019-04-25 16:58:37 [pool-1-thread-17] DEBUG com.zaxxer.hikari.HikariConfig - transactionIsolation............default 2019-04-25 16:58:37 [pool-1-thread-17] DEBUG com.zaxxer.hikari.HikariConfig - username........................"root" 2019-04-25 16:58:37 [pool-1-thread-17] DEBUG com.zaxxer.hikari.HikariConfig - validationTimeout...............5000

I have on purpose left a connection open which results in connection timeouts but I never get leak detection threshold error from hikari. I have tried setting leakDetectionthreshold value to the minimum possible 2000 as well. Metric registry is working fine though. What's happening? Also, I read somewhere that following

```


should give hikari logs every 30s. I see this only when the application starts.

Here's how I set values:

` if (ds == null || ds.isClosed()) {
                //after setting up basic ones

                config.addDataSourceProperty("autoReconnect", true);

                config.setLeakDetectionThreshold(15000);

                config.setMetricRegistry(ApplicationUtils.getMetricRegistry());

                config.setPoolName("MySQLConnectionPool");
                ds = new HikariDataSource(config);
                LOGGER.info("Created DS in MySQL", ds);
            }`

No logs from Hikari House keeper at all. Does not look like it's running
simran66 commented 5 years ago

@brettwooldridge

brettwooldridge commented 5 years ago

@simran66 This is very strange. Can you set a breakpoint in HikariCP's housekeeper code to see if the code is indeed executing?

I will note that autoReconnect should not be used.

simran66 commented 5 years ago

@brettwooldridge I took a thread dump by running kill -3 pidOfProcess and this is what it says: What does it mean exactly?

`
"MySQLConnectionPool housekeeper" #136 daemon prio=5 os_prio=31 tid=0x00007fd489342800 nid=0xe303 waiting on condition [0x000070000a9a0000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x000000074049fe60> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

"Abandoned connection cleanup thread" #135 daemon prio=5 os_prio=31 tid=0x00007fd488b42800 nid=0x11603 in Object.wait() [0x000070000a89d000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
    - locked <0x0000000740bc4f38> (a java.lang.ref.ReferenceQueue$Lock)
    at com.mysql.cj.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:70)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)`
simran66 commented 5 years ago

here is complete thread dump @brettwooldridge `2019-04-26 11:14:21 Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.192-b12 mixed mode):

"PostgresConnectionPool housekeeper" #140 daemon prio=5 os_prio=31 tid=0x00007fd48a279000 nid=0xe703 waiting on condition [0x000070000aca9000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"PostgreSQL-JDBC-SharedTimer-1" #139 daemon prio=5 os_prio=31 tid=0x00007fd48988d800 nid=0xe603 in Object.wait() [0x000070000aba6000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.util.TimerThread.mainLoop(Timer.java:552)

"MySQLConnectionPool housekeeper" #136 daemon prio=5 os_prio=31 tid=0x00007fd489342800 nid=0xe303 waiting on condition [0x000070000a9a0000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"Abandoned connection cleanup thread" #135 daemon prio=5 os_prio=31 tid=0x00007fd488b42800 nid=0x11603 in Object.wait() [0x000070000a89d000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)

"kafka-coordinator-heartbeat-thread | test-consumer-group" #133 daemon prio=5 os_prio=31 tid=0x00007fd48a4f9800 nid=0xe003 in Object.wait() [0x000070000a79a000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:906)

"kafka-coordinator-heartbeat-thread | test-consumer-group" #132 daemon prio=5 os_prio=31 tid=0x00007fd489338800 nid=0x11803 in Object.wait() [0x000070000a697000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:906)

"kafka-coordinator-heartbeat-thread | test-consumer-group" #131 daemon prio=5 os_prio=31 tid=0x00007fd489337800 nid=0x11903 in Object.wait() [0x000070000a594000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:906)

"kafka-coordinator-heartbeat-thread | test-consumer-group_data_science" #109 daemon prio=5 os_prio=31 tid=0x00007fd489bd5800 nid=0x11a03 waiting for monitor entry [0x000070000a491000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.disableWakeups(ConsumerNetworkClient.java:422)

"kafka-coordinator-heartbeat-thread | test-consumer-group_data_science" #111 daemon prio=5 os_prio=31 tid=0x00007fd48a131800 nid=0xdd03 waiting for monitor entry [0x000070000a38e000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.disableWakeups(ConsumerNetworkClient.java:422)

"kafka-coordinator-heartbeat-thread | test-consumer-group" #115 daemon prio=5 os_prio=31 tid=0x00007fd48a131000 nid=0x11c03 in Object.wait() [0x000070000a28b000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:906)

"kafka-coordinator-heartbeat-thread | test-consumer-group" #122 daemon prio=5 os_prio=31 tid=0x00007fd48a130000 nid=0x11e03 in Object.wait() [0x000070000a188000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:906)

"kafka-coordinator-heartbeat-thread | test-consumer-group_data_science" #129 daemon prio=5 os_prio=31 tid=0x00007fd489bd4800 nid=0x11f03 in Object.wait() [0x000070000a085000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:906)

"kafka-coordinator-heartbeat-thread | test-consumer-group_quick_reply" #130 daemon prio=5 os_prio=31 tid=0x00007fd489bd4000 nid=0xda03 in Object.wait() [0x0000700009f82000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:906)

"kafka-coordinator-heartbeat-thread | test-consumer-group" #128 daemon prio=5 os_prio=31 tid=0x00007fd489bd3000 nid=0x12203 in Object.wait() [0x0000700009e7f000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:906)

"kafka-coordinator-heartbeat-thread | test-consumer-group" #127 daemon prio=5 os_prio=31 tid=0x00007fd489b07000 nid=0x12403 waiting for monitor entry [0x0000700009d7c000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.disableWakeups(ConsumerNetworkClient.java:422)

"kafka-coordinator-heartbeat-thread | test-consumer-groupsmart_send" #126 daemon prio=5 os_prio=31 tid=0x00007fd489337000 nid=0x12603 in Object.wait() [0x0000700009c79000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:906)

"kafka-coordinator-heartbeat-thread | test-consumer-group" #125 daemon prio=5 os_prio=31 tid=0x00007fd489b06000 nid=0x12803 in Object.wait() [0x0000700009b76000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:906)

"kafka-coordinator-heartbeat-thread | test-consumer-group" #124 daemon prio=5 os_prio=31 tid=0x00007fd48a129800 nid=0xd903 in Object.wait() [0x0000700009a73000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:906)

"kafka-coordinator-heartbeat-thread | test-consumer-group" #123 daemon prio=5 os_prio=31 tid=0x00007fd48a129000 nid=0xd803 waiting for monitor entry [0x0000700009970000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.disableWakeups(ConsumerNetworkClient.java:422)

"kafka-coordinator-heartbeat-thread | test-consumer-group" #121 daemon prio=5 os_prio=31 tid=0x00007fd48a16c000 nid=0xd703 in Object.wait() [0x000070000986d000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:906)

"kafka-coordinator-heartbeat-thread | test-consumer-group" #120 daemon prio=5 os_prio=31 tid=0x00007fd488b3a800 nid=0x12d03 in Object.wait() [0x000070000976a000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:906)

"kafka-coordinator-heartbeat-thread | test-consumer-group" #119 daemon prio=5 os_prio=31 tid=0x00007fd489b05800 nid=0xd603 in Object.wait() [0x0000700009667000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:906)

"kafka-coordinator-heartbeat-thread | test-consumer-group" #118 daemon prio=5 os_prio=31 tid=0x00007fd489336000 nid=0xd403 in Object.wait() [0x0000700009564000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:906)

"kafka-coordinator-heartbeat-thread | test-consumer-groupsmart_send" #117 daemon prio=5 os_prio=31 tid=0x00007fd489b04800 nid=0xd203 in Object.wait() [0x0000700009461000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:906)

"kafka-coordinator-heartbeat-thread | test-consumer-group" #116 daemon prio=5 os_prio=31 tid=0x00007fd48a16b800 nid=0xd103 in Object.wait() [0x000070000935e000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:906)

"kafka-coordinator-heartbeat-thread | test-consumer-group_data_science" #114 daemon prio=5 os_prio=31 tid=0x00007fd489b04000 nid=0xcf03 in Object.wait() [0x000070000925b000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:906)

"kafka-coordinator-heartbeat-thread | test-consumer-group_quick_reply" #113 daemon prio=5 os_prio=31 tid=0x00007fd48a16a800 nid=0x13003 waiting for monitor entry [0x0000700009158000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.disableWakeups(ConsumerNetworkClient.java:422)

"kafka-coordinator-heartbeat-thread | test-consumer-group_data_science" #112 daemon prio=5 os_prio=31 tid=0x00007fd488b34000 nid=0x13203 in Object.wait() [0x0000700009055000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:906)

"kafka-coordinator-heartbeat-thread | test-consumer-group_data_science" #110 daemon prio=5 os_prio=31 tid=0x00007fd48a16a000 nid=0x13303 in Object.wait() [0x0000700008f52000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:906)

"kafka-coordinator-heartbeat-thread | test-consumer-group_data_science" #108 daemon prio=5 os_prio=31 tid=0x00007fd489335800 nid=0x13403 in Object.wait() [0x0000700008e4f000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:906)

"kafka-coordinator-heartbeat-thread | test-consumer-group" #107 daemon prio=5 os_prio=31 tid=0x00007fd489b03000 nid=0xcb03 in Object.wait() [0x0000700008d4c000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:906)

"kafka-coordinator-heartbeat-thread | test-consumer-group" #106 daemon prio=5 os_prio=31 tid=0x00007fd488b33800 nid=0x13703 in Object.wait() [0x0000700008c49000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:906)

"kafka-coordinator-heartbeat-thread | test-consumer-group_data_science" #105 daemon prio=5 os_prio=31 tid=0x00007fd488b32800 nid=0x13903 in Object.wait() [0x0000700008b46000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:906)

"kafka-coordinator-heartbeat-thread | test-consumer-group" #87 daemon prio=5 os_prio=31 tid=0x00007fd488b32000 nid=0xc903 in Object.wait() [0x0000700008a43000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:906)

"kafka-coordinator-heartbeat-thread | test-consumer-group_optin_welcome" #85 daemon prio=5 os_prio=31 tid=0x00007fd489b02800 nid=0xc703 waiting for monitor entry [0x0000700008940000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.disableWakeups(ConsumerNetworkClient.java:422)

"kafka-coordinator-heartbeat-thread | test-consumer-group_optin_welcome" #104 daemon prio=5 os_prio=31 tid=0x00007fd48a163800 nid=0xc603 in Object.wait() [0x000070000883d000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:906)

"kafka-coordinator-heartbeat-thread | test-consumer-group" #103 daemon prio=5 os_prio=31 tid=0x00007fd489b01800 nid=0xc403 in Object.wait() [0x000070000873a000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:906)

"kafka-coordinator-heartbeat-thread | test-consumer-group" #102 daemon prio=5 os_prio=31 tid=0x00007fd488aeb000 nid=0xc303 in Object.wait() [0x0000700008637000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:906)

"kafka-coordinator-heartbeat-thread | test-consumer-group" #101 daemon prio=5 os_prio=31 tid=0x00007fd489b01000 nid=0x13c03 in Object.wait() [0x0000700008534000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:906)

"kafka-coordinator-heartbeat-thread | test-consumer-group" #100 daemon prio=5 os_prio=31 tid=0x00007fd489334800 nid=0xc103 in Object.wait() [0x0000700008431000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:906)

"kafka-coordinator-heartbeat-thread | test-consumer-group" #99 daemon prio=5 os_prio=31 tid=0x00007fd48a1ac000 nid=0x13f03 in Object.wait() [0x000070000832e000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:906)

"kafka-coordinator-heartbeat-thread | test-consumer-group" #98 daemon prio=5 os_prio=31 tid=0x00007fd4890de000 nid=0x14103 in Object.wait() [0x000070000822b000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:906)

"kafka-coordinator-heartbeat-thread | test-consumer-group" #97 daemon prio=5 os_prio=31 tid=0x00007fd4890dd000 nid=0x14303 in Object.wait() [0x0000700008128000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:906)

"kafka-coordinator-heartbeat-thread | test-consumer-group" #96 daemon prio=5 os_prio=31 tid=0x00007fd489b00000 nid=0xbf03 in Object.wait() [0x0000700008025000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:906)

"kafka-coordinator-heartbeat-thread | test-consumer-group" #95 daemon prio=5 os_prio=31 tid=0x00007fd4890dc800 nid=0xbd03 in Object.wait() [0x0000700007f22000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:906)

"kafka-coordinator-heartbeat-thread | test-consumer-group" #94 daemon prio=5 os_prio=31 tid=0x00007fd48a1ab800 nid=0xbb03 waiting for monitor entry [0x0000700007e1f000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.disableWakeups(ConsumerNetworkClient.java:422)

"kafka-coordinator-heartbeat-thread | test-consumer-group" #93 daemon prio=5 os_prio=31 tid=0x00007fd489b60000 nid=0xba03 in Object.wait() [0x0000700007d1c000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:906)

"kafka-coordinator-heartbeat-thread | test-consumer-group_data_science" #92 daemon prio=5 os_prio=31 tid=0x00007fd488a8c000 nid=0x14503 in Object.wait() [0x0000700007c19000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:906)

"kafka-coordinator-heartbeat-thread | test-consumer-group" #91 daemon prio=5 os_prio=31 tid=0x00007fd489b5f000 nid=0xb803 in Object.wait() [0x0000700007b16000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:906)

"kafka-coordinator-heartbeat-thread | test-consumer-group_data_science" #90 daemon prio=5 os_prio=31 tid=0x00007fd489b5e800 nid=0x14803 in Object.wait() [0x0000700007a13000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:906)

"kafka-coordinator-heartbeat-thread | test-consumer-group" #89 daemon prio=5 os_prio=31 tid=0x00007fd489a68800 nid=0x14903 in Object.wait() [0x0000700007910000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:906)

"kafka-coordinator-heartbeat-thread | test-consumer-group" #88 daemon prio=5 os_prio=31 tid=0x00007fd488af7000 nid=0x14a03 in Object.wait() [0x000070000780d000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:906)

"kafka-coordinator-heartbeat-thread | test-consumer-group" #86 daemon prio=5 os_prio=31 tid=0x00007fd48a063000 nid=0x14b03 in Object.wait() [0x000070000770a000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:906)

"DestroyJavaVM" #84 prio=5 os_prio=31 tid=0x00007fd489a66000 nid=0x2603 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE

"Timer-1" #82 prio=5 os_prio=31 tid=0x00007fd48a432000 nid=0xb303 in Object.wait() [0x0000700007607000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method)

"Timer-0" #81 prio=5 os_prio=31 tid=0x00007fd48a431800 nid=0xb203 in Object.wait() [0x0000700007504000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method)

"pool-1-thread-49" #80 prio=5 os_prio=31 tid=0x00007fd4890d7000 nid=0x14e03 runnable [0x0000700007401000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

"pool-1-thread-48" #79 prio=5 os_prio=31 tid=0x00007fd489b68800 nid=0x14f03 runnable [0x00007000072fe000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

"pool-1-thread-47" #78 prio=5 os_prio=31 tid=0x00007fd489b68000 nid=0xaf03 runnable [0x00007000071fb000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

"pool-1-thread-46" #77 prio=5 os_prio=31 tid=0x00007fd488af3800 nid=0xad03 runnable [0x00007000070f8000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

"pool-1-thread-45" #76 prio=5 os_prio=31 tid=0x00007fd489b67000 nid=0x15003 runnable [0x0000700006ff5000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

"pool-1-thread-44" #75 prio=5 os_prio=31 tid=0x00007fd488af2800 nid=0xab03 runnable [0x0000700006ef2000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

"pool-1-thread-43" #74 prio=5 os_prio=31 tid=0x00007fd489329000 nid=0x15303 runnable [0x0000700006def000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

"pool-1-thread-42" #73 prio=5 os_prio=31 tid=0x00007fd489b66800 nid=0xaa03 runnable [0x0000700006cec000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

"pool-1-thread-41" #72 prio=5 os_prio=31 tid=0x00007fd489b2f800 nid=0x7703 runnable [0x0000700006be9000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

"pool-1-thread-40" #71 prio=5 os_prio=31 tid=0x00007fd489b2f000 nid=0x7503 runnable [0x0000700006ae6000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

"pool-1-thread-39" #70 prio=5 os_prio=31 tid=0x00007fd488af2000 nid=0x7a03 runnable [0x00007000069e3000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

"pool-1-thread-38" #69 prio=5 os_prio=31 tid=0x00007fd489b2e000 nid=0x7b03 runnable [0x00007000068e0000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

"pool-1-thread-37" #68 prio=5 os_prio=31 tid=0x00007fd489b2d800 nid=0x7d03 runnable [0x00007000067dd000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

"pool-1-thread-36" #67 prio=5 os_prio=31 tid=0x00007fd489b2c800 nid=0x7203 runnable [0x00007000066da000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

"pool-1-thread-35" #66 prio=5 os_prio=31 tid=0x00007fd489b2c000 nid=0x7f03 runnable [0x00007000065d7000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

"pool-1-thread-34" #65 prio=5 os_prio=31 tid=0x00007fd489328800 nid=0x7003 runnable [0x00007000064d4000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

"pool-1-thread-33" #64 prio=5 os_prio=31 tid=0x00007fd489315000 nid=0x6f03 runnable [0x00007000063d1000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

"pool-1-thread-32" #63 prio=5 os_prio=31 tid=0x00007fd489b2b000 nid=0x8203 runnable [0x00007000062ce000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

"pool-1-thread-31" #62 prio=5 os_prio=31 tid=0x00007fd489a6f800 nid=0x8403 runnable [0x00007000061cb000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

"pool-1-thread-30" #61 prio=5 os_prio=31 tid=0x00007fd489a6f000 nid=0x8603 runnable [0x00007000060c8000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

"pool-1-thread-29" #60 prio=5 os_prio=31 tid=0x00007fd489314800 nid=0x8803 runnable [0x0000700005fc5000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

"pool-1-thread-28" #59 prio=5 os_prio=31 tid=0x00007fd489a6e000 nid=0x8a03 runnable [0x0000700005ec2000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

"pool-1-thread-27" #58 prio=5 os_prio=31 tid=0x00007fd489a6d800 nid=0x8b03 runnable [0x0000700005dbf000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

"pool-1-thread-26" #57 prio=5 os_prio=31 tid=0x00007fd4891c6800 nid=0x8c03 runnable [0x0000700005cbc000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

"pool-1-thread-25" #56 prio=5 os_prio=31 tid=0x00007fd488af1000 nid=0x8e03 runnable [0x0000700005bb9000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

"pool-1-thread-24" #55 prio=5 os_prio=31 tid=0x00007fd489a6c800 nid=0x8f03 runnable [0x0000700005ab6000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

"pool-1-thread-23" #54 prio=5 os_prio=31 tid=0x00007fd489a6c000 nid=0x6b03 runnable [0x00007000059b3000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

"pool-1-thread-22" #53 prio=5 os_prio=31 tid=0x00007fd489a6b000 nid=0x9203 runnable [0x00007000058b0000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

"pool-1-thread-21" #52 prio=5 os_prio=31 tid=0x00007fd4891c6000 nid=0x9403 runnable [0x00007000057ad000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

"pool-1-thread-20" #51 prio=5 os_prio=31 tid=0x00007fd48933c000 nid=0x6903 runnable [0x00007000056aa000] java.lang.Thread.State: RUNNABLE at com.wingify.pushcrew.threads.QuickReplyConsumer.processMessages(QuickReplyConsumer.java:92) at com.wingify.pushcrew.threads.QuickReplyConsumer.run(QuickReplyConsumer.java:72) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

"pool-1-thread-19" #50 prio=5 os_prio=31 tid=0x00007fd4898a7800 nid=0x9603 runnable [0x00007000055a7000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

"pool-1-thread-18" #49 prio=5 os_prio=31 tid=0x00007fd488af0800 nid=0x9703 runnable [0x00007000054a4000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

"pool-1-thread-17" #48 prio=5 os_prio=31 tid=0x00007fd488aef800 nid=0x9903 waiting on condition [0x00007000053a1000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at com.wingify.pushcrew.utils.MysqlUtils.quickReplyClickCount(MysqlUtils.java:673) at com.wingify.pushcrew.threads.CountQuickReplyConsumer.run(CountQuickReplyConsumer.java:65) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

"pool-1-thread-16" #47 prio=5 os_prio=31 tid=0x00007fd48933b800 nid=0x6703 runnable [0x000070000529e000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

"pool-1-thread-15" #46 prio=5 os_prio=31 tid=0x00007fd488aef000 nid=0x6503 runnable [0x000070000519b000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

"pool-1-thread-14" #45 prio=5 os_prio=31 tid=0x00007fd488aee000 nid=0x6303 runnable [0x0000700005098000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

"pool-1-thread-13" #44 prio=5 os_prio=31 tid=0x00007fd488aed800 nid=0x9b03 runnable [0x0000700004f95000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

"pool-1-thread-12" #43 prio=5 os_prio=31 tid=0x00007fd488aec800 nid=0x6003 runnable [0x0000700004e92000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

"pool-1-thread-11" #42 prio=5 os_prio=31 tid=0x00007fd488aec000 nid=0x9d03 runnable [0x0000700004d8f000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

"pool-1-thread-10" #41 prio=5 os_prio=31 tid=0x00007fd488aea800 nid=0x9e03 runnable [0x0000700004c8c000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

"pool-1-thread-9" #40 prio=5 os_prio=31 tid=0x00007fd488ae9800 nid=0x9f03 runnable [0x0000700004b89000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

"pool-1-thread-8" #39 prio=5 os_prio=31 tid=0x00007fd488adf800 nid=0xa103 runnable [0x0000700004a86000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

"pool-1-thread-7" #38 prio=5 os_prio=31 tid=0x00007fd488adf000 nid=0xa203 runnable [0x0000700004983000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

"pool-1-thread-6" #37 prio=5 os_prio=31 tid=0x00007fd4898a7000 nid=0xa303 runnable [0x0000700004880000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

"pool-1-thread-5" #36 prio=5 os_prio=31 tid=0x00007fd489112000 nid=0x5a03 runnable [0x000070000477d000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

"pool-1-thread-4" #35 prio=5 os_prio=31 tid=0x00007fd489111800 nid=0xa503 runnable [0x000070000467a000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

"pool-1-thread-3" #34 prio=5 os_prio=31 tid=0x00007fd4892ec000 nid=0xa603 runnable [0x0000700004577000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

"pool-1-thread-2" #33 prio=5 os_prio=31 tid=0x00007fd4898f9800 nid=0x5803 runnable [0x0000700004474000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

"pool-1-thread-1" #32 prio=5 os_prio=31 tid=0x00007fd488ada000 nid=0xa803 runnable [0x0000700004371000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

"kafka-producer-network-thread | producer-2" #22 daemon prio=5 os_prio=31 tid=0x00007fd489a97000 nid=0xa903 runnable [0x000070000426e000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

"kafka-producer-network-thread | producer-1" #21 daemon prio=5 os_prio=31 tid=0x00007fd4899e3000 nid=0x4003 runnable [0x000070000416b000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

"Attach Listener" #13 daemon prio=9 os_prio=31 tid=0x00007fd4899fe800 nid=0x4103 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE

"metrics-logger-reporter-2-thread-1" #11 daemon prio=5 os_prio=31 tid=0x00007fd489a18800 nid=0x4303 waiting on condition [0x0000700003f65000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"metrics-console-reporter-1-thread-1" #10 daemon prio=5 os_prio=31 tid=0x00007fd48a396800 nid=0x3e03 waiting on condition [0x0000700003e62000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"Service Thread" #9 daemon prio=9 os_prio=31 tid=0x00007fd48915f800 nid=0x3b03 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE

"C1 CompilerThread2" #8 daemon prio=9 os_prio=31 tid=0x00007fd4888c8800 nid=0x4703 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" #7 daemon prio=9 os_prio=31 tid=0x00007fd4888ce800 nid=0x3a03 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #6 daemon prio=9 os_prio=31 tid=0x00007fd48a1c7000 nid=0x4903 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE

"Monitor Ctrl-Break" #5 daemon prio=5 os_prio=31 tid=0x00007fd48a1c5000 nid=0x3703 runnable [0x0000700003850000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:171) at java.net.SocketInputStream.read(SocketInputStream.java:141) at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284) at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326) at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)

"Signal Dispatcher" #4 daemon prio=9 os_prio=31 tid=0x00007fd488821000 nid=0x4a07 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE

"Finalizer" #3 daemon prio=8 os_prio=31 tid=0x00007fd488820000 nid=0x5203 in Object.wait() [0x00007000035c7000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)

"Reference Handler" #2 daemon prio=10 os_prio=31 tid=0x00007fd48881f000 nid=0x5303 in Object.wait() [0x00007000034c4000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at java.lang.ref.Reference.tryHandlePending(Reference.java:191)

"VM Thread" os_prio=31 tid=0x00007fd488818800 nid=0x5403 runnable

"GC task thread#0 (ParallelGC)" os_prio=31 tid=0x00007fd48a004800 nid=0x1c07 runnable

"GC task thread#1 (ParallelGC)" os_prio=31 tid=0x00007fd48a005000 nid=0x1e03 runnable

"GC task thread#2 (ParallelGC)" os_prio=31 tid=0x00007fd48a005800 nid=0x2a03 runnable

"GC task thread#3 (ParallelGC)" os_prio=31 tid=0x00007fd48a006000 nid=0x2c03 runnable

"VM Periodic Task Thread" os_prio=31 tid=0x00007fd48a048000 nid=0x3c03 waiting on condition

JNI global references: 298

Heap PSYoungGen total 252416K, used 65005K [0x0000000795580000, 0x00000007be480000, 0x00000007c0000000) eden space 251904K, 25% used [0x0000000795580000,0x00000007994d3588,0x00000007a4b80000) from space 512K, 31% used [0x00000007be400000,0x00000007be428000,0x00000007be480000) to space 512K, 0% used [0x00000007be380000,0x00000007be380000,0x00000007be400000) ParOldGen total 67584K, used 22919K [0x0000000740000000, 0x0000000744200000, 0x0000000795580000) object space 67584K, 33% used [0x0000000740000000,0x0000000741661e28,0x0000000744200000) Metaspace used 25953K, capacity 26312K, committed 26672K, reserved 1073152K class space used 2892K, capacity 3016K, committed 3120K, reserved 1048576K`

brettwooldridge commented 5 years ago

I don’t see anything wrong (other than, man, that is a lot of Kafka threads).

The housekeeper thread, which runs via a ScheduledExecutor, seems appropriately parked in a timed wait.

No clue as to why debug logging would not show housekeeping activity. Again, I recommend setting a breakpoint in the pool and stepping into the logger method to understand why logging is not occurring. Possibly something is altering the logging level? Like it or not this is going to require some debugging in your part.