Consumer fails due to WakeupExceptions when there are no messages for a while

[kciesielski]

As reported by @breadfan "(...) if there are not any new messages for some period of time we get a "[ERROR] from akka.kafka.KafkaConsumerActor in search-classified-cats-enhancer-akka.actor.default-dispatcher-5 - WakeupException limit exceeded, stopping."

A consumer should not receive WakeupExceptions when there are no messages.

[patriknw]

ouch, Isn't that strange? Didn't we cancel the wakeup task? poll should not block that long when there are no messages

[kciesielski]

I don't think that poll blocks. Calling poll() should keep the internal heartbeat work. Even if there are no messages, it should stay connected without any WakeupExceptions.

[l15k4]

I fixed this by setting consumer's session.timeout.ms and broker's group.max.session.timeout.ms to value that is higher than eventual absence of messages. And it stopped happening as only poll pings broker that would otherwise timeout the session.

I have a similar problem though, if consumer dies on some Fatal exception then Consumer doesn't (can't) close the connection and it can never reconnect due to : org.apache.kafka.common.errors.WakeupException when it tries to poll.

I don't know how to deal with this without stopping Kafka which is a bummer. I even added a jvm shutdown hook to close the connection in such case, but it is usually not invoked on OOM exceptions for instance. Restart of the microservice docker container then ends up with WakeupException...

I tried 0.12 with https://github.com/akka/reactive-kafka/pull/220 but instead of the exception I'm getting :

WARN] [09/28/2016 22:07:23.983] [AkkaSystem-akka.kafka.default-dispatcher-8] [akka://AkkaSystem/system/kafka-consumer-1] Consumer interrupted with WakeupException after timeout. Message: null. Current value of akka.kafka.consumer.wakeup-timeout is 3000 milliseconds

[kciesielski]

@l15k4 I'm trying to reproduce the problem with errors when there are no messages for a while. What timeout values did you have when you encountered the problem? How log was Kafka running without any messages in this topic?

[l15k4]

@kciesielski well, my producer yields a very dense stream of messages until it catches up with some other component, then it pushes only one message per hour and I fixed the problem by setting :

session.timeout.ms: 7300000 ( little more than 2 hours in case one message got lost) group.max.session.timeout.ms: 7301000 (must be higher then session.timeout.ms)

And it fixed the problem. Originally I had the default values which is 30 seconds only. Basically these settings must be always higher than the biggest potential time gap between consumer polls.

I think it really works like that as it fixed the problem. I didn't reproduce it though, I fixed this directly in my staging environment.

[kciesielski]

@l15k4 Thanks for the details. Our internal actor calls poll() in a loop so there should be no need to set large values of session.timeout.ms. I'll try to wait more than one hour with default settings to see if I can reproduce.

[l15k4]

@kciesielski I forgot to mention that I'm using committableSource and the jobs took more than those 30 seconds, some jobs got stuck even for more than an hour before I optimized everything. This is the code :

      Consumer
        .committableSource(consumerSettings(consumerClientId, topicConsumerGroup), Subscriptions.topics(topicName))
        .map( msg => msg.committableOffset -> deserialize(msg.record.value) )
        .mapAsync(1) { case (offset, msg)  =>
            ref.ask(msg)(2.hours).map( _ => offset.commitScaladsl())
        }.to(Sink.ignore).run()

That's probably the reason behind the actor not calling poll as you mentioned. In the end I had to lower granularity of the data-source partitioning to avoid having jobs that take minutes but rather max. 10 seconds

[kciesielski]

@l15k4 Thanks for even more details :) Long processing shouldn't affect the internal poll loop. It's done asynchronously by an actor which self-polls regardless of demand (only with "empty assignment"). I just checked with a test of processing longer than default session.timeout.ms of 30s and it's OK.

[l15k4]

I'm also getting WakeupExceptions when I stop the consumer by calling control.shutdown() and then start it again ... it then warns :

Consumer interrupted with WakeupException after timeout. Message: null.

and is unable to consume new messages ... however control.shutdown() is called from a sys.addShutdownHook() so I'm not 100% sure it gets called.

[kciesielski]

@l15k4 When you close your consumer and try to re-connect, you will receive WakeupExceptions for a while, but after some retries it should eventually connect. I'm not sure, we call consumer.close() to disconnect. Maybe Kafka itself needs a while to rebalance after disconnecting a consumer and allow a new one from the same host to re-connect. However, we may need to look deeper into that to verify if it's on our side.

[timothyakampa]

When I run my application in a docker container I get same issue (akka.kafka.KafkaConsumerActor in search-classified-cats-enhancer-akka.actor.default-dispatcher-5 - WakeupException limit exceeded, stopping.). And when I run it outside the docker container it works fine. I have also tried to run my docker container in host network mode but that is not helping. I also tried to check if kafka is reachable from within the docker container and it seems reachable.

[dialtahi]

Using different client and kafka server versions, consumer throws the exception mentioned

[aisven]

The following is somewhat artificial, but it can hopefully be a hint in terms of a report.

For study purposes I wrote an integration test case that uses just one broker running together with one zookeeper. Generally the setup is working. One particular test case is straight forward yet flaky, which is why I am investigating. I experience the WakeupException sometimes.

The test case is based on a single auto-created topic with only partition 0, however without explicit consumer to partition assignment. All consumers involved use the same groupId.

The scenario goes as follows, each step starting after the previous step has completed and then some sleep time passed on the test thread:

1. produce 10 messages, wait a bit and then assert that the CompletitionStage completed successfully
2. consume all messages in the topic, committing the respective offset while consuming each message, and then shutdown consumer
3. produce 10 more messages , wait a bit and then assert that the CompletitionStage completed successfully
4. consume 5 messages by using some flow.take(5), committing the respective offset while consuming each message, then shutdown consumer, and then assert that the 5 expected messages have been consumed here
5. consume all outstanding messages, committing the respective offset while consuming each message, then shutdown consumer, and then assert that the remaining 5 expected messages have been consumed here

Sometimes the test is green, but at least 50% of the times it is red, failing in the assertion of step 4 with 0 messages having been consumed.

Note that I keep Kafka and Zookeeper running, but I also tried to docker-compose down -v them and docker-compose up -d them again with no initial content. Usually this leads to the first few test runs after this re-setup of the Kafka broker to be green.

I already tried several technical and timing-related broker, producer and consumer settings, without noticeable improvement. I also tried adding more sleeps on the test thread between the steps. I may soon be able to share the test code.

[aisven]

Note that I am using Kafka Broker and Client version 0.10.0.1 and akka-stream-kafka_2.12:0.13

[13h3r]

@sourcekick could you please post the test somewhere?

[ogirardot]

reproduced this "bug" when re-creating the example from the documentation :

import akka.Done
import akka.actor.ActorSystem
import akka.kafka.scaladsl.Consumer
import akka.kafka.{ConsumerSettings, Subscriptions}
import akka.stream.ActorMaterializer
import akka.stream.scaladsl.Sink
import org.apache.kafka.clients.consumer.{ConsumerConfig, ConsumerRecord}
import org.apache.kafka.common.serialization.{ByteArrayDeserializer, StringDeserializer}

import scala.concurrent.Future

object Main {

  def main(args: Array[String]): Unit = {
    implicit val system = ActorSystem.create("toto")
    implicit val materializer = ActorMaterializer()

    val consumerSettings = ConsumerSettings(system, new ByteArrayDeserializer, new StringDeserializer)
      .withBootstrapServers("localhost:9092")
      .withGroupId("group2")
      .withProperty(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest")

    val db = new DB
    import scala.concurrent.ExecutionContext.Implicits.global

    Consumer.committableSource(consumerSettings, Subscriptions.topics("topic1"))
      .mapAsync(1) { msg =>
        db.update(msg.record.value).map(_ => msg)
      }
      .mapAsync(1) { msg =>
        msg.committableOffset.commitScaladsl()
      }
      .runWith(Sink.ignore)

  }
}

class DB {

  private val offset = new AtomicLong

  def save(record: ConsumerRecord[Array[Byte], String]): Future[Done] = {
    println(s"DB.save: ${record.value}")
    offset.set(record.offset)
    Future.successful(Done)
  }

  def loadOffset(): Future[Long] =
    Future.successful(offset.get)

  def update(data: String): Future[Done] = {
    println(s"DB.update: $data")
    Future.successful(Done)
  }
}

With this application.conf

# Properties for akka.kafka.ConsumerSettings can be
# defined in this section or a configuration section with
# the same layout.
akka.kafka.consumer {
  # Tuning property of scheduled polls.
  poll-interval = 50ms

  # Tuning property of the `KafkaConsumer.poll` parameter.
  # Note that non-zero value means that blocking of the thread that
  # is executing the stage will be blocked.
  poll-timeout = 50ms

  # The stage will be await outstanding offset commit requests before
  # shutting down, but if that takes longer than this timeout it will
  # stop forcefully.
  stop-timeout = 30s

  # How long to wait for `KafkaConsumer.close`
  close-timeout = 20s

  # If offset commit requests are not completed within this timeout
  # the returned Future is completed `TimeoutException`.
  commit-timeout = 15s

  # If the KafkaConsumer can't connect to the broker the poll will be
  # aborted after this timeout. The KafkaConsumerActor will throw
  # org.apache.kafka.common.errors.WakeupException which will be ignored
  # until max-wakeups limit gets exceeded.
  wakeup-timeout = 3s

  # After exceeding maxinum wakeups the consumer will stop and the stage will fail.
  max-wakeups = 10

  # Fully qualified config path which holds the dispatcher configuration
  # to be used by the KafkaConsumerActor. Some blocking may occur.
  use-dispatcher = "akka.kafka.default-dispatcher"

  # Properties defined by org.apache.kafka.clients.consumer.ConsumerConfig
  # can be defined in this configuration section.
  kafka-clients {
    # Disable auto-commit by default
    enable.auto.commit = false
  }
}

And a confluent 3.0.1 "freshly" downloaded tar.gz

[ogirardot]

To be more precise, "first" consumption is ok, but then if I restart, with no new messages (but still in earliest mode) then the exception becomes :

[WARN] [01/16/2017 19:38:21.646] [toto-akka.kafka.default-dispatcher-7] [akka://toto/system/kafka-consumer-1] Consumer interrupted with WakeupException after timeout. Message: null. Current value of akka.kafka.consumer.wakeup-timeout is 3000 milliseconds

Most of the time though it ends up being ok and the max number of wake up is not reached

[kciesielski]

@ogirardot looks like Kafka needs some time to recover after rebalance and allow re-connecting. Your example looks like it's trying for 30 seconds (10 * 3s) and still failing, am I correct?

[ademartini]

Is there a way to allow for unlimited WakeupExceptions? If kafka goes down for a while for some reason, I don't want to have to restart my consumer application. I would prefer that it simply carry on when kafka comes back up.

[patriknw]

@ademartini Increasing the max-wakeups? https://github.com/akka/reactive-kafka/blob/master/core/src/main/resources/reference.conf#L51-L58

[dbcfd]

For topics with sporadic messages (especially where you may never know when a message will arrive), set fetch-min-bytes to 0, timeout to some acceptable value (e.g. 1 second) that is less than heartbeat, and interval to some acceptable value (e.g. 1 second) that is less than heartbeat.

This will cause polls to return and be repeated in a timely manner without busy waiting the poll.

[mradityagoyal]

@dbcfd can you elaborate on your comment please. We have a sync process. We are trying to use reactive kafka to process messages. Our messages are sporadic and there might be cases when we have no messages for one or two days. Currently, our consumer stream dies after about 5 minutes of no messages on the topic with below error `` [ERROR] [03/03/2017 14:59:25.058] [PrimeClusterSystem-akka.actor.default-dispatcher-23] [akka://PrimeClusterSystem/system/kafka-consumer-1] null org.apache.kafka.common.errors.WakeupException at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.maybeTriggerWakeup(ConsumerNetworkClient.java:367) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.executeDelayedTasks(ConsumerNetworkClient.java:251) at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:988) at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:938) at akka.kafka.KafkaConsumerActor.poll(KafkaConsumerActor.scala:211) at akka.kafka.KafkaConsumerActor$$anonfun$receive$1.applyOrElse(KafkaConsumerActor.scala:113) at akka.actor.Actor$class.aroundReceive(Actor.scala:484) at akka.kafka.KafkaConsumerActor.aroundReceive(KafkaConsumerActor.scala:66) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526) at akka.actor.ActorCell.invoke(ActorCell.scala:495) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) at akka.dispatch.Mailbox.run(Mailbox.scala:224) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

{code}

[dbcfd]

There are a few cases where wakeupexception will occur:

• Your broker is down (guessing this is not the case)
• You are attempting to pull more data back in a poll than can be returned in your wakeup timeout (maybe, would need to know how big your messages are)
• Your settings are such that the poll for no data is not returning within the wakeup timeout

I'm guessing the you are being affected by no data within the wakeup timeout. A simple fix for this with very sporadic messages is to set fetch min bytes to 0 and increase your poll interval. Poll will return nearly immediately, but then wait to poll.

Another way is to set fetch min bytes to 1, and set your fetch max wait and poll timeout to something less than wakeup timeout, e.g. 500ms for fetch max wait, 1s for poll timeout, 3s for wakeup timeout.

[mradityagoyal]

@dbcfd Thank you very much for the response. Yes.. we hare facing the third issue .. no data within wakeup timeout. Our current fetch.min.bytes setting is the default 1. We will set that to 0. the current max.poll.interval.ms is at default 300000. Do you recommend increasing it further?

Out of the two approaches, which one is more suitable / fault safe?

[mradityagoyal]

@dbcfd never mind.. i figured out you were talking about the settings in reference.conf. Thanks. We are testing both approaches.

[dbcfd]

Yeah start with reference conf settings. The 0.10 kafka consumer settings seem to work for most applications.

[Jayasagar]

I could reproduce this issue using akka-stream-kafka.0.15 and Kafka 0.10.0.

We get this issue in production (Cloudera cluster), once in a while one of our nodes goes down in a cluster then consumer fails to recover and consume messages.

Locally, I tried to reproduce scenario and it is reproduced:

1. Start the broker (I have 1 node)
2. Produce an event
3. Consume an event(Works)
4. Kill the kafka-server (Reason could be anything)
5. After 30 seconds, start the kafka-server
6. Produce an event
7. Consume an event

At 7th step, the consumer fails to consume the message, we have restart the service to bring it back?

Apart from increase the max-wakeups value, Is there any other solution? Is anyone else experiencing this problem?

[dbcfd]

Jayasagar your issue is more of a kafka operational issue. Your kafka cluster should be replicated, and your controller not overloaded (topics/partitions not to exceed recommended), such that leader elections will occur in a timely fashion, and a new group coordinator is available such that consumption can resume.

If you violate that, you will need to increase max-wakeups to allow consume to properly recover. Alternatively, you can watch for consume control to fail, fail your stream, and restart it with materializer supervision or with upstart (or similar service) restart.

[patriknw]

Thanks for investigations. I would be interested in suggestions of what the default values should be to reduce trouble.

I agree that in the end you need to watch the stream for termination and start it again (after a backoff). However, good if we can survive "normal" connectivity issues.

[patriknw]

BTW, there is a PR in progress for documenting error handling. https://github.com/akka/reactive-kafka/pull/296 Please chime in there if you have suggestions.

[Jayasagar]

@dbcfd I don't think we have issues with Kafka configuration, we have defaults and we have not exceeded any recommendations! The only problem we see is sometimes one of the node in cluster goes down. And I agree, watch for consumer actor context stop and start it again might help. I will try.

@patriknw Thanks for the suggestions! I agree to stick with default values and watch for the termination and start again. Thanks for pointing me to #296

[jroper]

We were seeing these sorts of errors in our tests in Lagom. I am not 100% sure of what the problem was, but a summary of what I identified is here.

One thing to note here, it seems that if you use the same groupId for multiple different topics with different consumer instances, that these can interfere with each other, it appears that there is a race condition which can be triggered, for example, by rebalancing due to a new consumer of a different topic with the same group id being added, and in this situation a consumer can end up consuming no partitions (even though it might be the only consumer for that group id for that topic), and in that case, the wakeup exception will always be encountered.

Also something maybe to check, it may be the case (and it's the evening, I'm about to go home and don't have time to confirm this, I may have completely read the code wrong so don't take my word for this) that if a consumer ends up with no partitions assigned to it (for whatever reason, maybe you have 11 consumers and only 10 partitions), that the invocation to poll will never return, which will cause the wakeup exceptions.

[jroper]

Oh and by the way, tl;dr for the above post, try not reusing the same group id across topics, rather try setting a unique group id per topic, that fixed the problems we were seeing.

[ignasi35]

Oh and by the way, tl;dr for the above post, try not reusing the same group id across topics, rather try setting a unique group id per topic, that fixed the problems we were seeing.

@jroper can you confirm that comment still stands? IIUC the execution we observed after that change also failed so the comment is invalid.

[mallize]

Edit: After testing a bit more, the config below did resolve this issue for me.

Is anyone actively looking at this? It seems there are a lot of work arounds in this thread. @jroper I get this error when I'm running with only one topic and a brand new group Id.

`

def start(start: Start) = {

val consumerSettings = ConsumerSettings(context.system, new StringDeserializer, new StringDeserializer)
  .withBootstrapServers(start.brokers)
  .withGroupId("testGroupId")
  .withProperty(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest")

val (control, future) = Consumer.committablePartitionedSource(consumerSettings, Subscriptions.topics("StreamTest"))
  .map {
    case (topicPartition, source) =>
      debug(s"Consuming from topic partition: ${topicPartition.topic()}.${topicPartition.partition()}")
      source.mapAsync(start.partitions * start.topics.length) { msg =>
        debug(s"received....  ${msg.record.value()}, offset = ${msg.record.offset()}, partition ${msg.record.partition()}")
        handleKafkaMessage(msg.record.key(), msg.record.topic(), msg.record.value())
        Future.successful(msg.committableOffset)
      }
  }
  .flatMapMerge(start.partitions * start.topics.length, identity)
  .batch(max = start.batchSize, first => CommittableOffsetBatch.empty.updated(first)) { (batch, elem) => batch.updated(elem) }
  .mapAsync(start.partitions * start.topics.length)(_.commitScaladsl())
  .toMat(Sink.ignore)(Keep.both)
  .run()

future.onFailure {
  case ex =>
    Logger.error("Stream failed due to error. Shutting down.", ex)
    self ! StopStream
}

context become whenRunning(control)

}

`

'

akka.kafka.consumer {

  # Tuning property of scheduled polls.
  //poll-interval = 50ms

  # Tuning property of the `KafkaConsumer.poll` parameter.
  # Note that non-zero value means that blocking of the thread that
  # is executing the stage will be blocked.
  //poll-timeout = 50ms

  // This configures how long the consumer will keep tring to connect. Instead of making this number very large we use
  // a smaller number so that an error message is emitted indicating there is a problem with the connection. The
  // consumer is restarted automatically after stopping and emitting the error.
  max-wakeups = 5

  // when deployed in aws, with the default 3s wakeup-timeout there are periods of time where we get repeated WakeupExceptions
  // which eventually trigger a failure and restart of the stream. This causes scenarios where a message is delivered
  // more than once since we aren't using an at-most-once consumer.
  wakeup-timeout = 6s

  kafka-clients {
    auto.offset.reset: "earliest"
    max.poll.interval.ms: 20000 // default 30000
    fetch.min.bytes: 0
    fetch.max.wait.ms: 10000

    // Trying to avoid this error by reducing max poll records from the default...
    // org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed since the group has already
    // rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to
    // poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is
    // spending too much time message processing. You can address this either by increasing the session timeout or by
    // reducing the maximum size of batches returned in poll() with max.poll.records.
    max.poll.records: 100 // default = 500 (old 2147483647)
  }
}'

[jpeel]

@mallize One thing that stands out to me from your config is that your fetch.max.wait.ms is 10 seconds, but your wakeup timeout is 6s. If there isn't any data, the poll could wait 10 seconds during which a wake up exception will be thrown. I suppose fetch.min.bytes of 0 might prevent this, but I would recommend that the fetch.max.wait.ms should be less than the wakeup timeout.

[schrepfler]

Is there any investigation going on or future mitigation around this issue?

[sbrnunes]

Are there any updates regarding this issue? I believe we're facing some one of the problems described in this thread.

In our case, our partitions are not very well balanced. We have some partitions with a lot of data, while others may not have any data for a while. I suspect that, when our application starts, Consumers assigned to partitions for which there is no data during the Wakeup window get kicked-out from the group.

Does this make sense?

Another strange thing I realised was: even though we are handling the Future returned by the RunnableGraph, and forcing the stream to restart when that Future completes with a Failure, when this WakeupException limit exceeded, stopping errors occur, I never see the stream completing with that error. Does anyone know why?

[chbatey]

Having taken look through the KafkaConsumer code there are many places where KafkaConsumer#poll will block for a long time, ignoring the timeout we pass into poll e.g.

• When it can’t work out the coordinator (Integer.MAX_VALUE)
• Waiting for a metadata update (Integer.MAX_VALUE)
• When doing a poll to join the group (5 seconds)

To help aid in debugging which of the cases it is (all indicate a problem with Kafka rather than reactive-kafka) I've added some extra logging when a WakeupException happens, it will gather thread stacks so we can see what the consumer is up to.

In the meantime I'd suggest increasing the timeout and also increasing the number of threads for the Kafka dispatcher as AFAICT there is no way to avoid the kafka client gobbling up a thread for long durations. It appears to be an API that prefers a dedicated thread rather than a async thread per core model.

I also had the briefest of looks at the Kafka streams code to see what they do and they use the same API so will block a thread also.

[sbrnunes]

We're still being affected by this issues. More recently, a simple rolling restart of the cluster led to several consumer groups, across multiple services using this library, to stop consuming.

In a particular service, we did increase the wakeup-timeout and max-wakeups, but it didn't help. We do see an increase in the number of retries, but that's it. Then it just stops, after exceeding all those retries.

We also added a stream supervisor, as suggested in: https://doc.akka.io/docs/akka-stream-kafka/current/errorhandling.html. Every now and then, we see it kicking off following a stream completion with a Failure, meaning, the supervision does work, providing the RunnableGraph completes with a Failure. The problem is when it doesn't complete at all, which is what is happening most of the times.

Looking at this codebase, here's a couple of questions:

• What will happen if an exception occurs inside the thread where the wakeup task is running: https://github.com/akka/reactive-kafka/blob/master/core/src/main/scala/akka/kafka/KafkaConsumerActor.scala#L276 ?
• What will be the effect of an exception in the above thread to the consumer polling that will be happening in: https://github.com/akka/reactive-kafka/blob/master/core/src/main/scala/akka/kafka/KafkaConsumerActor.scala#L288 ?
• Is it possible that we're not stopping the consumer actor in the occurrence of an exception in one of the above places?

[chbatey]

Hi @sbrnunes what version are you running with? If it is 0.19 would you mind sharing your logs of it happening?

[chbatey]

Looking at the Kafka client code https://github.com/akka/reactive-kafka/blob/master/core/src/main/scala/akka/kafka/KafkaConsumerActor.scala#L276 is very unlikely to throw. It sets an atomic boolean as well as interrupts the selector.

If the wakeup is isn't set due to an exception then it would result in the poll blocking as long as kafka consumer wanted.

The actor is stopped for any NonFatal exception for the poll.

[sbrnunes]

Hi @chbatey , we're currently on v0.17.

We could try to upgrade to 0.19 and try to get those logs while testing some of the scenarios above (cluster restart, bringing down and up random nodes, etc.). I believe there's no major changes to the library between 0.17 and 0.19...

[sbrnunes]

Hi @chbatey ,

Sorry for the long post.

We upgraded to 0.19 and reproduced the exact same issue in our Master environment. Basically, we have shutdown the entire Kafka cluster (by stopping the processes in each of the nodes) while the applications were running, and this is what we saw in the logs (pasting only the relevant messages).

Maybe worth mentioning, this is our current configuration:

akka.kafka.consumer {
  wakeup-timeout = 6s
  max-wakeups = 20

  kafka-clients {
    heartbeat.interval.ms = 3000
    session.timeout.ms = 120000
    request.timeout.ms = 160000
    auto.commit.interval.ms = 5000
    max.partition.fetch.bytes = 786432
    enable.auto.commit = true
    auto.offset.reset = latest
  }
}

1. A lot of Consumer interrupted with WakeupException after timeout. Message: null. Current value of akka.kafka.consumer.wakeup-timeout is 6000 milliseconds., starting at 2018-02-12T14:14:03.945+00:00

   Click me for the full log

   ``` 2018-02-12T14:14:03.945+00:00 akka.kafka.KafkaConsumerActor WARN Consumer interrupted with WakeupException after timeout. Message: null. Current value of akka.kafka.consumer.wakeup-timeout is 6000 milliseconds. Exception: org.apache.kafka.common.errors.WakeupException at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.maybeTriggerWakeup(ConsumerNetworkClient.java:422) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:245) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:208) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:199) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.awaitMetadataUpdate(ConsumerNetworkClient.java:134) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:226) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:203) at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:286) at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1078) at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1043) at akka.kafka.KafkaConsumerActor.tryPoll$1(KafkaConsumerActor.scala:287) at akka.kafka.KafkaConsumerActor.poll(KafkaConsumerActor.scala:349) at akka.kafka.KafkaConsumerActor.akka$kafka$KafkaConsumerActor$$receivePoll(KafkaConsumerActor.scala:256) at akka.kafka.KafkaConsumerActor$$anonfun$receive$1.applyOrElse(KafkaConsumerActor.scala:164) at akka.actor.Actor$class.aroundReceive(Actor.scala:517) at akka.kafka.KafkaConsumerActor.aroundReceive(KafkaConsumerActor.scala:79) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:527) at akka.actor.ActorCell.invoke_aroundBody0(ActorCell.scala:496) at akka.actor.ActorCell$AjcClosure1.run(ActorCell.scala:1) at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) at akka.kamon.instrumentation.ActorMonitors$$anon$1$$anonfun$processMessage$1.apply(ActorMonitor.scala:59) at kamon.trace.Tracer$.withContext(TracerModule.scala:58) at akka.kamon.instrumentation.ActorMonitors$$anon$1.processMessage(ActorMonitor.scala:58) at akka.kamon.instrumentation.ActorCellInstrumentation.aroundBehaviourInvoke(ActorInstrumentation.scala:44) at akka.actor.ActorCell.invoke(ActorCell.scala:489) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) at akka.dispatch.Mailbox.run(Mailbox.scala:224) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) ```

2. A lot of akka.kafka.KafkaConsumerActor WARN KafkaConsumer poll is taking significantly longer (6000ms) to return from poll then the configured poll interval (50ms) after the message above

3. A few WakeupException limit exceeded, stopping., starting around 2018-02-12T14:16:13.415+00:00 (2 mins later)

4. A lot of akka.kafka.KafkaConsumerActor WARN Wake up has been triggered. Dumping stacks: Thread[AsyncExecutor.default-11,5,main], starting at 2018-02-12T14:16:38.085+00:00

   Click me for the full log

   ``` 2018-02-12T14:18:03.830+00:00 akka.kafka.KafkaConsumerActor WARN KafkaConsumer poll is taking significantly longer (6000ms) to return from poll then the configured poll interval (50ms). Waking up consumer to avoid thread starvation. 2018-02-12T14:18:03.832+00:00 akka.kafka.KafkaConsumerActor WARN Wake up has been triggered. Dumping stacks: Thread[AsyncExecutor.default-11,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) slick.util.ManagedArrayBlockingQueue$$anonfun$take$1.apply(ManagedArrayBlockingQueue.scala:115) slick.util.ManagedArrayBlockingQueue$$anonfun$take$1.apply(ManagedArrayBlockingQueue.scala:114) slick.util.ManagedArrayBlockingQueue.lockedInterruptibly(ManagedArrayBlockingQueue.scala:222) slick.util.ManagedArrayBlockingQueue.take(ManagedArrayBlockingQueue.scala:114) slick.util.ManagedArrayBlockingQueue.take(ManagedArrayBlockingQueue.scala:13) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[KafkaProcessors-akka.actor.default-dispatcher-122,5,main] sun.misc.Unsafe.park(Native Method) akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075) akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Thread[KafkaProcessors-akka.kafka.transactions-dispatcher-173,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[cluster1-nio-worker-3,5,main] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) com.datastax.shaded.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) com.datastax.shaded.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) com.datastax.shaded.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) com.datastax.shaded.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131) com.datastax.shaded.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:745) Thread[logback-1,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[AsyncExecutor.default-4,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) slick.util.ManagedArrayBlockingQueue$$anonfun$take$1.apply(ManagedArrayBlockingQueue.scala:115) slick.util.ManagedArrayBlockingQueue$$anonfun$take$1.apply(ManagedArrayBlockingQueue.scala:114) slick.util.ManagedArrayBlockingQueue.lockedInterruptibly(ManagedArrayBlockingQueue.scala:222) slick.util.ManagedArrayBlockingQueue.take(ManagedArrayBlockingQueue.scala:114) slick.util.ManagedArrayBlockingQueue.take(ManagedArrayBlockingQueue.scala:13) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[cluster1-timeouter-0,5,main] java.lang.Thread.sleep(Native Method) com.datastax.shaded.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:560) com.datastax.shaded.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:459) com.datastax.shaded.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:745) Thread[KafkaProcessors-akka.actor.default-dispatcher-121,5,main] java.util.regex.Pattern$BmpCharProperty.match(Pattern.java:3797) java.util.regex.Pattern$Curly.match0(Pattern.java:4250) java.util.regex.Pattern$Curly.match(Pattern.java:4234) java.util.regex.Pattern$GroupHead.match(Pattern.java:4658) java.util.regex.Pattern$Branch.match(Pattern.java:4604) java.util.regex.Pattern$Branch.match(Pattern.java:4602) java.util.regex.Pattern$BmpCharProperty.match(Pattern.java:3798) java.util.regex.Pattern$Start.match(Pattern.java:3461) java.util.regex.Matcher.search(Matcher.java:1248) java.util.regex.Matcher.find(Matcher.java:664) java.util.Formatter.parse(Formatter.java:2549) java.util.Formatter.format(Formatter.java:2501) java.util.Formatter.format(Formatter.java:2455) java.lang.String.format(String.java:2940) scala.collection.immutable.StringLike$class.format(StringLike.scala:318) scala.collection.immutable.StringOps.format(StringOps.scala:29) akka.util.Helpers$.currentTimeMillisToUTCString(Helpers.scala:64) akka.event.slf4j.Slf4jLogger.formatTimestamp(Slf4jLogger.scala:124) akka.event.slf4j.Slf4jLogger.withMdc_aroundBody0(Slf4jLogger.scala:100) akka.event.slf4j.Slf4jLogger$AjcClosure1.run(Slf4jLogger.scala:1) org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) akka.kamon.instrumentation.ActorLoggingInstrumentation$$anonfun$aroundWithMdcInvocation$1$$anonfun$apply$1.apply(ActorLoggingInstrumentation.scala:46) kamon.trace.logging.MdcKeysSupport$class.withMdc(MdcKeysSupport.scala:33) akka.kamon.instrumentation.ActorLoggingInstrumentation.withMdc(ActorLoggingInstrumentation.scala:25) akka.kamon.instrumentation.ActorLoggingInstrumentation$$anonfun$aroundWithMdcInvocation$1.apply(ActorLoggingInstrumentation.scala:45) kamon.trace.Tracer$.withContext(TracerModule.scala:58) akka.kamon.instrumentation.ActorLoggingInstrumentation.aroundWithMdcInvocation(ActorLoggingInstrumentation.scala:44) akka.event.slf4j.Slf4jLogger.withMdc(Slf4jLogger.scala:98) akka.event.slf4j.Slf4jLogger$$anonfun$receive$1.applyOrElse(Slf4jLogger.scala:77) akka.actor.Actor$class.aroundReceive(Actor.scala:517) akka.event.slf4j.Slf4jLogger.aroundReceive(Slf4jLogger.scala:57) akka.actor.ActorCell.receiveMessage(ActorCell.scala:527) akka.actor.ActorCell.invoke_aroundBody0(ActorCell.scala:496) akka.actor.ActorCell$AjcClosure1.run(ActorCell.scala:1) org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) akka.kamon.instrumentation.ActorMonitors$$anon$1$$anonfun$processMessage$1.apply(ActorMonitor.scala:59) kamon.trace.Tracer$.withContext(TracerModule.scala:58) akka.kamon.instrumentation.ActorMonitors$$anon$1.processMessage(ActorMonitor.scala:58) akka.kamon.instrumentation.ActorCellInstrumentation.aroundBehaviourInvoke(ActorInstrumentation.scala:44) akka.actor.ActorCell.invoke(ActorCell.scala:489) akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) akka.dispatch.Mailbox.run(Mailbox.scala:224) akka.dispatch.Mailbox.exec(Mailbox.scala:234) akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Thread[AsyncExecutor.default-1,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) slick.util.ManagedArrayBlockingQueue$$anonfun$take$1.apply(ManagedArrayBlockingQueue.scala:115) slick.util.ManagedArrayBlockingQueue$$anonfun$take$1.apply(ManagedArrayBlockingQueue.scala:114) slick.util.ManagedArrayBlockingQueue.lockedInterruptibly(ManagedArrayBlockingQueue.scala:222) slick.util.ManagedArrayBlockingQueue.take(ManagedArrayBlockingQueue.scala:114) slick.util.ManagedArrayBlockingQueue.take(ManagedArrayBlockingQueue.scala:13) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[kamon-akka.io.pinned-dispatcher-6,5,main] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) akka.io.SelectionHandler$ChannelRegistryImpl$$anon$3.tryRun(SelectionHandler.scala:128) akka.io.SelectionHandler$ChannelRegistryImpl$Task.run(SelectionHandler.scala:246) akka.io.SelectionHandler$ChannelRegistryImpl$$anon$3.run(SelectionHandler.scala:161) akka.util.SerializedSuspendableExecutionContext.run$1(SerializedSuspendableExecutionContext.scala:67) akka.util.SerializedSuspendableExecutionContext.run(SerializedSuspendableExecutionContext.scala:71) akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[KafkaProcessors-akka.kafka.transactions-dispatcher-164,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[HikariPool-1 housekeeper,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[KafkaProcessors-akka.kafka.transactions-dispatcher-174,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[AsyncExecutor.default-7,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) slick.util.ManagedArrayBlockingQueue$$anonfun$take$1.apply(ManagedArrayBlockingQueue.scala:115) slick.util.ManagedArrayBlockingQueue$$anonfun$take$1.apply(ManagedArrayBlockingQueue.scala:114) slick.util.ManagedArrayBlockingQueue.lockedInterruptibly(ManagedArrayBlockingQueue.scala:222) slick.util.ManagedArrayBlockingQueue.take(ManagedArrayBlockingQueue.scala:114) slick.util.ManagedArrayBlockingQueue.take(ManagedArrayBlockingQueue.scala:13) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[AsyncExecutor.default-2,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) slick.util.ManagedArrayBlockingQueue$$anonfun$take$1.apply(ManagedArrayBlockingQueue.scala:115) slick.util.ManagedArrayBlockingQueue$$anonfun$take$1.apply(ManagedArrayBlockingQueue.scala:114) slick.util.ManagedArrayBlockingQueue.lockedInterruptibly(ManagedArrayBlockingQueue.scala:222) slick.util.ManagedArrayBlockingQueue.take(ManagedArrayBlockingQueue.scala:114) slick.util.ManagedArrayBlockingQueue.take(ManagedArrayBlockingQueue.scala:13) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[Hashed wheel timer #3,5,main] java.lang.Thread.sleep(Native Method) org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:445) org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:364) org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) java.lang.Thread.run(Thread.java:745) Thread[kamon-akka.actor.default-dispatcher-4,5,main] sun.misc.Unsafe.park(Native Method) akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075) akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Thread[KafkaProcessors-akka.kafka.transactions-dispatcher-176,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[AsyncExecutor.default-10,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) slick.util.ManagedArrayBlockingQueue$$anonfun$take$1.apply(ManagedArrayBlockingQueue.scala:115) slick.util.ManagedArrayBlockingQueue$$anonfun$take$1.apply(ManagedArrayBlockingQueue.scala:114) slick.util.ManagedArrayBlockingQueue.lockedInterruptibly(ManagedArrayBlockingQueue.scala:222) slick.util.ManagedArrayBlockingQueue.take(ManagedArrayBlockingQueue.scala:114) slick.util.ManagedArrayBlockingQueue.take(ManagedArrayBlockingQueue.scala:13) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[AsyncExecutor.default-13,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) slick.util.ManagedArrayBlockingQueue$$anonfun$take$1.apply(ManagedArrayBlockingQueue.scala:115) slick.util.ManagedArrayBlockingQueue$$anonfun$take$1.apply(ManagedArrayBlockingQueue.scala:114) slick.util.ManagedArrayBlockingQueue.lockedInterruptibly(ManagedArrayBlockingQueue.scala:222) slick.util.ManagedArrayBlockingQueue.take(ManagedArrayBlockingQueue.scala:114) slick.util.ManagedArrayBlockingQueue.take(ManagedArrayBlockingQueue.scala:13) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[KafkaProcessors-akka.actor.default-dispatcher-110,5,main] sun.misc.Unsafe.park(Native Method) akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075) akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Thread[KafkaProcessors-akka.kafka.transactions-dispatcher-171,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[KafkaProcessors-akka.actor.notifications-dispatcher-13,5,main] sun.misc.Unsafe.park(Native Method) akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075) akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Thread[New I/O worker #3,5,main] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434) org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[KafkaProcessors-akka.actor.notifications-dispatcher-21,5,main] sun.misc.Unsafe.park(Native Method) akka.dispatch.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135) akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067) akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Thread[cluster1-nio-worker-0,5,main] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) com.datastax.shaded.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) com.datastax.shaded.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) com.datastax.shaded.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) com.datastax.shaded.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131) com.datastax.shaded.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:745) Thread[AsyncExecutor.default-12,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) slick.util.ManagedArrayBlockingQueue$$anonfun$take$1.apply(ManagedArrayBlockingQueue.scala:115) slick.util.ManagedArrayBlockingQueue$$anonfun$take$1.apply(ManagedArrayBlockingQueue.scala:114) slick.util.ManagedArrayBlockingQueue.lockedInterruptibly(ManagedArrayBlockingQueue.scala:222) slick.util.ManagedArrayBlockingQueue.take(ManagedArrayBlockingQueue.scala:114) slick.util.ManagedArrayBlockingQueue.take(ManagedArrayBlockingQueue.scala:13) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[KafkaProcessors-akka.kafka.transactions-dispatcher-172,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[Finalizer,8,system] java.lang.Object.wait(Native Method) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164) java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209) Thread[KafkaProcessors-akka.kafka.transactions-dispatcher-159,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[DestroyJavaVM,5,main] Thread[KafkaProcessors-akka.actor.default-dispatcher-120,5,main] sun.misc.Unsafe.park(Native Method) akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075) akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Thread[cluster1-connection-reaper-0,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) com.datastax.shaded.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:745) Thread[KafkaProcessors-akka.kafka.transactions-dispatcher-160,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[Signal Dispatcher,9,system] Thread[KafkaProcessors-akka.kafka.transactions-dispatcher-175,5,main] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.kafka.common.network.Selector.select(Selector.java:529) org.apache.kafka.common.network.Selector.poll(Selector.java:321) org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:433) org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:232) org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:208) org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:199) org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.awaitMetadataUpdate(ConsumerNetworkClient.java:134) org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:226) org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:203) org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:286) org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1078) org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1043) akka.kafka.KafkaConsumerActor.tryPoll$1(KafkaConsumerActor.scala:287) akka.kafka.KafkaConsumerActor.poll(KafkaConsumerActor.scala:334) akka.kafka.KafkaConsumerActor.akka$kafka$KafkaConsumerActor$$receivePoll(KafkaConsumerActor.scala:256) akka.kafka.KafkaConsumerActor$$anonfun$receive$1.applyOrElse(KafkaConsumerActor.scala:164) akka.actor.Actor$class.aroundReceive(Actor.scala:517) akka.kafka.KafkaConsumerActor.aroundReceive(KafkaConsumerActor.scala:79) akka.actor.ActorCell.receiveMessage(ActorCell.scala:527) akka.actor.ActorCell.invoke_aroundBody0(ActorCell.scala:496) akka.actor.ActorCell$AjcClosure1.run(ActorCell.scala:1) org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) akka.kamon.instrumentation.ActorMonitors$$anon$1$$anonfun$processMessage$1.apply(ActorMonitor.scala:59) kamon.trace.Tracer$.withContext(TracerModule.scala:58) akka.kamon.instrumentation.ActorMonitors$$anon$1.processMessage(ActorMonitor.scala:58) akka.kamon.instrumentation.ActorCellInstrumentation.aroundBehaviourInvoke(ActorInstrumentation.scala:44) akka.actor.ActorCell.invoke(ActorCell.scala:489) akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) akka.dispatch.Mailbox.run(Mailbox.scala:224) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[RMI TCP Accept-0,5,system] java.net.PlainSocketImpl.socketAccept(Native Method) java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409) java.net.ServerSocket.implAccept(ServerSocket.java:545) java.net.ServerSocket.accept(ServerSocket.java:513) sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:400) sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:372) java.lang.Thread.run(Thread.java:745) Thread[KafkaProcessors-akka.kafka.transactions-dispatcher-170,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[KafkaProcessors-akka.kafka.transactions-dispatcher-169,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[cluster1-nio-worker-2,5,main] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) com.datastax.shaded.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) com.datastax.shaded.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) com.datastax.shaded.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) com.datastax.shaded.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131) com.datastax.shaded.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:745) Thread[New I/O boss #5,5,main] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434) org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42) org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[New I/O worker #1,5,main] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434) org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[kamon-akka.actor.default-dispatcher-5,5,main] sun.misc.Unsafe.park(Native Method) akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075) akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Thread[KafkaProcessors-akka.actor.default-dispatcher-108,5,main] java.lang.Thread.dumpThreads(Native Method) java.lang.Thread.getAllStackTraces(Thread.java:1607) akka.kafka.KafkaConsumerActor$$anonfun$1.apply$mcV$sp(KafkaConsumerActor.scala:272) akka.actor.Scheduler$$anon$4.run(Scheduler.scala:140) akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40) akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:43) akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Thread[AsyncExecutor.default-8,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) slick.util.ManagedArrayBlockingQueue$$anonfun$take$1.apply(ManagedArrayBlockingQueue.scala:115) slick.util.ManagedArrayBlockingQueue$$anonfun$take$1.apply(ManagedArrayBlockingQueue.scala:114) slick.util.ManagedArrayBlockingQueue.lockedInterruptibly(ManagedArrayBlockingQueue.scala:222) slick.util.ManagedArrayBlockingQueue.take(ManagedArrayBlockingQueue.scala:114) slick.util.ManagedArrayBlockingQueue.take(ManagedArrayBlockingQueue.scala:13) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[ForkJoinPool-1-worker-3,5,main] sun.misc.Unsafe.park(Native Method) scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075) scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Thread[Abandoned connection cleanup thread,5,main] java.lang.Object.wait(Native Method) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143) com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:43) Thread[kamon-akka.actor.default-dispatcher-2,5,main] sun.misc.Unsafe.park(Native Method) akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075) akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Thread[KafkaProcessors-scheduler-1,5,main] java.lang.Thread.sleep(Native Method) akka.actor.LightArrayRevolverScheduler.waitNanos(LightArrayRevolverScheduler.scala:85) akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:265) akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:235) java.lang.Thread.run(Thread.java:745) Thread[KafkaProcessors-akka.kafka.transactions-dispatcher-157,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[AsyncExecutor.default-9,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) slick.util.ManagedArrayBlockingQueue$$anonfun$take$1.apply(ManagedArrayBlockingQueue.scala:115) slick.util.ManagedArrayBlockingQueue$$anonfun$take$1.apply(ManagedArrayBlockingQueue.scala:114) slick.util.ManagedArrayBlockingQueue.lockedInterruptibly(ManagedArrayBlockingQueue.scala:222) slick.util.ManagedArrayBlockingQueue.take(ManagedArrayBlockingQueue.scala:114) slick.util.ManagedArrayBlockingQueue.take(ManagedArrayBlockingQueue.scala:13) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[KafkaProcessors-akka.kafka.transactions-dispatcher-165,5,main] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.kafka.common.network.Selector.select(Selector.java:529) org.apache.kafka.common.network.Selector.poll(Selector.java:321) org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:433) org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:232) org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:208) org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:199) org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.awaitMetadataUpdate(ConsumerNetworkClient.java:134) org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:226) org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:203) org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:286) org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1078) org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1043) akka.kafka.KafkaConsumerActor.tryPoll$1(KafkaConsumerActor.scala:287) akka.kafka.KafkaConsumerActor.poll(KafkaConsumerActor.scala:334) akka.kafka.KafkaConsumerActor.akka$kafka$KafkaConsumerActor$$receivePoll(KafkaConsumerActor.scala:256) akka.kafka.KafkaConsumerActor$$anonfun$receive$1.applyOrElse(KafkaConsumerActor.scala:164) akka.actor.Actor$class.aroundReceive(Actor.scala:517) akka.kafka.KafkaConsumerActor.aroundReceive(KafkaConsumerActor.scala:79) akka.actor.ActorCell.receiveMessage(ActorCell.scala:527) akka.actor.ActorCell.invoke_aroundBody0(ActorCell.scala:496) akka.actor.ActorCell$AjcClosure1.run(ActorCell.scala:1) org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) akka.kamon.instrumentation.ActorMonitors$$anon$1$$anonfun$processMessage$1.apply(ActorMonitor.scala:59) kamon.trace.Tracer$.withContext(TracerModule.scala:58) akka.kamon.instrumentation.ActorMonitors$$anon$1.processMessage(ActorMonitor.scala:58) akka.kamon.instrumentation.ActorCellInstrumentation.aroundBehaviourInvoke(ActorInstrumentation.scala:44) akka.actor.ActorCell.invoke(ActorCell.scala:489) akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) akka.dispatch.Mailbox.run(Mailbox.scala:224) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[RMI TCP Accept-3301,5,system] java.net.PlainSocketImpl.socketAccept(Native Method) java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409) java.net.ServerSocket.implAccept(ServerSocket.java:545) java.net.ServerSocket.accept(ServerSocket.java:513) sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:400) sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:372) java.lang.Thread.run(Thread.java:745) Thread[java-sdk-http-connection-reaper,5,main] java.lang.Thread.sleep(Native Method) com.amazonaws.http.IdleConnectionReaper.run(IdleConnectionReaper.java:181) Thread[KafkaProcessors-akka.kafka.transactions-dispatcher-162,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[KafkaProcessors-akka.actor.transactions-dispatcher-153,5,main] sun.misc.Unsafe.park(Native Method) akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075) akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Thread[KafkaProcessors-akka.actor.radar-dispatcher-20,5,main] sun.misc.Unsafe.park(Native Method) akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075) akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Thread[New I/O worker #2,5,main] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434) org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[logback-2,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[Hashed wheel timer #2,5,main] java.lang.Thread.sleep(Native Method) org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:445) org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:364) org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) java.lang.Thread.run(Thread.java:745) Thread[AsyncExecutor.default-5,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) slick.util.ManagedArrayBlockingQueue$$anonfun$take$1.apply(ManagedArrayBlockingQueue.scala:115) slick.util.ManagedArrayBlockingQueue$$anonfun$take$1.apply(ManagedArrayBlockingQueue.scala:114) slick.util.ManagedArrayBlockingQueue.lockedInterruptibly(ManagedArrayBlockingQueue.scala:222) slick.util.ManagedArrayBlockingQueue.take(ManagedArrayBlockingQueue.scala:114) slick.util.ManagedArrayBlockingQueue.take(ManagedArrayBlockingQueue.scala:13) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[threadDeathWatcher-9-1,1,main] java.lang.Thread.sleep(Native Method) com.datastax.shaded.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:152) com.datastax.shaded.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:745) Thread[KafkaProcessors-akka.kafka.transactions-dispatcher-168,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[cluster1-scheduled-task-worker-0,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) com.datastax.shaded.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:745) Thread[KafkaProcessors-akka.actor.default-dispatcher-125,5,main] sun.misc.Unsafe.park(Native Method) akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075) akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Thread[cluster1-nio-worker-1,5,main] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) com.datastax.shaded.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) com.datastax.shaded.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) com.datastax.shaded.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) com.datastax.shaded.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131) com.datastax.shaded.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:745) Thread[AsyncExecutor.default-14,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) slick.util.ManagedArrayBlockingQueue$$anonfun$take$1.apply(ManagedArrayBlockingQueue.scala:115) slick.util.ManagedArrayBlockingQueue$$anonfun$take$1.apply(ManagedArrayBlockingQueue.scala:114) slick.util.ManagedArrayBlockingQueue.lockedInterruptibly(ManagedArrayBlockingQueue.scala:222) slick.util.ManagedArrayBlockingQueue.take(ManagedArrayBlockingQueue.scala:114) slick.util.ManagedArrayBlockingQueue.take(ManagedArrayBlockingQueue.scala:13) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[AsyncExecutor.default-3,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) slick.util.ManagedArrayBlockingQueue$$anonfun$take$1.apply(ManagedArrayBlockingQueue.scala:115) slick.util.ManagedArrayBlockingQueue$$anonfun$take$1.apply(ManagedArrayBlockingQueue.scala:114) slick.util.ManagedArrayBlockingQueue.lockedInterruptibly(ManagedArrayBlockingQueue.scala:222) slick.util.ManagedArrayBlockingQueue.take(ManagedArrayBlockingQueue.scala:114) slick.util.ManagedArrayBlockingQueue.take(ManagedArrayBlockingQueue.scala:13) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[kamon-scheduler-1,5,main] java.lang.Thread.sleep(Native Method) akka.actor.LightArrayRevolverScheduler.waitNanos(LightArrayRevolverScheduler.scala:85) akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:265) akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:235) java.lang.Thread.run(Thread.java:745) Thread[kamon-akka.actor.default-dispatcher-3,5,main] sun.misc.Unsafe.park(Native Method) akka.dispatch.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135) akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067) akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Thread[New I/O worker #4,5,main] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434) org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[KafkaProcessors-akka.kafka.transactions-dispatcher-163,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[Reference Handler,10,system] java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) java.lang.ref.Reference.tryHandlePending(Reference.java:191) java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153) Thread[AsyncExecutor.default-6,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) slick.util.ManagedArrayBlockingQueue$$anonfun$take$1.apply(ManagedArrayBlockingQueue.scala:115) slick.util.ManagedArrayBlockingQueue$$anonfun$take$1.apply(ManagedArrayBlockingQueue.scala:114) slick.util.ManagedArrayBlockingQueue.lockedInterruptibly(ManagedArrayBlockingQueue.scala:222) slick.util.ManagedArrayBlockingQueue.take(ManagedArrayBlockingQueue.scala:114) slick.util.ManagedArrayBlockingQueue.take(ManagedArrayBlockingQueue.scala:13) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[KafkaProcessors-akka.actor.events-dispatcher-18,5,main] sun.misc.Unsafe.park(Native Method) akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075) akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) 2018-02-12T14:18:03.920+00:00 akka.kafka.KafkaConsumerActor WARN KafkaConsumer poll is taking significantly longer (6000ms) to return from poll then the configured poll interval (50ms). Waking up consumer to avoid thread starvation. 2018-02-12T14:18:03.921+00:00 akka.kafka.KafkaConsumerActor WARN Wake up has been triggered. Dumping stacks: Thread[AsyncExecutor.default-11,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) slick.util.ManagedArrayBlockingQueue$$anonfun$take$1.apply(ManagedArrayBlockingQueue.scala:115) slick.util.ManagedArrayBlockingQueue$$anonfun$take$1.apply(ManagedArrayBlockingQueue.scala:114) slick.util.ManagedArrayBlockingQueue.lockedInterruptibly(ManagedArrayBlockingQueue.scala:222) slick.util.ManagedArrayBlockingQueue.take(ManagedArrayBlockingQueue.scala:114) slick.util.ManagedArrayBlockingQueue.take(ManagedArrayBlockingQueue.scala:13) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[KafkaProcessors-akka.actor.default-dispatcher-122,5,main] sun.misc.Unsafe.park(Native Method) akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075) akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Thread[KafkaProcessors-akka.kafka.transactions-dispatcher-173,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[cluster1-nio-worker-3,5,main] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) com.datastax.shaded.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) com.datastax.shaded.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) com.datastax.shaded.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) com.datastax.shaded.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131) com.datastax.shaded.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:745) Thread[logback-1,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[AsyncExecutor.default-4,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) slick.util.ManagedArrayBlockingQueue$$anonfun$take$1.apply(ManagedArrayBlockingQueue.scala:115) slick.util.ManagedArrayBlockingQueue$$anonfun$take$1.apply(ManagedArrayBlockingQueue.scala:114) slick.util.ManagedArrayBlockingQueue.lockedInterruptibly(ManagedArrayBlockingQueue.scala:222) slick.util.ManagedArrayBlockingQueue.take(ManagedArrayBlockingQueue.scala:114) slick.util.ManagedArrayBlockingQueue.take(ManagedArrayBlockingQueue.scala:13) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[cluster1-timeouter-0,5,main] java.lang.Thread.sleep(Native Method) com.datastax.shaded.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:560) com.datastax.shaded.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:459) com.datastax.shaded.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:745) Thread[KafkaProcessors-akka.actor.default-dispatcher-121,5,main] java.util.regex.Pattern$Branch.match(Pattern.java:4604) java.util.regex.Pattern$BranchConn.match(Pattern.java:4568) java.util.regex.Pattern$GroupTail.match(Pattern.java:4717) java.util.regex.Pattern$Curly.match0(Pattern.java:4272) java.util.regex.Pattern$Curly.match(Pattern.java:4234) java.util.regex.Pattern$GroupHead.match(Pattern.java:4658) java.util.regex.Pattern$Branch.match(Pattern.java:4604) java.util.regex.Pattern$Branch.match(Pattern.java:4602) java.util.regex.Pattern$BmpCharProperty.match(Pattern.java:3798) java.util.regex.Pattern$Start.match(Pattern.java:3461) java.util.regex.Matcher.search(Matcher.java:1248) java.util.regex.Matcher.find(Matcher.java:664) java.util.Formatter.parse(Formatter.java:2549) java.util.Formatter.format(Formatter.java:2501) java.util.Formatter.format(Formatter.java:2455) java.lang.String.format(String.java:2940) scala.collection.immutable.StringLike$class.format(StringLike.scala:318) scala.collection.immutable.StringOps.format(StringOps.scala:29) akka.util.Helpers$.currentTimeMillisToUTCString(Helpers.scala:64) akka.event.slf4j.Slf4jLogger.formatTimestamp(Slf4jLogger.scala:124) akka.event.slf4j.Slf4jLogger.withMdc_aroundBody0(Slf4jLogger.scala:100) akka.event.slf4j.Slf4jLogger$AjcClosure1.run(Slf4jLogger.scala:1) org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) akka.kamon.instrumentation.ActorLoggingInstrumentation$$anonfun$aroundWithMdcInvocation$1$$anonfun$apply$1.apply(ActorLoggingInstrumentation.scala:46) kamon.trace.logging.MdcKeysSupport$class.withMdc(MdcKeysSupport.scala:33) akka.kamon.instrumentation.ActorLoggingInstrumentation.withMdc(ActorLoggingInstrumentation.scala:25) akka.kamon.instrumentation.ActorLoggingInstrumentation$$anonfun$aroundWithMdcInvocation$1.apply(ActorLoggingInstrumentation.scala:45) kamon.trace.Tracer$.withContext(TracerModule.scala:58) akka.kamon.instrumentation.ActorLoggingInstrumentation.aroundWithMdcInvocation(ActorLoggingInstrumentation.scala:44) akka.event.slf4j.Slf4jLogger.withMdc(Slf4jLogger.scala:98) akka.event.slf4j.Slf4jLogger$$anonfun$receive$1.applyOrElse(Slf4jLogger.scala:77) akka.actor.Actor$class.aroundReceive(Actor.scala:517) akka.event.slf4j.Slf4jLogger.aroundReceive(Slf4jLogger.scala:57) akka.actor.ActorCell.receiveMessage(ActorCell.scala:527) akka.actor.ActorCell.invoke_aroundBody0(ActorCell.scala:496) akka.actor.ActorCell$AjcClosure1.run(ActorCell.scala:1) org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) akka.kamon.instrumentation.ActorMonitors$$anon$1$$anonfun$processMessage$1.apply(ActorMonitor.scala:59) kamon.trace.Tracer$.withContext(TracerModule.scala:58) akka.kamon.instrumentation.ActorMonitors$$anon$1.processMessage(ActorMonitor.scala:58) akka.kamon.instrumentation.ActorCellInstrumentation.aroundBehaviourInvoke(ActorInstrumentation.scala:44) akka.actor.ActorCell.invoke(ActorCell.scala:489) akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) akka.dispatch.Mailbox.run(Mailbox.scala:224) akka.dispatch.Mailbox.exec(Mailbox.scala:234) akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Thread[AsyncExecutor.default-1,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) slick.util.ManagedArrayBlockingQueue$$anonfun$take$1.apply(ManagedArrayBlockingQueue.scala:115) slick.util.ManagedArrayBlockingQueue$$anonfun$take$1.apply(ManagedArrayBlockingQueue.scala:114) slick.util.ManagedArrayBlockingQueue.lockedInterruptibly(ManagedArrayBlockingQueue.scala:222) slick.util.ManagedArrayBlockingQueue.take(ManagedArrayBlockingQueue.scala:114) slick.util.ManagedArrayBlockingQueue.take(ManagedArrayBlockingQueue.scala:13) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[kamon-akka.io.pinned-dispatcher-6,5,main] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) akka.io.SelectionHandler$ChannelRegistryImpl$$anon$3.tryRun(SelectionHandler.scala:128) akka.io.SelectionHandler$ChannelRegistryImpl$Task.run(SelectionHandler.scala:246) akka.io.SelectionHandler$ChannelRegistryImpl$$anon$3.run(SelectionHandler.scala:161) akka.util.SerializedSuspendableExecutionContext.run$1(SerializedSuspendableExecutionContext.scala:67) akka.util.SerializedSuspendableExecutionContext.run(SerializedSuspendableExecutionContext.scala:71) akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[KafkaProcessors-akka.kafka.transactions-dispatcher-164,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[HikariPool-1 housekeeper,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[KafkaProcessors-akka.kafka.transactions-dispatcher-174,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[AsyncExecutor.default-7,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) slick.util.ManagedArrayBlockingQueue$$anonfun$take$1.apply(ManagedArrayBlockingQueue.scala:115) slick.util.ManagedArrayBlockingQueue$$anonfun$take$1.apply(ManagedArrayBlockingQueue.scala:114) slick.util.ManagedArrayBlockingQueue.lockedInterruptibly(ManagedArrayBlockingQueue.scala:222) slick.util.ManagedArrayBlockingQueue.take(ManagedArrayBlockingQueue.scala:114) slick.util.ManagedArrayBlockingQueue.take(ManagedArrayBlockingQueue.scala:13) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[AsyncExecutor.default-2,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) slick.util.ManagedArrayBlockingQueue$$anonfun$take$1.apply(ManagedArrayBlockingQueue.scala:115) slick.util.ManagedArrayBlockingQueue$$anonfun$take$1.apply(ManagedArrayBlockingQueue.scala:114) slick.util.ManagedArrayBlockingQueue.lockedInterruptibly(ManagedArrayBlockingQueue.scala:222) slick.util.ManagedArrayBlockingQueue.take(ManagedArrayBlockingQueue.scala:114) slick.util.ManagedArrayBlockingQueue.take(ManagedArrayBlockingQueue.scala:13) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[Hashed wheel timer #3,5,main] java.lang.Thread.sleep(Native Method) org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:445) org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:364) org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) java.lang.Thread.run(Thread.java:745) Thread[kamon-akka.actor.default-dispatcher-4,5,main] sun.misc.Unsafe.park(Native Method) akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075) akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Thread[KafkaProcessors-akka.kafka.transactions-dispatcher-176,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[AsyncExecutor.default-10,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) slick.util.ManagedArrayBlockingQueue$$anonfun$take$1.apply(ManagedArrayBlockingQueue.scala:115) slick.util.ManagedArrayBlockingQueue$$anonfun$take$1.apply(ManagedArrayBlockingQueue.scala:114) slick.util.ManagedArrayBlockingQueue.lockedInterruptibly(ManagedArrayBlockingQueue.scala:222) slick.util.ManagedArrayBlockingQueue.take(ManagedArrayBlockingQueue.scala:114) slick.util.ManagedArrayBlockingQueue.take(ManagedArrayBlockingQueue.scala:13) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[AsyncExecutor.default-13,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) slick.util.ManagedArrayBlockingQueue$$anonfun$take$1.apply(ManagedArrayBlockingQueue.scala:115) slick.util.ManagedArrayBlockingQueue$$anonfun$take$1.apply(ManagedArrayBlockingQueue.scala:114) slick.util.ManagedArrayBlockingQueue.lockedInterruptibly(ManagedArrayBlockingQueue.scala:222) slick.util.ManagedArrayBlockingQueue.take(ManagedArrayBlockingQueue.scala:114) slick.util.ManagedArrayBlockingQueue.take(ManagedArrayBlockingQueue.scala:13) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[KafkaProcessors-akka.actor.default-dispatcher-110,5,main] sun.misc.Unsafe.park(Native Method) akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075) akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Thread[KafkaProcessors-akka.kafka.transactions-dispatcher-171,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[KafkaProcessors-akka.actor.notifications-dispatcher-13,5,main] sun.misc.Unsafe.park(Native Method) akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075) akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Thread[New I/O worker #3,5,main] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434) org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[KafkaProcessors-akka.actor.notifications-dispatcher-21,5,main] sun.misc.Unsafe.park(Native Method) akka.dispatch.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135) akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067) akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Thread[cluster1-nio-worker-0,5,main] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) com.datastax.shaded.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) com.datastax.shaded.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) com.datastax.shaded.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) com.datastax.shaded.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131) com.datastax.shaded.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:745) Thread[AsyncExecutor.default-12,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) slick.util.ManagedArrayBlockingQueue$$anonfun$take$1.apply(ManagedArrayBlockingQueue.scala:115) slick.util.ManagedArrayBlockingQueue$$anonfun$take$1.apply(ManagedArrayBlockingQueue.scala:114) slick.util.ManagedArrayBlockingQueue.lockedInterruptibly(ManagedArrayBlockingQueue.scala:222) slick.util.ManagedArrayBlockingQueue.take(ManagedArrayBlockingQueue.scala:114) slick.util.ManagedArrayBlockingQueue.take(ManagedArrayBlockingQueue.scala:13) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[KafkaProcessors-akka.kafka.transactions-dispatcher-172,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[Finalizer,8,system] java.lang.Object.wait(Native Method) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164) java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209) Thread[KafkaProcessors-akka.kafka.transactions-dispatcher-159,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[DestroyJavaVM,5,main] Thread[KafkaProcessors-akka.actor.default-dispatcher-120,5,main] sun.misc.Unsafe.park(Native Method) akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075) akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Thread[cluster1-connection-reaper-0,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) com.datastax.shaded.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:745) Thread[KafkaProcessors-akka.kafka.transactions-dispatcher-160,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[Signal Dispatcher,9,system] Thread[KafkaProcessors-akka.kafka.transactions-dispatcher-175,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[RMI TCP Accept-0,5,system] java.net.PlainSocketImpl.socketAccept(Native Method) java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409) java.net.ServerSocket.implAccept(ServerSocket.java:545) java.net.ServerSocket.accept(ServerSocket.java:513) sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:400) sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:372) java.lang.Thread.run(Thread.java:745) Thread[KafkaProcessors-akka.kafka.transactions-dispatcher-170,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[KafkaProcessors-akka.kafka.transactions-dispatcher-169,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[cluster1-nio-worker-2,5,main] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) com.datastax.shaded.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) com.datastax.shaded.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) com.datastax.shaded.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) com.datastax.shaded.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131) com.datastax.shaded.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:745) Thread[New I/O boss #5,5,main] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434) org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42) org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[New I/O worker #1,5,main] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434) org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[kamon-akka.actor.default-dispatcher-5,5,main] sun.misc.Unsafe.park(Native Method) akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075) akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Thread[KafkaProcessors-akka.actor.default-dispatcher-108,5,main] java.lang.Thread.dumpThreads(Native Method) java.lang.Thread.getAllStackTraces(Thread.java:1607) akka.kafka.KafkaConsumerActor$$anonfun$1.apply$mcV$sp(KafkaConsumerActor.scala:272) akka.actor.Scheduler$$anon$4.run(Scheduler.scala:140) akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40) akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:43) akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Thread[AsyncExecutor.default-8,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) slick.util.ManagedArrayBlockingQueue$$anonfun$take$1.apply(ManagedArrayBlockingQueue.scala:115) slick.util.ManagedArrayBlockingQueue$$anonfun$take$1.apply(ManagedArrayBlockingQueue.scala:114) slick.util.ManagedArrayBlockingQueue.lockedInterruptibly(ManagedArrayBlockingQueue.scala:222) slick.util.ManagedArrayBlockingQueue.take(ManagedArrayBlockingQueue.scala:114) slick.util.ManagedArrayBlockingQueue.take(ManagedArrayBlockingQueue.scala:13) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[ForkJoinPool-1-worker-3,5,main] sun.misc.Unsafe.park(Native Method) scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075) scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Thread[Abandoned connection cleanup thread,5,main] java.lang.Object.wait(Native Method) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143) com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:43) Thread[kamon-akka.actor.default-dispatcher-2,5,main] sun.misc.Unsafe.park(Native Method) akka.dispatch.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135) akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067) akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Thread[KafkaProcessors-scheduler-1,5,main] java.lang.Thread.sleep(Native Method) akka.actor.LightArrayRevolverScheduler.waitNanos(LightArrayRevolverScheduler.scala:85) akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:265) akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:235) java.lang.Thread.run(Thread.java:745) Thread[KafkaProcessors-akka.kafka.transactions-dispatcher-157,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[AsyncExecutor.default-9,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) slick.util.ManagedArrayBlockingQueue$$anonfun$take$1.apply(ManagedArrayBlockingQueue.scala:115) slick.util.ManagedArrayBlockingQueue$$anonfun$take$1.apply(ManagedArrayBlockingQueue.scala:114) slick.util.ManagedArrayBlockingQueue.lockedInterruptibly(ManagedArrayBlockingQueue.scala:222) slick.util.ManagedArrayBlockingQueue.take(ManagedArrayBlockingQueue.scala:114) slick.util.ManagedArrayBlockingQueue.take(ManagedArrayBlockingQueue.scala:13) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[KafkaProcessors-akka.kafka.transactions-dispatcher-165,5,main] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.kafka.common.network.Selector.select(Selector.java:529) org.apache.kafka.common.network.Selector.poll(Selector.java:321) org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:433) org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:232) org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:208) org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:199) org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.awaitMetadataUpdate(ConsumerNetworkClient.java:134) org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:226) org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:203) org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:286) org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1078) org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1043) akka.kafka.KafkaConsumerActor.tryPoll$1(KafkaConsumerActor.scala:287) akka.kafka.KafkaConsumerActor.poll(KafkaConsumerActor.scala:334) akka.kafka.KafkaConsumerActor.akka$kafka$KafkaConsumerActor$$receivePoll(KafkaConsumerActor.scala:256) akka.kafka.KafkaConsumerActor$$anonfun$receive$1.applyOrElse(KafkaConsumerActor.scala:164) akka.actor.Actor$class.aroundReceive(Actor.scala:517) akka.kafka.KafkaConsumerActor.aroundReceive(KafkaConsumerActor.scala:79) akka.actor.ActorCell.receiveMessage(ActorCell.scala:527) akka.actor.ActorCell.invoke_aroundBody0(ActorCell.scala:496) akka.actor.ActorCell$AjcClosure1.run(ActorCell.scala:1) org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) akka.kamon.instrumentation.ActorMonitors$$anon$1$$anonfun$processMessage$1.apply(ActorMonitor.scala:59) kamon.trace.Tracer$.withContext(TracerModule.scala:58) akka.kamon.instrumentation.ActorMonitors$$anon$1.processMessage(ActorMonitor.scala:58) akka.kamon.instrumentation.ActorCellInstrumentation.aroundBehaviourInvoke(ActorInstrumentation.scala:44) akka.actor.ActorCell.invoke(ActorCell.scala:489) akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) akka.dispatch.Mailbox.run(Mailbox.scala:224) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[RMI TCP Accept-3301,5,system] java.net.PlainSocketImpl.socketAccept(Native Method) java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409) java.net.ServerSocket.implAccept(ServerSocket.java:545) java.net.ServerSocket.accept(ServerSocket.java:513) sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:400) sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:372) java.lang.Thread.run(Thread.java:745) Thread[java-sdk-http-connection-reaper,5,main] java.lang.Thread.sleep(Native Method) com.amazonaws.http.IdleConnectionReaper.run(IdleConnectionReaper.java:181) Thread[KafkaProcessors-akka.kafka.transactions-dispatcher-162,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[KafkaProcessors-akka.actor.transactions-dispatcher-153,5,main] sun.misc.Unsafe.park(Native Method) akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075) akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Thread[KafkaProcessors-akka.actor.radar-dispatcher-20,5,main] sun.misc.Unsafe.park(Native Method) akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075) akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Thread[New I/O worker #2,5,main] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434) org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[logback-2,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[Hashed wheel timer #2,5,main] java.lang.Thread.sleep(Native Method) org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:445) org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:364) org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) java.lang.Thread.run(Thread.java:745) Thread[AsyncExecutor.default-5,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) slick.util.ManagedArrayBlockingQueue$$anonfun$take$1.apply(ManagedArrayBlockingQueue.scala:115) slick.util.ManagedArrayBlockingQueue$$anonfun$take$1.apply(ManagedArrayBlockingQueue.scala:114) slick.util.ManagedArrayBlockingQueue.lockedInterruptibly(ManagedArrayBlockingQueue.scala:222) slick.util.ManagedArrayBlockingQueue.take(ManagedArrayBlockingQueue.scala:114) slick.util.ManagedArrayBlockingQueue.take(ManagedArrayBlockingQueue.scala:13) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[threadDeathWatcher-9-1,1,main] java.lang.Thread.sleep(Native Method) com.datastax.shaded.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:152) com.datastax.shaded.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:745) Thread[KafkaProcessors-akka.kafka.transactions-dispatcher-168,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[cluster1-scheduled-task-worker-0,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) com.datastax.shaded.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:745) Thread[KafkaProcessors-akka.actor.default-dispatcher-125,5,main] sun.misc.Unsafe.park(Native Method) akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075) akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Thread[cluster1-nio-worker-1,5,main] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) com.datastax.shaded.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) com.datastax.shaded.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) com.datastax.shaded.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) com.datastax.shaded.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131) com.datastax.shaded.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:745) Thread[AsyncExecutor.default-14,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) slick.util.ManagedArrayBlockingQueue$$anonfun$take$1.apply(ManagedArrayBlockingQueue.scala:115) slick.util.ManagedArrayBlockingQueue$$anonfun$take$1.apply(ManagedArrayBlockingQueue.scala:114) slick.util.ManagedArrayBlockingQueue.lockedInterruptibly(ManagedArrayBlockingQueue.scala:222) slick.util.ManagedArrayBlockingQueue.take(ManagedArrayBlockingQueue.scala:114) slick.util.ManagedArrayBlockingQueue.take(ManagedArrayBlockingQueue.scala:13) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[AsyncExecutor.default-3,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) slick.util.ManagedArrayBlockingQueue$$anonfun$take$1.apply(ManagedArrayBlockingQueue.scala:115) slick.util.ManagedArrayBlockingQueue$$anonfun$take$1.apply(ManagedArrayBlockingQueue.scala:114) slick.util.ManagedArrayBlockingQueue.lockedInterruptibly(ManagedArrayBlockingQueue.scala:222) slick.util.ManagedArrayBlockingQueue.take(ManagedArrayBlockingQueue.scala:114) slick.util.ManagedArrayBlockingQueue.take(ManagedArrayBlockingQueue.scala:13) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[kamon-scheduler-1,5,main] java.lang.Thread.sleep(Native Method) akka.actor.LightArrayRevolverScheduler.waitNanos(LightArrayRevolverScheduler.scala:85) akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:265) akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:235) java.lang.Thread.run(Thread.java:745) Thread[kamon-akka.actor.default-dispatcher-3,5,main] sun.misc.Unsafe.park(Native Method) akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075) akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Thread[New I/O worker #4,5,main] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434) org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[KafkaProcessors-akka.kafka.transactions-dispatcher-163,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[Reference Handler,10,system] java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) java.lang.ref.Reference.tryHandlePending(Reference.java:191) java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153) Thread[AsyncExecutor.default-6,5,main] sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) slick.util.ManagedArrayBlockingQueue$$anonfun$take$1.apply(ManagedArrayBlockingQueue.scala:115) slick.util.ManagedArrayBlockingQueue$$anonfun$take$1.apply(ManagedArrayBlockingQueue.scala:114) slick.util.ManagedArrayBlockingQueue.lockedInterruptibly(ManagedArrayBlockingQueue.scala:222) slick.util.ManagedArrayBlockingQueue.take(ManagedArrayBlockingQueue.scala:114) slick.util.ManagedArrayBlockingQueue.take(ManagedArrayBlockingQueue.scala:13) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread[KafkaProcessors-akka.actor.events-dispatcher-18,5,main] sun.misc.Unsafe.park(Native Method) akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075) akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) ```

Hoping this sheds some light into the issue. Let me know if you find anything awkward in here.

Thanks in advance for any help around this.

[chbatey]

Thanks for getting the logs @sbrnunes the stacks we added to the logs can tell us what is going on in the KafkaConsumer

Thread[KafkaProcessors-akka.kafka.transactions-dispatcher-165,5,main]
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
org.apache.kafka.common.network.Selector.select(Selector.java:529)
org.apache.kafka.common.network.Selector.poll(Selector.java:321)
org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:433)
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:232)
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:208)
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:199)
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.awaitMetadataUpdate(ConsumerNetworkClient.java:134)
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:226)
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:203)
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:286)
org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1078)
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1043)
akka.kafka.KafkaConsumerActor.tryPoll$1(KafkaConsumerActor.scala:287)
akka.kafka.KafkaConsumerActor.poll(KafkaConsumerActor.scala:334)
akka.kafka.KafkaConsumerActor.akka$kafka$KafkaConsumerActor$$receivePoll(KafkaConsumerActor.scala:256)
akka.kafka.KafkaConsumerActor$$anonfun$receive$1.applyOrElse(KafkaConsumerActor.scala:164)
akka.actor.Actor$class.aroundReceive(Actor.scala:517)
akka.kafka.KafkaConsumerActor.aroundReceive(KafkaConsumerActor.scala:79)

It is currently stuck in the Kafka client in a method that blocks for every (well Long.MAX milliseconds) waiting for the consumer group coordinator to be ready:

     * Block until the coordinator for this group is known and is ready to receive requests.
     */
    public synchronized void ensureCoordinatorReady() {
        // Using zero as current time since timeout is effectively infinite
        ensureCoordinatorReady(0, Long.MAX_VALUE);
    }

The ensureCoordinaroReady inner method loops for every, sleeping after each try, for the coordinator to be ready.

So there's nothing we can do inside reactive-kafka to prevent this our options are:

• Leave a thread blocked in the Kafka client indefinitely
• Trigger the wake up and timeout after so many wake ups (what we currently do).

What do other users think? I think it is a matter of increasing the timeout/number of retries.

However there could be other reasons so do keep pasting any of the logs which have the stack traces.

The kafka client logs that is doing this constant loop but only at debug level, set the org.apache.kafka.clients.consumer.internals logger to DEBUG to see it, looks like this:

   log.debug("Coordinator discovery failed for group {}, refreshing metadata", groupId);

[sbrnunes]

Thanks for your quick response @chbatey. Appreciated.

Trigger the wake up and timeout after so many wake ups (what we currently do).

I did notice that reactive-kafka was doing something to recover from the permanently blocked treads, from your previous comments and by looking at the custom configuration that it provides (wakeup-timeout and max-wakeups).

What should we expect to happen when the max-wakeups expire? Should the stream just die, regardless of the supervision techniques we put in place, or should the BackoffSupervisor kick off and restart the stream, in which case it would go into another loop of wakeup attempts? I suppose the Supervisor should kick off, I'm just confirming.

I believe this is where we're having issues. After the max-wakeups expire, it seems that reactive-kafka, in certain situations, is not shutting down the ConsumerStage, meaning, our Supervisor never kicks off.

I forgot to mention something in my previous comment that might be relevant: this issues are happening when we shut down a real cluster, i.e., when we shut down our Kafka processes running in VMs. If we try to reproduce this locally, by spinning up a Kafka cluster in Docker Compose, everything works just fine (supervisor kicks off and keeps restarting the stream over and over again).

Maybe there's a difference, from a networking perspective, between:

• Shutting down the Kafka processes running in the VMs, in which case the VMs IP Addresses / Hostnames will still be available
• Shutting down the docker containers where Kafka is running, in which case the IP Addresses / Hostnames won't be available

Leave a thread blocked in the Kafka client indefinitely

Would this mean that the ConsumerStage, and the stream, would get stuck until the coordinator is back? Would it just recover automatically and start consuming the moment the coordinator is back?

BTW: all this issues should be easily reproducible if you're able to:

• Spin up a Kakfa cluster, in bare-metal servers or in VMs
• Shut it down
• Wait for the max-wakeups to expire

In the meantime, we can definitely try to increase the log level and run the same tests, but I believe they will just confirm what you shared.

[jorkzijlstra]

We noticed the same issue. For us it was not reproducible using Docker on a single machine but it was on 2 separate machine.

1) Machine 1: boot up a Kafka Cluster 2) Machine 2: Start a consumer 3) Machine 1: Produce a message 4) Machine 2: Remove network cable and wait for the wakeup exception 5) Machine 1: Produce a message 6) Machine 2: Insert network cable and see nothing happening

When debugging we do see that the following lines are hit:

if (wakeups == settings.maxWakeups) {
  log.error("WakeupException limit exceeded, stopping.")
  context.stop(self)
}

but using a supervisor to restart onStop doensn't seem to help.

val supervisorProps = BackoffSupervisor.props(
        Backoff.onStop(
          kafkaConsumerProps,
          childName = "kafkaConsumer",
          minBackoff = 2 seconds,
          maxBackoff = 30 seconds,
          randomFactor = 0.2
)

In the end we made a change in out service so that our health endpoint became unhealthy when this happened and have a supervisor outside the service restart the whole service itself.

[sbrnunes]

Hi again,

Just so we are aware, is anyone actively looking at or debugging this issue, following the previous post with tips on how to reproduce the problem locally?

[jonbuffington]

I found a work around for my use case. I have six topics where some topics have records produced at a steady but low rate. The other topics have frequent bursts of records being produced. If all six topics are listed in one Consumer.committableSource(), the offset commit timeout warnings and eventual failure occurs. After I implemented one Consumer.committableSource() per topic, I no longer see the offset commit timeout warnings and eventual failure.

[sbrnunes]

Interesting that this worked for you @jonbuffington. But do you mean a separate committable source per topic or per partition? Do your topics have only one partition? Or all those partitions are consumed by only one commitableSource?

In our case, we do have a separate consumer group for each topic and a separate commitableSource subscribed to each consumer group. However, each topic has a few very unbalanced partitions, due to the partition key we use. So, a single committableSource is actually consuming from a few partitions (around 8 partitions) and we start a few of those committableSources for each consumer group.

We see problems happening with any of this consumer groups.

I don't think our particular problem is related to this. I do suspect there's something going on at the library level when there's an issue connecting to the cluster, as @jorkzijlstra also shared above.