twitter / finagle

A fault tolerant, protocol-agnostic RPC system
https://twitter.github.io/finagle
Apache License 2.0
8.79k stars 1.45k forks source link

Unable to stop Netty 4 Timer #757

Closed dani-e-l closed 2 years ago

dani-e-l commented 5 years ago

Hello we are using Finagle as a HttpClient - it works really good. Great job.

Unfortunately Finagle generates resource leak under standard behavior that applications are deployed and undeployed many times on JBoss AS, Tomcat etc. By resource leak I mean that if I 100 times deploy and undeploy an application that uses Finagle I'll get at least 100 threads

The problem is that finagle contains unstopable threads

eg.

private[netty4] class Netty4HashedWheelTimer
    extends Netty4Timer(HashedWheelTimer.instance)
    with ServiceLoadedTimer {

  private[this] val log = Logger.get()

  // This timer is "unstoppable".
  override def stop(): Unit =
    log.warning(
      s"Ignoring call to `Timer.stop()` on an unstoppable Netty4Timer.\n" +
        s"Current stack trace: ${Thread.currentThread.getStackTrace.mkString("\n")}"
    )
}
kevinoliver commented 5 years ago

Ah. I suspect there may be other issues like this as haven't designed for that type of deployment model.

This might require a patch to create a flag that allows you to opt-out of this behavior.

dani-e-l commented 5 years ago

Thanks for quick reply Kevin. I was able to close many other threads because API allows me to use my own Timers/Executors/etc I have problem with last three threads.

In this case private object is used and I have no access to it.

mpnewcomb commented 5 years ago

HashedWheelTimer... why are you using this? It is a cpu hog. Start up linkerd and namerd and they just sit their chewing anywhere from 1-3% cpu when nothing is happening...

kevinoliver commented 5 years ago

@mpnewcomb do you have any suggestions on how to improve it? if there is no work to do, it should be a thread that sleeps for a few milliseconds, looks for work, then repeats.

drywet commented 5 years ago

Hello. This seems to work for me: Create an object with a method in package com.twitter.finagle.netty4 to access private properties and stop com.twitter.finagle.util.HashedWheelTimer.nettyHwt and HashedWheelTimer.instance timers

tigerlily-he commented 2 years ago

This feature to stop a Netty 4 Timer was added in https://github.com/twitter/finagle/commit/fa4899e1ba1e9b612dbf7428b0f457ae3855b5b8. Closing this issue now that it's resolved.