search

akka / akka-http

The Streaming-first HTTP server/module of Akka
https://doc.akka.io/libraries/akka-http/current/
Other
1.34k stars 594 forks source link

ConnectionPoolSpec leaking ActorSystems #1963

Open johanandren opened 6 years ago

johanandren commented 6 years ago

On MacOS, Java 8, All tests passing locally, but when I try to exit sbt: 

akka-http-root > [info] shutting down server
[INFO] [03/26/2018 18:06:04.182] [Thread-7] [CoordinatedShutdown(akka://sbt-web)] Starting coordinated shutdown from JVM shutdown hook

[WARN] [03/26/2018 18:06:14.186] [ConnectionPoolSpec-shutdown-hook-1] [CoordinatedShutdown(akka://ConnectionPoolSpec)] CoordinatedShutdown from JVM shutdown failed: Futures timed out after [10000 milliseconds]

After this things go completely bananas, the JVM does not show up with jps (but with ps) doesn't react to kill -9 etc. Didn't manage to get a thread dump either.

I have a feeling I have reported this before but couldn't find.

johanandren commented 6 years ago

Happened again, running against Akka 2.5-SNAPSHOT, all tests passing, managed to get a thread dump but I don't see any hints in it.

threaddump.txt

johanandren commented 6 years ago

More clues:

Failed to stop [ConnectionPoolSpec] within [10 seconds]
-> / LocalActorRefProvider$$anon$1 class akka.actor.LocalActorRefProvider$Guardian status=0 1 children
   ⌊-> system LocalActorRef class akka.actor.LocalActorRefProvider$SystemGuardian status=4 Terminating(Termination)
       |    toDie: Actor[akka://ConnectionPoolSpec/system/IO-TCP#549959584]
       ⌊-> IO-TCP RepointableActorRef class akka.io.TcpManager status=4 Terminating(Termination)
           |    toDie: Actor[akka://ConnectionPoolSpec/system/IO-TCP/selectors#1256998390]
           ⌊-> selectors RoutedActorRef class akka.routing.RouterPoolActor status=4 Terminating(Termination)
               |    toDie: Actor[akka://ConnectionPoolSpec/system/IO-TCP/selectors/$a#-125497430]
               ⌊-> $a LocalActorRef class akka.io.SelectionHandler status=4 Terminating(Termination)
                   |    toDie: Actor[akka://ConnectionPoolSpec/system/IO-TCP/selectors/$a/1638#1355620863]
                   |           Actor[akka://ConnectionPoolSpec/system/IO-TCP/selectors/$a/300#-1652885990]
                   |           Actor[akka://ConnectionPoolSpec/system/IO-TCP/selectors/$a/761#327052296]
                   ⌊-> 1638 LocalActorRef class akka.io.TcpOutgoingConnection status=2 no children
                   ⌊-> 300 LocalActorRef class akka.io.TcpOutgoingConnection status=2 no children
                   ⌊-> 761 LocalActorRef class akka.io.TcpOutgoingConnection status=2 no children
johanandren commented 6 years ago

More question marks, connecting to a stuck sbt that only ran NewConnectionPool and then got stuck, I get this in the terminal from jvisualvm:

ERROR: attach: task_for_pid(13310) failed: '(os/kern) failure' (5)

I can then see heap and metaspace usage is ok, but I cannot even see the tab that allows me to take a thread dump. jcmd though, claims there is no such process.

johanandren commented 6 years ago

New thread dump, this is after a single runOne of one of the tests in NewConnectionPoolSpec completed. Note the ~150 NIO threads, running something Netty... O.o

threaddump.txt