Metaspace memory leak

[sheaney]

Hi, I was hoping to see if anyone else is getting a memory leak with this application. This might be a more general issue and not specific to this project.

To reproduce this error, all I need to do is start up SBT, then repeatedly do the following process until exception occurs: start the application by typing run and then stopping it by pressing enter. This seems to cause SBT's ClassLoader to create enough classes to exceed the maximum metaspace allocated and get a java.lang.OutOfMemoryError: Metaspace exception.

If anyone has a pointer or suggestion on how to debug this I would be glad to look into it.

> uname -a Darwin XYZ-MacBook-Pro.local 15.6.0 Darwin Kernel Version 15.6.0: Sun Jun 4 21:43:07 PDT 2017; root:xnu-3248.70.3~1/RELEASE_X86_64 x86_64

> java -version java version "9" Java(TM) SE Runtime Environment (build 9+181) Java HotSpot(TM) 64-Bit Server VM (build 9+181, mixed mode)

JVM arguments

-Xms1024m -Xmx1024m -XX:ReservedCodeCacheSize=128m -XX:MaxMetaspaceSize=256m -Dscala.ext.dirs=/Users/xyz/.sbt/0.13/java9-rt-ext-oracle_corporation_9 --add-modules=java.xml.bind

VisualVM screenshots

Command line error

[info] p.c.s.AkkaHttpServer - Stopping server... java.lang.OutOfMemoryError: Metaspace Dumping heap to /var/folders/ph/n4_6pq_x6wx025j00w4dzwzr0000gn/T/visualvm.dat/localhost_22461/java_pid22461.hprof ... Heap dump file created [155098322 bytes in 0.996 secs] Uncaught error from thread [play-dev-mode-akka.actor.default-dispatcher-3]: Metaspace, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for for ActorSystem[play-dev-mode] java.lang.OutOfMemoryError: Metaspace at java.base/java.lang.ClassLoader.defineClass1(Native Method) at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1007) at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174) at java.base/java.net.URLClassLoader.defineClass(URLClassLoader.java:545) at java.base/java.net.URLClassLoader.access$100(URLClassLoader.java:83) at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:453) at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:447) at java.base/java.security.AccessController.doPrivileged(Native Method) at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:446) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:563) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:496) at java.base/java.lang.ClassLoader.defineClass1(Native Method) at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1007) at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174) at java.base/java.net.URLClassLoader.defineClass(URLClassLoader.java:545) at java.base/java.net.URLClassLoader.access$100(URLClassLoader.java:83) at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:453) at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:447) at java.base/java.security.AccessController.doPrivileged(Native Method) at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:446) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:563) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:496) at scala.collection.immutable.RedBlackTree$.valuesIterator(RedBlackTree.scala:115) at scala.collection.immutable.TreeMap.valuesIterator(TreeMap.scala:196) at akka.actor.dungeon.ChildrenContainer$ChildrenIterable.valuesIterator(ChildrenContainer.scala:61) at akka.util.Collections$PartialImmutableValuesIterable.iterator(Collections.scala:26) at akka.util.Collections$PartialImmutableValuesIterable.foreach(Collections.scala:52) at akka.actor.dungeon.FaultHandling.terminate(FaultHandling.scala:150) at akka.actor.dungeon.FaultHandling.terminate$(FaultHandling.scala:142) at akka.actor.ActorCell.terminate(ActorCell.scala:370) at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:468) at akka.actor.ActorCell.systemInvoke(ActorCell.scala:484)

[sergeych]

I do all the time

[wsargent]

Could be connected to https://github.com/playframework/playframework/issues/4649

[dqdinh]

Adding more memory to sbt while developing usually resolves this issue for me.

sbt -mem 2048

[mt40]

@dqdinh how do you find the -mem option, I could not find it anywhere in sbt doc

[dwijnand]

@mt40 it's in the -h for the sbt-launcher-package script: https://github.com/sbt/sbt-launcher-package/blob/v1.2.7/src/universal/bin/sbt#L82

[thobson]

I have this problem. Bumping sbt's mem to 2048 helps, but I still run into the issue pretty soon (5-10 restarts)

[benmccann]

@sheaney here are some instructions for debugging that you might want to try: https://github.com/playframework/playframework/issues/4469

If your SBT instance is crashing after only one or two reloads you can increase the memory to give you longer (-J-XX:MaxMetaspaceSize=512m)

[benmccann]

I see from jstack that "JNI global references" count increases with every reload

Here are some ideas for debugging:

• https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/memleaks005.html
• https://stackoverflow.com/questions/6423873/how-to-identify-the-cause-of-a-jni-global-reference-memory-leak

[benmccann]

I tried -XX:TraceJNICalls, but no luck. Perhaps someone else is willing to figure out how to get or compile a debug version of the VM and move the ball forward some more on debugging this:

Error: VM option 'TraceJNICalls' is develop and is available only in debug version of VM.

[Rogach]

Possibly related? https://github.com/sbt/sbt/issues/4166

I had metaspace leak problem as well, but after setting scalaVersion := 2.12.7 and sbt version 1.2.8 the metaspace leak went away.

[marcospereira]

Thanks for the update, @Rogach.

@sheaney, @sergeych, @dqdinh, @thobson, and @benmccann, do you have a chance to test with Scala 2.12.8 and sbt 1.2.8 to verify if it works for you as well?