Kamon 0.4.0 Memory leak with Spray.

[tsukaby]

Hi! Kamon developers. Thank you for release 0.4.0!

I did update my program from 0.3.5 to 0.4.0. But I got a memory leak problem with following environments.

• Scala 2.11.6
• Kamon 0.4.0 (core, spray, statsd, datadog, system-metrics, sigar-loader)
• Spray 1.3.2 (can, routing, json, client)
• aspectjweaver 1.7.4

a part of application.conf

kamon.metric {
  disable-aspectj-weaver-missing-error = true

  filters {
    akka-actor {
      includes = [ "**" ]
      excludes = [ "*/system/**", "appname/user/io-http/**" ]
    }
    trace {
      includes = [ "**" ]
      excludes = []
    }
    akka-dispatcher {
      includes = [ "**" ]
      excludes = []
    }
  }
}

Statistics via jvisualvm.

Heap dump via jvisualvm.

I think it caused by spray-client or ActorCell.

My program used hostConnector in spray-client.
Problem occured when it used. My test running mode that not occured because spray-client doesn't used. (I used mock client)

Do you have any solutions or ideas? I did disable metrics of akka-actor now. (akka-actor.includes = [])

Sorry I have poor skill of English. (I am Japanese speaker.)

[nilsga]

I think this is similar to an issue we are seeing. Have you been able to find the cause, or have you put it on hold?

[szimano]

I am experiencing a very similar problem using aspectjweaver 1.8.6. any ideas?

[ivantopo]

@tsukaby thanks for sharing this! As @nilsga mentioned, this probably is related to the issue he fixed with #229, given the fact that your dump shows that you have tons of min max counters in memory it is almost certain that the issue is related to that.. so, cold you please try the very latest snapshot that contain that fix? That would be 0.4.1-2ce80225a5ae4d0ed7ba53be1ebe0734d4e4bca6 from our snapshots repo at http://snapshots.kamon.io, hope that the issue is gone there!

[tsukaby]

Thank you fix and build! I'm going to try it on this week.

I will contact you again later.

[tsukaby]

Thank you @nilsga ! Tnank you @ivantopo !

This problem solved on our system. (use 0.4.1-2ce80225a5ae4d0ed7ba53be1ebe0734d4e4bca6)

before (0.4.0)

after (0.4.1-2ce80225a5ae4d0ed7ba53be1ebe0734d4e4bca6)

Thanks a lot!

[jozic]

@ivantopo when do you guys plan to release 0.4.1 with this fix?

[ivantopo]

It should happen really, really soon. I find it weird that not too many people complained, yet 0.4.0 is by far the most downloaded version we have! If possible, please use the snapshot now and then switch to 0.4.1 when it becomes available!

[tsukaby]

I'm also waiting for the next release. However, we are satisfied with the snapshot for now. (We are used it in production environment. :yum: )

Thank you!

[jozic]

thanks! I saw similar issue with lots of memory consumed after we started app with Kamon, but did have a chance to verify the exact reason. Will try snapshot and crossing fingers for the soon release :)

[dodnert]

FYI - My project was impacted by this issue as well. Using version 0.4.1-2ce80225a5ae4d0ed7ba53be1ebe0734d4e4bca6, the problem was resolved.