kamon-io / Kamon

Distributed Tracing, Metrics and Context Propagation for applications running on the JVM
https://kamon.io
Other
1.41k stars 328 forks source link

Kamon 0.4.0 Memory leak with Spray. #213

Closed tsukaby closed 9 years ago

tsukaby commented 9 years ago

Hi! Kamon developers. Thank you for release 0.4.0!

I did update my program from 0.3.5 to 0.4.0. But I got a memory leak problem with following environments.

a part of application.conf

kamon.metric {
  disable-aspectj-weaver-missing-error = true

  filters {
    akka-actor {
      includes = [ "**" ]
      excludes = [ "*/system/**", "appname/user/io-http/**" ]
    }
    trace {
      includes = [ "**" ]
      excludes = []
    }
    akka-dispatcher {
      includes = [ "**" ]
      excludes = []
    }
  }
}

Statistics via jvisualvm. java_visualvm

Heap dump via jvisualvm. java_visualvm java_visualvm

I think it caused by spray-client or ActorCell.

My program used hostConnector in spray-client.
Problem occured when it used. My test running mode that not occured because spray-client doesn't used. (I used mock client)

Do you have any solutions or ideas? I did disable metrics of akka-actor now. (akka-actor.includes = [])

Sorry I have poor skill of English. (I am Japanese speaker.)

nilsga commented 9 years ago

I think this is similar to an issue we are seeing. Have you been able to find the cause, or have you put it on hold?

szimano commented 9 years ago

I am experiencing a very similar problem using aspectjweaver 1.8.6. any ideas?

ivantopo commented 9 years ago

@tsukaby thanks for sharing this! As @nilsga mentioned, this probably is related to the issue he fixed with #229, given the fact that your dump shows that you have tons of min max counters in memory it is almost certain that the issue is related to that.. so, cold you please try the very latest snapshot that contain that fix? That would be 0.4.1-2ce80225a5ae4d0ed7ba53be1ebe0734d4e4bca6 from our snapshots repo at http://snapshots.kamon.io, hope that the issue is gone there!

tsukaby commented 9 years ago

Thank you fix and build! I'm going to try it on this week.

I will contact you again later.

tsukaby commented 9 years ago

Thank you @nilsga ! Tnank you @ivantopo !

This problem solved on our system. (use 0.4.1-2ce80225a5ae4d0ed7ba53be1ebe0734d4e4bca6)

before (0.4.0)

java_visualvm_before

after (0.4.1-2ce80225a5ae4d0ed7ba53be1ebe0734d4e4bca6)

java_visualvm_after

Thanks a lot!

jozic commented 9 years ago

@ivantopo when do you guys plan to release 0.4.1 with this fix?

ivantopo commented 9 years ago

It should happen really, really soon. I find it weird that not too many people complained, yet 0.4.0 is by far the most downloaded version we have! If possible, please use the snapshot now and then switch to 0.4.1 when it becomes available!

tsukaby commented 9 years ago

I'm also waiting for the next release. However, we are satisfied with the snapshot for now. (We are used it in production environment. :yum: )

Thank you!

jozic commented 9 years ago

thanks! I saw similar issue with lots of memory consumed after we started app with Kamon, but did have a chance to verify the exact reason. Will try snapshot and crossing fingers for the soon release :)

dodnert commented 9 years ago

FYI - My project was impacted by this issue as well. Using version 0.4.1-2ce80225a5ae4d0ed7ba53be1ebe0734d4e4bca6, the problem was resolved.