search

trinidad / trinidad_init_services

Run Trinidad as a service (based on Commons-Daemon and JRuby-JSVC)
Other
28 stars 10 forks source link

Memory usage grows over -J-Xmx? #42

Closed mrbrdo closed 10 years ago

mrbrdo commented 10 years ago

When using ps, I can see that jsvc-trinidad has the argument -Xmx500m. However, after a few days of running, my Trinidad uses over 1GB of RAM and continues to grow to the point when it runs out of RAM and then I start getting out of memory errors in my app. Any idea why this happens and how to prevent it? My jsvc-trinidad starts with using about 350MB RAM and then gradually builds up to over 1.5GB.

kares commented 10 years ago

might be a leak (e.g. if you're doing hot-deploys) ... the default sizes depend on JVM vendor/version and system architecture ... so it's hard to tell exactly, is it a heap or perm-gen out of memory error ?

... you can do a java -XX:+PrintFlagsFinal to find out the calculated default perm-gen size used (also take a look at https://github.com/trinidad/trinidad/wiki/JOPTS#memory to limiting perm if that is the issue)

mrbrdo commented 10 years ago

The error was Java::JavaLang::OutOfMemoryError: Java heap space. It is always mentioning heap space (after this happens, it seems all subsequent requests fail with heap errors, until restart). I am doing hot deploys, but I'm not sure that I did one since last restarting. In any case it does not happen very soon after a deploy.

java -XX:+PrintFlagsFinal

uintx MaxPermSize                               = 67108864        {pd product}
uintx PermSize                                  = 12582912        {pd product}

from ps aux (actual parameters to jsvc contain -Xmx500m):

jsvc-trinidad -home /usr/lib/jvm/java-7-openjdk-i386/jre -wait 20 -pidfile /home/project/project/shared/pids/trinidad.pid -procname jsvc-trinidad -jvm server -outfile /home/project/project/shared/log/trinidad.log -errfile &1 -cp /home/project/project/releases/46/vendor/bundle/jruby/1.9/gems/trinidad_init_services-1.2.3/trinidad-libs/jruby-jsvc.jar:/home/project/project/releases/46/vendor/bundle/jruby/1.9/gems/trinidad_init_services-1.2.3/trinidad-libs/commons-daemon.jar:/home/project/.rvm/rubies/jruby-1.7.4/lib/jruby.jar -Djruby.shell=/bin/sh -Djffi.boot.library.path=/home/project/.rvm/rubies/jruby-1.7.4/lib/native/arm-Linux:/home/project/.rvm/rubies/jruby-1.7.4/lib/native/i386-Linux:/home/project/.rvm/rubies/jruby-1.7.4/lib/native/x86_64-Linux -Djruby.home=/home/project/.rvm/rubies/jruby-1.7.4 -Djruby.lib=/home/project/.rvm/rubies/jruby-1.7.4/lib -Djruby.script=jruby -Djruby.daemon.module.name=Trinidad -Djruby.compat.version=RUBY1_9 -Xmx500m -Xss2048k -Xbootclasspath/a:/home/project/.rvm/rubies/jruby-1.7.4/lib/jruby.jar -Dfile.encoding=UTF-8 com.msp.jsvc.JRubyDaemon /home/project/project/releases/46/vendor/bundle/jruby/1.9/gems/trinidad_init_services-1.2.3/lib/trinidad/daemon.rb -d /home/project/project/current --config /home/project/project/current/config/trinidad_production.yml
mrbrdo commented 10 years ago

Also I should note that I was running puma before I started using trinidad, and I never had these kind of errors (I used it for over 1 year on production), so it's probably unlikely that I have leaks in my app itself. It just seems to me like it just scales up and up over time and then hits the RAM limit and dies. I don't understand, though, why does the process grow over 1.5gig of RAM when it has -Xmx500m? I thought that option is supposed to limit that.

kares commented 10 years ago

ok, thanks. without looking deeper into why your memory requirements get that big (-Xmx only limits heap size, there's still perm-gen and possibly native memory size as well - but still looks weird to go that high ... maybe try updating OpenJDK 7 to it's latest or installing Oracle JDK 7) here's the deal with hot-deploys. historically there's been issues with Java hot deploys in general and even the JDK did not support class-unloading by default. (please note that this is something entirely different than a reload with passenger/puma - there a new process gets created). JRuby and it's libraries are no exception to this ... unfortunately there's some work-arounds that should work for most ... to test this out you'll need to use the beta (although you might need to try master due an incompatibility with init services).

OpenJDK 7's G1 GC should support unloading by default (thus no further JVM option tuning should be necessary - I'm not 100% sure here ... might depend on the actual build version).

to test things out locally simply run trinidad with some lower settings for memory (1-2 hot-deploys should perform fine) and do hot-deploys as long as you hit a out-of memory issue, than try the beta ... it should not happen. recommended settings :

-Xss1152k -Xmx500m (put this one lower while testing locally e.g. 300M) -XX:MaxPermSize=120M (put this one lower while testing locally e.g. 80M)

kares commented 10 years ago

closing as an invalid ... would really need to see what's going on in detail to confirm there's an issue. anyways 1.3.0 supports configuring memory "requirements" - somehow optimistic, and we always advise to make your memory setting based on how the application is doing, but still better than none. thanks for reporting, let us know if the issue persists!