Memory usage in jruby under high load

GoogleCodeExporter commented 9 years ago

Config:

input {
  stdin { 
    type => "foo"
  }
}

output {
  stdout { }
} 

---

After shpiping in a few thousand events, quickly, memory usage gets up to 
400-500mb at which point java/jruby start doing aggreessive GC, and eventually 
it dies exits code 1 with no error message.

jmap -histo output:

 num     #instances         #bytes  class name
----------------------------------------------
   1:       1498246       83901776  org.jruby.RubyString
   2:        512791       77903760  [Lorg.jruby.runtime.builtin.IRubyObject;
   3:       1144161       73226304  org.jruby.RubyHash$RubyHashEntry
   4:        962203       53883368  org.jruby.util.ByteList
   5:        969937       45393160  [B
   6:        513009       32832576  org.jruby.RubyArray
   7:        180358       20201024  [Lorg.jruby.RubyHash$RubyHashEntry;
   8:        180356       20199872  org.jruby.RubyHash
   9:         33657        5239592  <constMethodKlass>
  10:        100897        4956992  [Ljava.lang.Object;
  11:        180388        4329312  java.util.concurrent.atomic.AtomicInteger
  12:        180358        4328592  org.jruby.RubyHash$33
  13:        180358        4328592  org.jruby.RubyHash$34

Original issue reported on code.google.com by jls.semi...@gmail.com on 22 Feb 2011 at 4:20

GoogleCodeExporter commented 9 years ago

snack(~) !1! % sum 3 < /tmp/mem
709628896
snack(~) % sum 3 < /tmp/mem | bytes
676.75M

Original comment by jls.semi...@gmail.com on 22 Feb 2011 at 4:26

Attachments:

mem

GoogleCodeExporter commented 9 years ago

Swapped out Queue.new with SizedQueue.new(10) - memory usage is much better now 
and doesn't grow mostly-unbounded.

Original comment by jls.semi...@gmail.com on 22 Feb 2011 at 5:03

GoogleCodeExporter commented 9 years ago

Original comment by jls.semi...@gmail.com on 25 Feb 2011 at 11:24

Changed state: Fixed

ankiit / logstash

Memory usage in jruby under high load #44