Reducing memory use?

[tetherit]

Hi there :)

Eye is fantastic and I'm enjoying using it very much, however it takes up more memory than my rails app, 9.6% of the available 2GB of ram :( - any advice/tips for reducing the memory footprint?

[kostya]

I think i optimized all what i can, in memory usage. i can't reduce it more, this how ruby threads work.

in my experience:

• eye eat > 100Mb when monitor > 100 processes
• with 1.9.3 eye used less memory, than with 2.0
• with celluloid 0.15 eye used less memory (0.6.4), than with 0.16 (0.7.pre).

normal eye 0.6.4 with ruby 1.9.3 and with 6 processes, eats 45Mb for half of year.

[tetherit]

I must be doing something wrong.

On one server, using eye v0.6.4 and ruby 2.2, it's currently the most memory hungry process on the system. It is taking up 4.3% of 4GB of ram or 169MB. More than Puma and other ruby processes.

This is monitoring 29 processes, in 7 groups.

Any tips specific with Ruby 2.2 to reduce this? -- I'm going to dig around in my eye file to see if I am introducing a leak anywhere.

[kostya]

try also 1.9.3, or may be some leak in your config

[tetherit]

It seems eye leaks memory when trying to restart a process multiple times (in my case endlessly because that camera is currently offline).

  trigger :transition, to: :starting, do: -> {
    unless _check_camera_online(url)
      info "Waiting for camera #{url} to be up first"
      not_ready = :'Camera offline, retry in 30 seconds'
      stop = 'Stop transition because process if not ready to start.'

      process.switch :unmonitoring, Eye::Reason.new(not_ready)
      process.schedule_in 30.seconds, :start, Eye::Reason.new(:"retrying...")
      raise Eye::Process::StateError.new(stop)
    end
  }

Currently, eye is already at 5.7% since my last message :(

[kostya]

i tried run this transition for 2 days, no leaks, maybe something other leaks, method _check_camera_online of something.

btw in 0.7.pre, this can be written as, which is doing the same:

trigger :starting_guard, every: 30.seconds, should: -> { _check_camera_online(url) }

[tetherit]

The _check_camera_online method just runs this code:

system('nice -n19 ffmpeg -loglevel warning -y -stimeout 15000000 ' \
      "-i '#{url}' -c:v copy -an -vframes 1 -f rawvideo /dev/null")

Thank you for the tip, I updated my code to use the new cleaner syntax.

I have one other check which is:

def process_started?(name)
  p = Eye::Control.process_by_full_name(name)
  p && p.state_name == :up
end

def wait_for_process(name)
  trigger :starting_guard, every: 30.seconds, should: -> { process_started?(name) }
end

I'm using latest trunk.

I started eye just before, waited 10 minutes, it was at 4.0% memory, monitoring 29 processes, in 7 groups. After 30 minutes or so, it is now at 5.3% (puma/rails sit at 4.2% - of 4GB of ram). I will follow up in the next few hours to report if it grows anymore.

One thing I noticed was Puma leaks memory with Ruby 2.2 (there is a bug report for this), but no leaking with Ruby 2.0, I will re-run this experiment with Ruby 2.0 soon.

[tetherit]

It has grown to 5.5% in an hour :( - I am going to try with Ruby 2.0 and see if that solves the problem.

[kostya]

was this problems always, or appears recently?

[tetherit]

It started when upgrading to Ubuntu 14.04 last year - something is leaking memory on the server - but even after killing just about every process - the memory seems to be "gone" - full details here: http://superuser.com/questions/793192/what-is-using-up-all-my-memory-ubuntu-14-04-lts-server

So trying to eliminate any possible causes.

[tetherit]

ok, wow! - Going from 2.2.2 to 2.0 dropped eye's memory use on boot from 4% down to 2% - it now takes less memory than Puma/Rails also! -- I will report back if that increases over time.

[tetherit]

1 hour later, no memory leak with Ruby 2.0 - it is only leaking with 2.2.2, hmmmm

[kostya]

solved?

[tetherit]

No, something is still leaking :( - but not sure if it's eye or not - it's not showing what is taking memory in top :/

[tetherit]

It seems the problem was related to the 3.13 kernel, after upgrading to 3.19 the problem is gone :) - closing.

But saying that, are there any tips to reduce memory use with Ruby 2.2? - It seems double the memory is used in 2.2 compared to 2.0 :(

[marshall-lee]

Maybe it's not a leak but bloat?

I tried to reproduce a leak with this example:

Eye.config do
  logger '/tmp/eye.log'
end

Eye.application 'crasher' do
  working_dir File.expand_path(File.dirname(__FILE__))
  20.times do |i|
    process "crasher#{i}" do
      start_command './crasher'
      daemonize true
      pid_file "./crasher#{i}.pid"
    end
  end
end

And the crasher is a simple C program:

#include <unistd.h>
#include <stdlib.h>

int main() {
  unsigned long x = rand();
  usleep(500 * (x % 10));
}

So this is the application that restarts everytime.

I profiled Eye's ObjectSpace - it constantly bloats but in one moment it stops. The reason is that Eye maintains a process history that is maximum to 100 elements. When process history reaches 100 elements, Eye's memory consumption becomes constant.

Maybe you mistaken this thing with a leak?

[kostya]

process history its a little array, it should be small. also, not need to test c program like this :), you can just start_command "sleep #{rand}"

[marshall-lee]

Sure :)