Memory usage keeps growing when using remote transport

[salaerts]

We are currently building a distributed application that uses the built-in http transport. To simplify the scenario: we have van API (service A) that takes requests from clients and a background service that stores/fetches data (service B).

When we put high load on this system the memory usage grows fast and the claimed memory is never released. Both service A & B have this problem, although service A (the client of B) has much higher memory growth when compared to service B. (Sending 15.000 requests to service A from 4 separate processes makes the memory usage go from 80MB to 500+MB in a couple of minutes.)

When stripping down the code we managed to pin-point the issue to seneca.act calls. So we started on a repro project to simplify this.

First we made a small app that just used seneca.act to call a local plugin.

var seneca = require('seneca')();

seneca.ready(function(err) {
  seneca.add({ role: 'test', cmd: 'run' }, function(msg, respond) {
    console.log('received call, replying to msg # ' + msg.count);
    respond(null, { reply: 'OK ' + msg.count });
  });

  executeAct(1);
});

function executeAct(count) {
  if (count > 50000) {
    return;
  }

  var pattern = { role: 'test', cmd: 'run', count: count };

  seneca.act(pattern, function(err, result) {
    if (err) {
      console.error(err);
      process.exit();
      return;
    }

    console.log(result);
    count = count + 1;
    executeAct(count);
  });
}

We can see some minor memory usages spikes while running but the memory seems to be properly managed. (Starts at about 80Mb, spikes to 101Mb but drops down to 80-90Mb again while running.)

Next we made a small example that does the exact same thing but using the http transport. Client code:

var seneca = require('seneca')();

seneca.ready(function(err) {
  seneca.client({
    type: 'web',
    host: 'localhost',
    port: 4455,
    pin: 'role:test'
  });

  executeAct(1);
});

function executeAct(count) {
  if (count > 10000) {
    return;
  }

  var pattern = { role: 'test', cmd: 'run', count: count };

  seneca.act(pattern, function(err, result) {
    if (err) {
      console.error(err);
      process.exit();
      return;
    }

    console.log(result);
    count = count + 1;
    executeAct(count);
  });
}

Server code:

var seneca = require('seneca')();

seneca.ready(function(err) {
  seneca.add({ role: 'test', cmd: 'run' }, function(msg, respond) {
    console.log('received call, replying to msg # ' + msg.count);
    respond(null, { reply: 'OK ' + msg.count });
  });

  seneca.listen({
    type: 'web',
    host: 'localhost',
    port: 4455,
    pin: 'role:test'
  });
});

When running this sample the memory usage keeps rising and never drops. We also tried changing the transport to seneca-amqp-transport with the exact same results.

We tried this with: NodeJS 4.4 Seneca 1.2.0, 1.3.0, 1.4.0 and 2.0.1

We tried all of these seneca versions and the problem seems to exist everywhere. Which begs the question: are doing something wrong here?

[mcdonnelldean]

via, https://github.com/senecajs/seneca/issues/408

• needs test added to suite to reproduce

[salaerts]

I'll also post the first replies I got when I created the issue in the main senecajs repo:

@mcollina wrote: Just to confirm, is your process crashing for out of memory? Can you try running this long enough to get to this point?

Memory spikes under high traffic are normal in node apps under high load, as V8 runs the collector when it has time, i.e. when the load is lower. There techniques to mitigate this, but we need to ensure that we are in this case rather than a memory leak.

[salaerts]

I'll do some additional tests later today, but I actually did one test a couple of times where I would put the server under high load until memory usage got above 500Mb and then stopped all requests. After 15 minutes of idle time the memory still wasn't released but I'll double check this.

I will also provide a better example to test that with a node server that keeps listening.

[salaerts]

Ran another test against seneca 2.0.1 last night. Memory filled up to about 930Mb when I stopped the load and then let it sit idle for an entire night. By the morning memory usage had dropped to about 893Mb.

[mcollina]

@salaerts that is not what I am saying, and what I asked. Can you run your load test forever, up until we are certain that the process do not crash due to "out of memory" errors: depending on your host memory, it can even reach some GBs under load.

You can also force garbage collection by using --expose-gc, and calling gc(true) from time to time.

If we are allocating memory that is never collected is a kind of bug, if we are allocating memory that can be collected but V8 defers collecting it is another. The techniques to solve the problem are radically different.

[salaerts]

Sorry, should have been more explicit on that: running the load test forever was my next test but I had some other stuff to do first.

In the meanwhile I had a chance to keep the load running forever and apparently the memory gets freed as soon as it hits about 1.5Gb.

I do find this behavior kind of strange: I would expect that memory would be freed as soon as the load stops since that should give the collector plenty of time but that doesn't happen. But is does free up memory during the high load once it reaches a certain amount of used memory.

Anyway, seems like it's not a memory leak issue in seneca. One thing I seem to have noticed is that seneca has become a bit more memory consuming since version 1.2.0. (It takes way longer in version 1.2.0 to reach the same memory consumtion vs running in 2.0.1.)

[mcollina]

As I suspected. The answer is that currently seneca allocates a lot of objects and functions to work. Reducing the overhead is part of our roadmap.

[mcdonnelldean]

Going to mark this as closed for now. @salaerts we are aware that Seneca is greedy. It's not leaking but it can be a hog. As part of v5 we are looking to reduce this consumption across the core and plugins, that should bring the base usage down.

[CelsoSantos]

I'm sorry to reopen this but I really must ask, what are the expected values here when deploying and application into production?

If seneca can take up to 1.5GB per service (even in a worst-case scenario), then I'm in trouble because in the application I'm developing right now I currently have 10 services and I'm probably going to grow up to 15/16 in the next month, so I'm basically up to 15GB of RAM right now + another 7.5GB in a mont's time.

I've also noticed that my services while running "idle" use around 200MB of RAM upon start and even that I find excessive.

When taking into consideration a host where to host the application (let's say OpenShift or Google Cloud Engine) having to account with a maximum of 24GB of RAM just for the application seems a bit excessive.

What can be done to account for this? Should we just be aware that might happen or are there any techniques that can be applied to reduce the memory footprint?