Publish performance is very slow

[prystupa]

We have severe performance problems with node-amqp when under load in production. To reproduce I created this little publisher program:

var amqp = require('amqp');

var conn = amqp.createConnection({url: "amqp://guest:guest@localhost:5672"});
conn.on('ready', function(){

    for(var i = 0; i < 50000; i++){
        conn.publish('labs', {my: "paylaod" + i});
    }

    console.log('published ' + i + ' messages');
});

On the other end of RabbitMQ this little program reads messages as they are published and display some timing statistics:

var amqp = require('amqp');

var conn = amqp.createConnection({url: "amqp://guest:guest@localhost:5672"});

conn.on('ready', function(){
    var n = 0;
    conn.queue('labs', function(queue){
        var prev = new Date().getTime();
        queue.subscribe(function (message, headers, deliveryInfo) {
            n++;
            if(n % 1000 === 0){
                var next = new Date().getTime();
                console.log('read ' + n + ' messages in ' + (next - prev) + " ms");
                prev = next;
            }
        });
    });
});

As you can see the test run results show that performance is significantly deteriorated as more and more messages are published:

read 1000 messages in 352 ms
read 2000 messages in 395 ms
read 3000 messages in 418 ms
read 4000 messages in 500 ms
read 5000 messages in 459 ms
read 6000 messages in 479 ms
read 7000 messages in 469 ms
...
read 46000 messages in 2778 ms
read 47000 messages in 2767 ms
read 48000 messages in 2818 ms
read 49000 messages in 2814 ms
read 50000 messages in 2847 ms

I re-wrote the publisher program in Python using pika, for comparison:

import pika

def main():
    connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
    channel = connection.channel()

    for i in range(50000):
        channel.basic_publish(
            exchange='',
            routing_key='labs',
            body='{"my": "payload"}')

    print " [x] Sent " + str(i) + " messages"

    connection.close()

if __name__ == '__main__':
    main()

As you can see the test run with pika shows much better performance that stays constant for the whole run:

read 1000 messages in 232 ms
read 2000 messages in 235 ms
read 3000 messages in 221 ms
read 4000 messages in 223 ms
read 5000 messages in 227 ms
read 6000 messages in 224 ms
read 7000 messages in 225 ms
...
read 46000 messages in 237 ms
read 47000 messages in 218 ms
read 48000 messages in 247 ms
read 49000 messages in 225 ms
read 50000 messages in 229 ms

Would you be able to look into why the difference in performance between two technologies is so significant and if this can be addressed? Thanks!

[cpsubrian]

Would love more action on this problem. Has performance been an issue for others? Any app or OS level tweaks to improve response time?

[bergman]

@prystupa did you ever solve this problem? I think it might be related to #57. Did you watch the memory usage in both the javascript and python publisher? I did and found that the javascript version went up from ~40 mb to ~400 mb while the python version stayed at ~15 mb throughout.

[life0fun]

I second this issue....I have python version and node version. I can only do about 400 publishes per second with this lib while python version is faster. The avg size of my json string is about around 200 bytes. Do we ever have any performance benchmark anywhere ?