search

Foo-Foo-MQ / foo-foo-mq

Abstractions around RabbitMQ
MIT License
48 stars 24 forks source link

Performance issue with rabbit.request function caused by request which aren't answered #24

Open pascaltozzi opened 3 years ago

pascaltozzi commented 3 years ago

I've modified the demo project by forking v5.0.0 into https://github.com/pascaltozzi/foo-foo-mq Inside publisher.js on the first line I have a flag const TESTING_REQUEST = true;

Set it to false to test publish and set it to true to test request. One of the main issue detected with request are with server having overwhelming amount of messages coming in and have a delay before answering cause a big performance issue.

If we send 30,000 messages. If the first 10,000 messages are being answered 20 seconds later. The next 20,000 messages are being answered right away but are sent right after the first 10,000 messages that are delayed.

This create an overwhelming lag. After 40s, we receive the first 500 answers back and the rabbitmq raise the event of unreachable soon after.

Doing the same example using publish instead of request cause the subscriber to receive all 30,000 messages in 8 seconds and the publisher to receive all answers back within 22 seconds (there is a 20s delay for the first 10,000 messages before sending back those answers).

The process hanging and the rabbitmq raising the unreachable events tend to point to a performance issue during high load caused by messages that have a delay and aren't answered right away.

The short version of my log from running the modified demo:

node publisher

node subscriber

If we send 500 messages, get the answer of those 500 messages right away and loop, we don't see any performance issue.

pascaltozzi commented 3 years ago

Issue seem to be coming from postal.js

.\foo-foo-mq\nodemodules\postal\lib\postal.js Line 537 ` .each( this.subscriptions[ channel ], function( candidates ) { _.each( candidates, cacherFn ); } );`

It loop via all responses and for all of them, it compare them... Thus if we have 10000 request that haven't been answered, for every newer request or answer back, it loop all the request to find which one match our request. Line 409 return function( subDef ) { var cache; if ( _config.resolver.compare( subDef.topic, topic, headers ) ) {

It use the compare method Line 309 compare: function compare( binding, topic, headerOptions ) {

zlintz commented 3 years ago

@pascaltozzi That is interesting. Are you suggesting this is a problem with postal then? Have you reached out to the owner of the postal.js package? I would be curious to see if there is already an open issue for this or something similar.

pascaltozzi commented 3 years ago

@pascaltozzi That is interesting. Are you suggesting this is a problem with postal then? Have you reached out to the owner of the postal.js package? I would be curious to see if there is already an open issue for this or something similar.

I replaced postal with this quick and dirty handler.js https://github.com/x2omedia/foo-foo-mq/blob/master/src/handlers.js

I didn't contact postal since their source code enable to use REGEX in the list of topic to subscribe for... Which in our case, we only use 1-1 relationship of topic thus we don't need to loop all the item of the queue but simply look into a map/dictionary if it exist or not.

Most likely a mix mode of having a queue for REGEX and a map for subscription that are 1-1 relationship might give them a big boost in performance when there multiple message not answered. That being said, I am not aware of all their use-case and it might break some of their features.

https://github.com/postaljs/postal.js/ their project has 0 activity in the past 4 years...