search

mcollina / mqemitter-mongodb

MongoDB based MQEmitter
MIT License
18 stars 10 forks source link

this._waiting keeps growing #9

Closed teomurgi closed 6 years ago

teomurgi commented 6 years ago

We are using mqemitter-mongodb inside Aedes.

We started noticing that in a high-throughput scenario not all the messages were published (even with QoS 1!).

With "high"-throughput I mean 6000 messages in 20 seconds (not so high); broker, client, mongodb running on the same machine. No mqtt-client subscribed, it takes just 1 subscriber to make the issue disappear; hence this really seems to be some sort of fine-grained concurrency/event-loop problem.

We tracked the issue down to mqemitter-mongodb and discovered that the object _waiting keeps growing (at a slow rate, like 1 message over 6000, but it is not a constant). When it leaks messages, the related callback does not get fired and the publish does not occur.

In order to make evidence of the behaviour we added a log line here:

mqemitter-mongodb.js line 128 console.log("------------- _waiting", Object.keys(that._waiting).length)

And, as already said, we noticed that the size of the _waiting object is constantly increasing.

mcollina commented 6 years ago

Can you upload your example to reproduce?

teomurgi commented 6 years ago

Thanks @mcollina for the reply.

You can find our tester tool here https://github.com/teomurgi/aedes-tester in order to reproduce the problem.

Keep in mind that the effect is more noticeable on slow/limited resources machines.

On a VM with ubuntu it's very noticeable, on my late-2014 macbook pro it loses 1 message over 100000, on a brand new macbook or a NASA-worth PC it can be very difficult to debug.

For this reason I suggest using a VM or run it in a container (dockerfiles are provided in the linked repo).

One additional note: the bug does not ever happen with redis adapter, confirming that is an issue with mqemitter-mongodb.

Thank you.