Many redis connections

[intech]

If we are talking about high performance, then a large number of connections create locks at the redis level, and in our case, these are 15 connections per 1 Producer and 1 consumer: Consumer - 11 connections Producer - 4 connections

I researched the code and saw no reason to use that many connections. For a producer, 1 connection is enough. For the consumer, 3 connections are enough, one for the main process and two for the forks.

[weyoss]

If we are talking about high performance, then a large number of connections create locks at the redis level

Not true. First of all, a large number of connections is not the reason behind database locking issues. It does simply tell you that you have a problem somewhere in your code and that your should handle it, maybe a concurrency issue. In general, when dealing with locking issues (no matter at which level), looking at the number of requests (connections) is useless. Your should investigate the matter of your requests and figure out WHAT they are doing.

By the way, have you found any locks? I would be grateful if you could show me at least one issue in this regard.

[weyoss]

For a producer, 1 connection is enough. For the consumer, 3 connections are enough, one for the main process and two for the forks.

Well, if you are confident enough about your early statements, I highly encourage you to start your own project and propose it to the open source community, to be used in production-ready environments. I mean, go ahead! )

[weyoss]

The number of connections for both the producer and the consumer is JUSTIFIED.

Let me make it clear.

By design, the RedisMQ is composed of many layers. Because we are talking about high performance (c), the main requirements for the requests to the Redis server of each layer are:

1. Requests should be executed as fast as possible
2. Requests from one layer should not block/delay the requests from other layers.

Considering the requirements, the best solution would be if each layer could operate on its own connections to Redis. Of course, new connections to Redis are made only when needed. For managing distributed locks, RedisSMQ uses the redlock package, which also maintains its own Redis connections (see https://www.npmjs.com/package/redlock).

Here is the minimum number of connections needed for a producer :

1. A connection for publishing messages
2. A connection for publishing statistical data (stats can be disabled)
3. A connection for the scheduler
4. A connection for the lock manager which is used by the scheduler

The number and the purpose of the connections used by a consumer instance can be verified the same way. Consumers are more complex than producers, and include more layers (gc, heartbeat, etc.).

RedisSMQ has been rigorously tested before being used in production at large scale for many projects that I developed. When benchmarking RedisSMQ, the numbers speak for themselves.

[weyoss]

@intech In anyway, thank you for your feedback

[intech]

Why create another product when it is possible to improve existing ones ;) Thanks for the detailed explanation of the design. I will do a detailed analysis to confirm or refute my statements about connections and locks in redis (single thread). I will also do a comparative test with keydb (multi-thread redis) and share the results in this issue.

[weyoss]

You're welcome!

If you find any issue please feel free to report it.