I generally recommend around 5 Channels to be built on top of each connection
I have consumer worker, it consumes from 20 to 100 messages/second. I set 1 max connection and 5 max channels. It can vary from 1 to 4 queries to Mysql on each handled message. To save DB queries I use BigCache to cache user settings, so I call once when a message is received and settings are not in Cache, then until 2 hours later.
When I tested the worker, the mysql connections went crazy to 700 in less than a minute. I tuned the mysl max connections to 10, and another problem appeared, now the message handler was freezing, when the used connections got to 10, every routine was put on idle.
So I tuned the max channels to 2 and problem solved, no more freezes or crazy Mysql connections.
So my question is, Whats the downside about using only 2 max channels?
Thank you
** Update 13/05/2022 **
I read the RabbitMQ docs, especially the section about "Consumer Prefetch" and "QoS", then I realized that there is an option in consumer settings to "override default QoS count", my default number was set to 100. So I was basically saying "Hey RabbitMQ my worker can handle 100 messages at the same time", a basic "hello world" maybe can handle that and much much more, but when you have http requests, a mongo logger, some mysql queries, the performance of the worker dont depend on Go, it depends on the response from those other services.
So, there always 100 go routines at the same time trying to make 100 or more queries to the DB at the same time, and the sql library was putting those routines on a idle. And I say 100 go routines because I said to RabbitMQ that I can handle 100 messages at the same time with the QoS.
After many tests, I had the conclusion that my app can handle maximum 10 messages per second, so I tuned the QoS override setting to 10.
Also I changed the max mysql connections to 5, and set rabbitmq connection to 1 and 5 channels.
After that, the problems went to the past.
Maybe someone can have this problem, and this is the way that I resolved.
Hi, in the readme you recommend this:
I generally recommend around 5 Channels to be built on top of each connectionI have consumer worker, it consumes from 20 to 100 messages/second. I set 1 max connection and 5 max channels. It can vary from 1 to 4 queries to Mysql on each handled message. To save DB queries I use BigCache to cache user settings, so I call once when a message is received and settings are not in Cache, then until 2 hours later.
When I tested the worker, the mysql connections went crazy to 700 in less than a minute. I tuned the mysl max connections to 10, and another problem appeared, now the message handler was freezing, when the used connections got to 10, every routine was put on idle.
So I tuned the max channels to 2 and problem solved, no more freezes or crazy Mysql connections.
So my question is, Whats the downside about using only 2 max channels?
Thank you
** Update 13/05/2022 ** I read the RabbitMQ docs, especially the section about "Consumer Prefetch" and "QoS", then I realized that there is an option in consumer settings to "override default QoS count", my default number was set to 100. So I was basically saying "Hey RabbitMQ my worker can handle 100 messages at the same time", a basic "hello world" maybe can handle that and much much more, but when you have http requests, a mongo logger, some mysql queries, the performance of the worker dont depend on Go, it depends on the response from those other services.
So, there always 100 go routines at the same time trying to make 100 or more queries to the DB at the same time, and the sql library was putting those routines on a idle. And I say 100 go routines because I said to RabbitMQ that I can handle 100 messages at the same time with the QoS.
After many tests, I had the conclusion that my app can handle maximum 10 messages per second, so I tuned the QoS override setting to 10.
Also I changed the max mysql connections to 5, and set rabbitmq connection to 1 and 5 channels.
After that, the problems went to the past.
Maybe someone can have this problem, and this is the way that I resolved.
Thank you!