search

neumino / rethinkdbdash

An advanced Node.js driver for RethinkDB with a connection pool, support for streams etc.
MIT License
848 stars 109 forks source link

question about pools #394

Closed Koyamie closed 3 years ago

Koyamie commented 5 years ago

Hello, I would like to start using rethinkdbdash but was wondering why it was not working when creating lot of pools (like 74) ? I am trying to use it for a Discord bot that has 74 shards so 74 instances but I am getting this error :

ReqlDriverError: None of the pools have an opened connection and failed to open a new one.

What could I do to still use rethinkdbdash for this amount of instances ?

AaronRobinson commented 5 years ago

It's hard to say what the cause is without knowing more - but I use the RethinkDBDash driver successfully.

I can tell you two things that might help.

The error above suggests the pool/connections have close. A way to investigate this is to log as connections that are created and available using the poolMaster.on('size') and size-available events. You can then see in relation to your code what's happening with the connections.

A second thing I can tell you is by coincidence I've just resolved an issue where I was creating too many pools using the default buffer and max size. Buffer is set to 50 - so 50 get created very quickly at the start. I discovered that if I ramped up too many pools too close to each other RethinkDB couldn't cope (the TCP accept queue overflowed and this caused the SYN queue to overflow too - if you don't know what this is I can explain further). The upshot is that RethinkDB would fail to accept established connects as quickly as they were being created. However, RethinkDBDash should recover through retrying, but if you have so many processes trying to grab 50 connections per pool (assuming defaults) then I can see that it would struggle and may even timeout. My solution was to drop the buffer size down to 5 so that collections above that are created organically.