search

neumino / rethinkdbdash

An advanced Node.js driver for RethinkDB with a connection pool, support for streams etc.
MIT License
848 stars 108 forks source link

Fail to create a new connection for the connection pool. #337

Open Panoplos opened 7 years ago

Panoplos commented 7 years ago

Just trying out the library for the first time, with a simple

> r = require('rethinkdbdash')({db: 'test'})

I am getting the following:

Creating a pool connected to localhost:28015
{ [Function: _r]
  row: { [Function: term] _query: [ 13 ], _r: [Circular] },
  monday: { [Function: term] _query: [ 107 ], _r: [Circular] },
  tuesday: { [Function: term] _query: [ 108 ], _r: [Circular] },
  wednesday: { [Function: term] _query: [ 109 ], _r: [Circular] },
  thursday: { [Function: term] _query: [ 110 ], _r: [Circular] },
  friday: { [Function: term] _query: [ 111 ], _r: [Circular] },
  saturday: { [Function: term] _query: [ 112 ], _r: [Circular] },
  sunday: { [Function: term] _query: [ 113 ], _r: [Circular] },
  january: { [Function: term] _query: [ 114 ], _r: [Circular] },
  february: { [Function: term] _query: [ 115 ], _r: [Circular] },
  march: { [Function: term] _query: [ 116 ], _r: [Circular] },
  april: { [Function: term] _query: [ 117 ], _r: [Circular] },
  may: { [Function: term] _query: [ 118 ], _r: [Circular] },
  june: { [Function: term] _query: [ 119 ], _r: [Circular] },
  july: { [Function: term] _query: [ 120 ], _r: [Circular] },
  august: { [Function: term] _query: [ 121 ], _r: [Circular] },
  september: { [Function: term] _query: [ 122 ], _r: [Circular] },
  october: { [Function: term] _query: [ 123 ], _r: [Circular] },
  november: { [Function: term] _query: [ 124 ], _r: [Circular] },
  december: { [Function: term] _query: [ 125 ], _r: [Circular] },
  minval: { [Function: term] _query: [ 180 ], _r: [Circular] },
  maxval: { [Function: term] _query: [ 181 ], _r: [Circular] },
  nextVarId: 1,
  _Term: [Function: Term],
  _poolMaster: 
   PoolMaster {
     _r: [Circular],
     _line: Dequeue { start: 0, end: 0, buffer: [Object] },
     _pools: { unknownPools: [Object] },
     _healthyPools: [ [Object] ],
     _healthy: true,
     _init: false,
     _index: 0,
     _indexUnknown: 0,
     _discovery: false,
     _options: { db: 'test', buffer: 50, max: 1000 },
     _log: [Function],
     _draining: false,
     _numConnections: 0,
     _numAvailableConnections: 0,
     _hasPrintWarningLocalhost: false,
     _feed: null,
     _consecutiveFails: -1,
     _timeoutError: 1000,
     _maxExponent: 6,
     _seed: 0,
     _servers: [ [Object] ] },
  _options: {} }
> Entering slow growth mode
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to localhost:28015 in less than 20s","message":"Failed to connect to localhost:28015 in less than 20s.","isOperational":true}
Exiting slow growth mode

RethinkDB is running, as I am able to access it via both the dashboard and the rethinkdb package, and the port is default.

I am able to use the resultant r object to query the DB, too.

> r.tableList().run().then( console.dir )
Promise {
  _bitField: 0,
  _fulfillmentHandler0: undefined,
  _rejectionHandler0: undefined,
  _promise0: undefined,
  _receiver0: undefined }
> [ 'consumers' ]

Environment Info

Machine: Intel 64bit OS: Ubuntu 17.04 (Zesty) Node: v7.10.0 RethinkDB: built from git (next branch) rethinkdbdash: 2.3.29

neumino commented 7 years ago

Did you start rethinkdb after starting your nodejs script?

Panoplos commented 7 years ago

No, it is running as a daemon on the system.

Panoplos commented 7 years ago

Any insight into this? This is obviously a show-stopper for us.

Panoplos commented 7 years ago

I tested a slightly different configuration, where I set pool: false, then attempted the following:

import Rethinkdbdash from 'rethinkdbdash'
const r = Rethinkdbdash({pool: false})
r.connect()
  .then(debug.log)
  .error(debug.error)

Running as follows:

⟩ env DEBUG=test node --require 'babel-register' test.js

Results in:

  test
---------------------------------------
ReqlDriverError
Failed to connect to localhost:28015 in less than 20s.
ReqlDriverError
    at Timeout._onTimeout (/home/panoplos/test/node_modules/rethinkdbdash/lib/connection.js:103:12)
    at ontimeout (timers.js:386:14)
    at tryOnTimeout (timers.js:250:5)
    at Timer.listOnTimeout (timers.js:214:5)
---------------------------------------

Note that I have also tested the following:

import r from 'rethinkdb'
r.connect( {host: 'localhost', port: 28015}, (err, conn) => {
  if (err) {
    debug.error(err)
  } else {
    debug.log("Connected successfully!")
  }
})

With the following result:

⟩ env DEBUG=test node --require 'babel-register' test.js 
  test   Connected successfully!
Panoplos commented 7 years ago

More testing:

I have tested this against next, v2.4.x and v2.3.x, and it works only on v2.3.x of rethinkdb, so it seems the driver is not working with v2.4+ versions.

neumino commented 7 years ago

I don't see a 2.4.x release for RethinkDB - Does it work with the official driver?

Panoplos commented 7 years ago

v2.4.x is a branch of the git repo. It is not an official release, but it is stable, as is next. And, yes, the official driver works against these.

neumino commented 7 years ago

Hum, rethinkdb doesn't build on my workstation (next or v2.4.x). That being said, it's probably an issue with rethinkdb though.

I don't see any change in the official driver for 2.4, so it's supposed to use the same protocol (and I don't see any mention of a different API in the changelog)

Panoplos commented 7 years ago

What system are you attempting to build on? I had a small issue with building it on Ubuntu, but it was just a matter of compiling with the correct version of gcc.

BhaskaranR commented 7 years ago

I get same error with rethinkdb docker image package verion - 2.3.5, Initially it connects and then throws the error

{
    servers: [
        { host: '192.168.99.100', port: 32780 },
        { host: '192.168.99.100', port: 32779 }
    ],
    pool: true,
    buffer: 300,
    max: 3000
}

Creating a pool connected to 192.168.99.100:32780
Creating a pool connected to 192.168.99.100:32780
(node:24627) [DEP0016] DeprecationWarning: 'root' is deprecated, use 'global'
{"kind":"notice","notice":"hello seneca cmj0k3p0xwii/1502666893090/24627/3.3.0/-","level":"info","when":1502666895646}
Entering slow growth mode
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to 192.168.99.100:32780 in less than 20s","message":"Failed to connect to 192.168.99.100:32780 in less than 20s.","isOperational":true}
Entering slow growth mode
Fail to create a new connection for the connection pool. Error:{"msg":"Failed to connect to 192.168.99.100:32780 in less than 20s","message":"Failed to connect to 192.168.99.100:32780 in less than 20s.","isOperational":true}
BhaskaranR commented 7 years ago

@Panoplos were you able to fix this?

Panoplos commented 7 years ago

No. I had to downgrade my RethinkDB installation.

neumino commented 7 years ago

@BhaskaranR - you are running the stable version, you probably have a different problem. Please open a new bug instead

dustinrouillard commented 6 years ago

This really needs to be fixed.. I hate the default rdb module.. I love this for the ease of use and would love to continue to use it. But this is preventing me from doing so... This doesn't seem to be a rethinkdb server issue.. This is only happening since updating rdb.

adamierymenko commented 6 years ago

Yes this an issue here as well.

adamierymenko commented 6 years ago

With next.

adamierymenko commented 6 years ago

It looks as if _compareDigest is not being called with newer versions which is never calling clearTimeout to clear the connection timeout timer.

Zalasanjay commented 6 years ago

i'm also getting this same issue i have installed rethinkdb 2.3.5 (GCC 5.4.0) rethinkdbdash 2.3.31,

heathweaver commented 6 years ago

Everything was running well and suddenly this started happening for me as well. rethinkdb 2.3.6~0xenial (GCC 5.3.1)

(restarted my docker container, not rethinkdb, but the application and it was okay).

SatyaVaraprasad1978 commented 6 years ago

I am also facing the same problem, Is it resolved after restarting the docker container?

gamebak commented 6 years ago

Same problem over here. I have 2 containers, one works while the other gets connection error mentioned above and it gets stuck...

I even attempted to try healthy checks as mentioned in the documentation, but those aren't working either. Example of attempts:

r.getPoolMaster().on('healthy', function(healthy) {
  if (healthy === true) {
    console.log('We can run queries.');
  }
  else {
    console.log('No queries can be run.');
  }
});

Another attempt:

//attempt to get connections length and available connections, always seems to be 0
r.getPoolMaster().getLength()

What works is to restart container with a timeout... but that's a bad practice on a big project in production... Any ideas on this? (I downgraded from the recent August update, but it seems I' still having the same issue)

sudomoose commented 6 years ago

Still having this issue too, it makes rethink nearly unusable. I still haven't been able to find a solution. This issue really needs more attention.

gamebak commented 6 years ago

My current solution is to have my own rate limit, because my current architecture allows it. I am reading from the message queue at an interval of time and I kept a close eye when I had timeouts, when that happen I reduced the number of messages read.

I hope this gives someone at least a solution on how to deal with this sort of thing. Another solution that I had in mind was to drain the pool periodically, but you will need to add extra checks in your code, in case if a pool is being drained you will need to not do any query and to wait for that to finish, by default you'll get an error if it's not treated.

ssribeiro commented 6 years ago

My current solution is to have my own rate limit, because my current architecture allows it. I am reading from the message queue at an interval of time and I kept a close eye when I had timeouts, when that happen I reduced the number of messages read.

I hope this gives someone at least a solution on how to deal with this sort of thing. Another solution that I had in mind was to drain the pool periodically, but you will need to add extra checks in your code, in case if a pool is being drained you will need to not do any query and to wait for that to finish, by default you'll get an error if it's not treated.

good solution, but a hell code situation. better dive into the origin of the bug. lots of headaches here

MattMacGregor commented 5 years ago

I was wondering if anyone has figured out what is going on that is causing this bug. I dug around and it does seem like _compareDigest isn't being called. After looking even more only one data event is being sent per a connection, which seems to be causing the time out, as the self.state in connection.js is only reaches 1 then stops, where to execute _compareDigest it needs to be equal to 2.