neumino / rethinkdbdash

An advanced Node.js driver for RethinkDB with a connection pool, support for streams etc.
MIT License
848 stars 108 forks source link

Performance issue - too many TCP connections #335

Open lprhodes opened 7 years ago

lprhodes commented 7 years ago

Hi,

I'm at a bit of a dead end with a performance issue.

I started out with a connection pool but it kept running out and I ended up needing 10k connections in the pool.

I then changed the code to create a single connection with rethinkdbdash (pool turned off) then I pass the connection too any bit of code that runs a query on rethinkDB. Unfortunately running the command lsof -i -n | wc -l on the rethinkdb server is showing that it still has 1300 open connections with barely anyone using the application. It seems that a TCP connection is being opened for every query but then not being closed.

I had hoped to just open a single connection and then query everything through that.

Here's how I have things setup:

  host: process.env.RDB_HOST || '127.0.0.1',
  port: process.env.RDB_PORT || 28015,
  password: process.env.RDB_PASSWORD,
  pool: false
}

export default rethinkdbdash({
  servers: [ server ],
  db: process.env.RDB_DB_NAME || 'database'
})```

const rDBConnection = await rDB.connect(server)

async function doSomething (rDBConnection) { const a = await rDB.table('comment').getAll(activity.id, { index: 'activity' }).run(rDBConnection) const b = await rDB.table('taskQueue').insert(task).run(rDBConnection) const c = await rDB.table('device').getAll(user.id, { index: 'user' }).run(rDBConnection) }

doSomething(rDBConnection)



Any ideas?

Thanks!
neumino commented 7 years ago

Rethinkdbdash is using one connection per query. By default it will open up to 1000 connections per server - that is if you have at some time in the past 1000 concurrent queries.

I would need a bit more detail to understand your problem. What's the actual problem? Having thousands of TCP connections on a server is fine.