Add API to allow tuning of connection pool

[tleyden]

As reported in https://github.com/couchbase/sync_gateway/issues/842, we're seeing situations where we have lots of goroutines in Sync Gateway that appear to be blocked waiting for connections from the pool. We're also seeing huge discrepancies in reported round trip times by go-couchbase vs Couchbase Server, probably because that time includes waiting to get a connection (and as mentioned, goroutines are blocked waiting for connections from the pool).

We're still in the root cause analysis phase, but I have some questions on the connection pooling in go-couchbase:

• What is the current connection pool size defaulting to?
• Are any known "best practices" regarding what the optimal pool size is?

/cc @steveyen @adamcfraser

[maniktaneja]

Hi Traun,

No we haven't really done any performance tuning with regards to the connection pool sizing, so I am afraid that there are no reference numbers or best practises regarding that.

From your use-case it does seem like you would need to increase the number of connections per host in the pool. Currently this number is set to 4. See pools.go:27

// PoolSize is the size of each connection pool (per host). var PoolSize = 4 // PoolOverflow is the number of overflow connections allowed in a // pool. var PoolOverflow = PoolSize

You could try increasing that number and let me know if it helps. I'd imagine in the sync-gateway scenario since the server is remote the round trip time would be relatively high so you'd need more cached connections between the client and the server.

There is no API to increase that number, I will implement that as part of this ticket.

[tleyden]

@maniktaneja ok thanks. I changed the title of the ticket accordingly.

Also, a snippet from @ingenthr via email that is relevant to this discussion:

me: how should smartclients do connection pooling? how many connections can a couchbase cluster handle?

Matt: This will vary a bit based on the client’s multi-processing model, but in general we aim to use as few connections as possible. K/V accepts up to 9000/node without any tuning. Views and configuration handle many connections, but can get very bad in memory usage with many concurrent requests.

me: how much increased throughput would clients expect by doing bulk gets vs one-at-a-time gets.

Matt: The tradeoff here is generally between latency and throughput. I’m assuming you’re generally talking about pipelining gets, which is the best ‘bulk’ method. We expose that a bit in libcouchbase, but otherwise library users don’t see it other than tune-ables. You will get higher throughput with a higher average latency if you pipeline. In general, we do this since the latencies aren’t that much higher and the throughput and network usage are better benefits. It requires more work in the library implementation, of course.

[ingenthr]

Note there are some failure cases you have to be careful with. For instance, 'core switch failure' can cause a large number of client systems to reconnect and as far as the cluster is concerned, the other connections (now half open) can still be valid. The cluster doesn't always defend itself well, so it's best to be as efficient as possible on the client side.

[tleyden]

@maniktaneja can you explain what this parameter means?

var MaxIdleConnsPerHost = 256

[maniktaneja]

That's the maximum number of idle connections that golang HTTP client will maintain. HTTP connections in go-couchbase are mainly used for querying view indexes and getting cluster configuration. Doesn't relate to your original connection pool issue.

[maniktaneja]

Fixed with this commit : https://github.com/couchbase/go-couchbase/commit/e0dd36705313cb8ce2157609b6307f72417510e5