Remove stale (closed) connections from pool when trying to allocate a new connection

Currently with the current implementation, if the connection pool encounters an error in one of the requests while trying to process, it closes the connection:

connection.go:

func (c *Connection) processResponses() {
    readRequests := make([]tokenAndPromise, 0, 16)
    responses := make([]*Response, 0, 16)
    for {
        var response *Response
        var readRequest tokenAndPromise
        var ok bool

        select {
        case respPair := <-c.responseChan:
            if respPair.err != nil {
                // Transport socket error, can't continue to work
                // Don't know return to who - return to all
                for _, rr := range readRequests {
                    if rr.promise != nil {
                        rr.promise <- responseAndCursor{err: respPair.err}
                        close(rr.promise)
                    }
                }
                readRequests = []tokenAndPromise{}
                c.Close()
                continue
...

However when allocating new connections for the pool in the pool.go conn() function, it only allocates it if the connection is nil (which is what this PR changes), but having closed connections in the pool essentially renders them useless as in connection.go:

func (c *Connection) Query(ctx context.Context, q Query) (*Response, *Cursor, error) {
    if c == nil {
        return nil, nil, ErrConnectionClosed
    }
    if c.Conn == nil || c.isClosed() {
        c.setBad()
        return nil, nil, ErrConnectionClosed
    }
...

any queries attempted with the connection now return ErrConnectionClosed.

The use case here is the database is currently down, so we poll for it to come up, however each of these requests ends up causing the connection to close, and then the pool is full of closed connections.

rethinkdb / rethinkdb-go

Remove stale (closed) connections from pool when trying to allocate a new connection #482