[BUG] Stale connection in connection pool cannot properly recover itself

[stdc105]

Steps to reproduce the bug:

• Start an aiohttp server, initiate an asynch connection pool (without configuring max/min connection)
• Leave it running for an hour or two
• Run any query by acquiring a connection in the pool
• The server may raise an exception: bytearray index out of range
  • Here is an actual log from my server:

    Error handling request
    Traceback (most recent call last):
    <stack trace from aiohttp, mainly processes the request and assembles the query>
    File "/opt/pyenv/versions/3.9.6/lib/python3.9/site-packages/asynch/cursors.py", line 61, in execute
    response = await execute(query, args=args, with_column_types=True, **execute_kwargs)
    File "/opt/pyenv/versions/3.9.6/lib/python3.9/site-packages/asynch/proto/connection.py", line 526, in execute
    await self.force_connect()
    File "/opt/pyenv/versions/3.9.6/lib/python3.9/site-packages/asynch/proto/connection.py", line 575, in force_connect
    elif not await self.ping():
    File "/opt/pyenv/versions/3.9.6/lib/python3.9/site-packages/asynch/proto/connection.py", line 246, in ping
    packet_type = await self.reader.read_varint()
    File "/opt/pyenv/versions/3.9.6/lib/python3.9/site-packages/asynch/proto/io.py", line 127, in read_varint
    packet = self._read_one()
    File "/opt/pyenv/versions/3.9.6/lib/python3.9/site-packages/asynch/proto/io.py", line 117, in _read_one
    packet = self.buffer[self.position]
    IndexError: bytearray index out of range

Environment:

• Ubuntu 20.10 on amd64
• python 3.9.6 (pyenv install)
• ClickHouse (docker install, tag 21.7.3.14-alpine)

Preliminary Analysis:

It seems that proto/connection::ping would read a varint from ClickHouse after sending the ping packet. However, in my case ClickHouse did not reply anything after a ping request, and thus buffer reader failed to extract anything, raising the error stating that the buffer is empty. I am unsure if the connection is actually closed, or in whatever weird state, since ClickHouse does not have proper specification on its native TCP protocol.

[stankudrow]

@stdc105 , hello. Could you close the issue or is it still actual?