Fails and hangs with ProactorEventLoop

[AndreLouisCaron]

Hi there!

I hit an issue when trying to use aiomysql with the ProactorEventLoop on Windows. This is pretty much a show stopper for me given the limitations of the (default) SelectorEventLoop on Windows (bad performance, no pipes, no subprocesses, 512 socket limit).

I'm trying out the library with this example slightly adapted from the one in the documentation:

import asyncio
import aiomysql
import sys

async def test_example(loop=None):
    loop = loop or asyncio.get_event_loop()

    conn = await aiomysql.connect(host='192.168.99.100', port=3306,
                                       user='root', password='ch4t',
                                       db='mysql', loop=loop)
    cursor = await conn.cursor()
    await cursor.execute("SELECT Host, User FROM user")
    r = await cursor.fetchall()
    await cursor.close()
    conn.close()
    print('results:', r)

if __name__ == '__main__':
    #if sys.platform == 'win32':
    #    asyncio.set_event_loop(asyncio.ProactorEventLoop())
    loop = asyncio.get_event_loop()
    loop.run_until_complete(test_example(loop=loop))

Here is the output when using the default event loop:

$ python test-mysql.py
results: (('%', 'Test'), ('%', 'root'), ('localhost', 'Test'), ('localhost', 'root'))

Here is the output when uncommenting the two lines that select the ProactorEventLoop:

$ python test-mysql.py
Exception in callback _ProactorReadPipeTransport._loop_reading(<_OverlappedF...op_reading()]>)
handle: <Handle _ProactorReadPipeTransport._loop_reading(<_OverlappedF...op_reading()]>)>
Traceback (most recent call last):
  File "F:\software\python\3.5.2-x64\Lib\asyncio\events.py", line 125, in _run
    self._callback(*self._args)
  File "F:\software\python\3.5.2-x64\Lib\asyncio\proactor_events.py", line 183, in _loop_reading
    data = fut.result()  # deliver data later in "finally" clause
  File "F:\software\python\3.5.2-x64\Lib\asyncio\futures.py", line 268, in result
    raise InvalidStateError('Result is not ready.')
asyncio.futures.InvalidStateError: Result is not ready.
Exception in callback _ProactorReadPipeTransport._loop_reading(<_OverlappedF...password\x00">)
handle: <Handle _ProactorReadPipeTransport._loop_reading(<_OverlappedF...password\x00">)>
Traceback (most recent call last):
  File "F:\software\python\3.5.2-x64\Lib\asyncio\events.py", line 125, in _run
    self._callback(*self._args)
  File "F:\software\python\3.5.2-x64\Lib\asyncio\proactor_events.py", line 181, in _loop_reading
    self._closing)
AssertionError

In addition to the unexpected output, the Python process hangs, does not respond to CTRL-C and I need to explicitly terminate it.

[AndreLouisCaron]

If it's any help, I worker around this problem by setting no_delay=False.

I'm not sure why there is a problem with this, and maybe we need to draw attention to asyncio maintainers, but my spider sense tells me this is somewhat of an anti-pattern with a proactor:

    def _set_nodelay(self, value):
        ...
        transport.pause_reading()
        raw_sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, flag)
        transport.resume_reading()

Because of the nature of (true) asynchronous I/O, there's no way you can guarantee that I/O operations will not complete between the calls to .pause_reading() and .resume_reading().

[AndreLouisCaron]

FYI, as of Python 3.6, the TCP_NODELAY option is automatically set on all TCP connections (see python/asyncio#373).

It's still not the default on the proactor event loop though :-/

[jettify]

Thanks for updates! no_delay option is deprecated on PyMySQL as result we deprecated it too, but looks like we need to keep it until asyncio fix this issue.

[Harmon758]

For posterity, this was a Python bug that was fixed in Python 3.7 and 3.6.6.

[AndreLouisCaron]

@Harmon758 Can you provide a link to the Python bug tracker? Would really appreciate :-)

[Harmon758]

I did so as a Markdown link, but here it is again: https://bugs.python.org/issue26819 Unless this isn't what you meant?