Asyncio Interface

[jamesba]

This adds an additional optional asyncio compatible interface to the library for use only in versions of python above 3.5. In doing so it replicates the existing file-like interface for the most part, but many of the methods become coroutines instead, and for any method that returns a list the asynchronous equivalent is an asynchronous generator.

[jamesba]

@barneygale What do you think of this? There doesn't seem to be an existing standard for how to do seekable file-like io with asyncio yet so I've had to improvise somewhat.

[barneygale]

I've tried to get my head around python's asyncio a couple of times and found it completely baffling, so I'm not much use for a code review. Let me know if/when you're comfortable for this to be merged and I'll merge it.

[jamesba]

Honestly it's not well documented at all, and has been something of a moving target. Hopefully it's a bit more stable now, and better documentation and tutorials should be forthcoming.

My understanding is roughly thus:

• The key to the whole thing is runloops. A runloop is an object which can be instantiated normally, and has a callable run_until_complete which can be called to make it run. When a run-loop is running it takes up the current thread exactly like a normal python method call and only returns when all the tasks in the run loop have completed. You have to explicitly call a runloop and give it at least one task in order to make any asynchronous code run at all.
• A task is a thing currently assigned to a runloop. A single runloop can have multiple tasks or just one. Only one task is actually running at any time, and all the others are dormant. Whenever the current task goes dormant the runloop uses a scheduling algorithm to determine which of its tasks to wake up. If none can be woken up it waits and tries again when one can be.
• Each task has a code execution stack much like an ordinary thread. The difference is that the code in a task is asynchronous code, which means it can go dormant and wait for conditions before waking up again.
• Asynchronous code is written by defining coroutines using async def.
• Inside asynchronous code you can call ordinary synchronous code and it's handled normally. You can't call asynchronous code from synchronous code.
• Inside asynchronous code you can await another asynchronous coroutine. This essentially acts like a function call except that the code being called is asynchronous and so can become dormant during the call and can await other things itself.
• Inside asynchronous code you can await a task. This causes the current task to become dormant until that other task has completed.
• There are also other things you can await, like semaphores and conditions, and the time.
• async with and async for are like with and for except that the magic methods used for context management/iteration are asynchronous coroutines and so can contain asynchronous code.

So basically you can only use the await, async with and async for keywords inside a coroutine defined with async def. A coroutine can only be run inside a task inside a runloop. A runloop will schedule across multiple tasks each time the current one is suspended. A task will never be suspended inside ordinary synchronous code, but can be suspended inside any await, async with or async for statement.

That's how I understand it, anyway.

[philipnbbc]

I think open should call HTTPIOFile.open so that the behaviour is the same as before and it does what a user expects, e.g. raise an exception at the point if it fails to connect.

[philipnbbc]

Should httpio.async be renamed to httpio.asyncio (as you've done elsewhere) to avoid clashes with a reserved name?

[philipnbbc]

Could a .flake8 file be added? It has exposed 1 real issue.

[jamesba]

@philipnbbc care to take another look? This should have fixed all your comments.

[philipnbbc]

@jamesba the tests seem to take an awful lot longer after the new commits, i.e. after 52db224121c2abaa8a3639ddc41f64593425cd28. I can't figure out what change could be causing that.

Running tox -e py36 using the latest code now results in this:

GLOB sdist-make: /store1/external/jamesba-httpio/setup.py
py36 inst-nodeps: /store1/external/jamesba-httpio/.tox/dist/httpio-0.2.0.zip
py36 installed: aiohttp==3.5.4,async-timeout==3.0.1,attrs==19.1.0,certifi==2019.3.9,chardet==3.0.4,coverage==4.5.3,httpio==0.2.0,idna==2.8,idna-ssl==1.1.0,mock==2.0.0,multidict==4.5.2,pbr==5.1.3,requests==2.21.0,six==1.12.0,typing-extensions==3.7.2,urllib3==1.24.2,yarl==1.3.0
py36 runtests: PYTHONHASHSEED='1234660441'
py36 runtests: commands[0] | python -m unittest discover -s tests -p test*.py
Executing <Task finished coro=<TestAsyncHTTPIOFile.test_tell_starts_at_zero() done, defined at /store1/external/jamesba-httpio/tests/test_async_httpio.py:268> result=None created at /usr/lib/python3.7/asyncio/base_events.py:563> took 127.231 seconds
.Executing <Task finished coro=<test_readline() done, defined at /store1/external/jamesba-httpio/tests/test_async_httpio.py:247> result=None created at /usr/lib/python3.6/asyncio/base_events.py:452> took 127.231 seconds
.Executing <Task finished coro=<TestAsyncHTTPIOFile.test_throws_exception_when_get_returns_error() done, defined at /store1/external/jamesba-httpio/tests/test_async_httpio.py:192> result=None created at /usr/lib/python3.7/asyncio/base_events.py:563> took 127.231 seconds
.Executing <Task finished coro=<test_readlines() done, defined at /store1/external/jamesba-httpio/tests/test_async_httpio.py:254> result=None created at /usr/lib/python3.6/asyncio/base_events.py:452> took 127.230 seconds
.Executing <Task finished coro=<test_aiter() done, defined at /store1/external/jamesba-httpio/tests/test_async_httpio.py:261> result=None created at /usr/lib/python3.6/asyncio/base_events.py:452> took 127.304 seconds
...

at which point I cancelled it.

If I run it at commit 52db224121c2abaa8a3639ddc41f64593425cd28 it takes around 30 seconds.

Do you see the same thing?

[jamesba]

@philipnbbc I did, I thought it was a local problem with the network drive I was using. Right, time to try and do a git bisect to work out what has caused this.

[philipnbbc]

The changes LGTM however. The only thing I spotted that might need changing is async def close t check that self._session is not None for case where open was not called.

[jamesba]

Bisect identifies commit 9a3751b9e839da68361c378d7ad7abae71bc5c61 as having introduced the problem. I will take a look at cutting that commit up further on Monday.

[jamesba]

@philipnbbc I have identified and reverted the change that caused the timing problem. I will explore further to try and work out how it caused that, and push a version without the problem.

[jamesba]

@philipnbbc this should now be working

[jamesba]

@philipnbbc @barneygale right, all changes made, history cleaned up. Ready to merge.

[philipnbbc]

@jamesba LGTM.

[barneygale]

Thanks both