Asynchonous context managers and iterators

[pstch]

README.md says :

The closing of a file may block, and yielding from a coroutine while exiting from a context manager isn't possible, so aiofiles file objects can't be used as context managers. Use the try/finally construct from the introductory section to ensure files are closed.

Iteration is also unsupported. To iterate over a file, call readline repeatedly until an empty result is returned. Keep in mind readline doesn't strip newline characters.

Python 3.5 provides asynchronous context managers and iterators, using __aenter__, __aexit__, __aiter__ and __anext__. I think adding support for these could provide a nice API for aiofiles, and it wouldn'bt be harder than something along the lines of:

class IterableContextManagerFileWrapper:  # yes it needs a new name
    def __init__(self, *args, **kwargs):
        # store arguments so that file can be created in __aenter__
        self.__args, self.__kwargs = args, kwargs
        self.__file = None  # set in __aenter__

     def __getattr__(self, name):
         assert self.__file is not None, "file context manager not entered"
         return getattr(self.__file, name)   # wrap the async file object

     async def __aenter__(self):
         self.__file = await open(*self.args, **self.kwargs)
         return self  # return self and not file so that the value can be used as iterable

     async def __aexit__(self, exc_type, exc_value, exc_tb):
         await self.close()
         return False  # no reason to intercept exception

     async def __aiter__(self):
         return self  # this object is an iterator, so just has to return itself to be an iterable

     async def __anext__(self):
          line = await self.readline()
          if not line:  # EOF
              raise StopAsyncIteration  # not StopIteration !
          else:
              return line

The resulting wrapper could now be used as :

async with IterableContextManagerFileWrapper("/tmp/lol") as aiofile:
    async for line in aiofile:
        line = line.rstrip()
        ...

My proposed POC has a weird name because I don't really know to integrate it into the class hierarchy, so I made a wrapper that provides the proposed features on top of the objects returned by aiofiles coroutines.

This is also why I submit this is an issue and not a PR, because I would like to get your opinion on what would be the best way to implement.

[Tinche]

Hi @pstch,

first of all, thanks for helping to make aiofiles better. I agree we should support the new 3.5 protocols, I just haven't gotten around to it yet.

Let me think a little about the nicest way of implementing this.

[jettify]

I think API should be:

async with aiofiles.open('foo.bin', 'wb') as f:
    await f.write(b'data')

similar to what aiohttp[1], aiopg[2] and others do. aiofiles.open should behave like coroutine and async context manager in same time. Example how to do this you could find for instance in aiopg[3](original idea located in aiohttp repository)

[1] http://aiohttp.readthedocs.org/en/stable/client.html#make-a-request [2] https://github.com/aio-libs/aiopg/blob/master/examples/simple_sa.py#L22-L32 [3] https://github.com/aio-libs/aiopg/blob/master/aiopg%2Futils.py#L13

[Tinche]

@jettify,

agreed, it'd be best if we can continue mimicking the sync file API. Thanks for the links!

[jettify]

Let me know if you have questions about implementation. But basic idea is following:

1) copy paste that huge context manger object ( it mimics coroutine api) 2) implement custom __aenter__, __aexit__ 3) return context manger object in aiofiles.open without decorating method with @asyncio.coroutine

[Tinche]

Alright, I think I've got this done. Just need to update the tests and docs.

[pstch]

I knew about aiopg's _ContextManager, but had never used it in my own code and was thus scared to propose it. It seems like a good alternative indeed.

[Tinche]

Alright, 0.3.0 commited and pushed to PyPI. Thanks folks!

Also, if this _ContextManager is so useful, why not put it into a nice separate package for everyone to depend on? Would help with code coverage in any case, I don't actually know how to test it all :)