asyncio support

[lukesneeringer]

From @graingert on March 6, 2017 16:45

I want to use google-cloud-python in a Sanic web service, however I'm having to use run_in_executor() to use this library without stalling the eventloop.

This is much less efficient than an Asyncio native client.

Copied from original issue: GoogleCloudPlatform/google-cloud-python#3103

[lukesneeringer]

From @graingert on March 6, 2017 18:35

@jonparrott it would be possible to support python 2.x and python 3.x with various event loops from the same source by using a custom deferred, future or IOAction implementation.

[lukesneeringer]

From @jonparrott on March 6, 2017 18:36

This might be possible once we exorcise httplib2 from this project. @lukesneeringer

[lukesneeringer]

That sounds right to me (and that has moved up in priority slightly).

Tagging #1998 for reference.

[lukesneeringer]

From @graingert on March 6, 2017 18:40

the core code would use Action composition:

return get_url(xys).map(process_request).flat_map(make_another_request)

Then there could be various, pluggable http transport implementations, that would take an Action instance and return a Future from various event systems (asyncio, twisted), or even use blocking IO.

[lukesneeringer]

From @graingert on March 8, 2017 9:59

import asyncio
import functools

class AsyncIOAction(object):
    def __init__(self, fn):
        this.exec = fn

    def map(self, fn):
        @functools.wraps(fn)
        async def exec():
            value = await self.exec()
            return fn(value)

        return AsyncIOAction(exec)

    def flat_map(self, fn):
        @functools.wraps(fn)
        async def exec():
            value = await self.exec()
            return await fn(value).exec()

        return AsyncIOAction(exec)

    @classmethod
    def all(items):
        async def exec():
            return await asyncio.wait([x.exec() for x in items])

        return AsyncIOAction(exec)

def aiohttp_action(request):
    async def exec():
        return await aiohttp.request(**request)

    return AsyncIOAction(exec)

# blocking.py
import functools

class BlockingIOAction(object):
    def __init__(self, fn):
        this.exec = fn

    def map(self, fn):
        @functools.wraps(self, fn)
        def exec():
            value = self.exec()
            return fn(value)
        return BlockingIOAction(exec)

    def flat_map(self, fn):
        @functools.wraps(fn)
        def exec():
            value = self.exec()
            return fn(value).exec()

        return BlockingIOAction(exec)

    @classmethod
    def all(items):
        def exec():
            return [x.exec() for x in items]

        return BlockingIOAction(exec)

def requests_action(request): # or httplib2 etc.
    def exec():
        requests.request(**request)

    return BlockingIOAction(exec)

# twisted.py

class TwistedIOAction(object):
    ...

Should be pretty easy with twisted deferred too. Idea being, calling code would choose an IO type, configure the google cloud platform library appropriately, call a method on some API endpoint and then be able to call .exec() on the returned action. The calling code would then be able to receive the value, await and receive the value, or yield it in twisted inline callbacks.

[lukesneeringer]

From @graingert on March 8, 2017 13:57

Here's how you might to inline callbacks: https://gist.github.com/graingert/ca6cdd5d9ae2e18ca917b4594ac8a633#file-asyncio_and_blocking_io-py-L141-L148

[lukesneeringer]

From @graingert on March 8, 2017 14:0

inside google code platform:

def some_api_method(IOAction, ham, spam):
    @IOAction.inline_callbacks
    def _some_api_method():
        v = yield some_other_method(IOAction, ham)
        v2 = yield another_method(IOAction, spam, v)
        Return(v + v2)

return _some_api_method()

In some calling code:

async def async_caller():
    await some_api_method(AsyncIOAction, ham="ham", spam="spam").exec()

def blocking_caller():
    some_api_method(BlockingIOAction, ham="ham", spam="spam").exec()

[lukesneeringer]

From @graingert on July 24, 2017 12:8

of course https://pypi.python.org/pypi/txaio might be a better way of implementing it.

[lukesneeringer]

From @jonparrott on July 24, 2017 16:9

@graingert oh that's super interesting. However, this seems to re-enforce the fact that we can't really support this without breaking our public interface and without seriously re-thinking all of our internals. We'd have to start from scratch with this in mind. :/

[lukesneeringer]

From @graingert on July 24, 2017 16:11

@jonparrott with https://github.com/crossbario/txaio/issues/113 and https://github.com/crossbario/txaio/issues/110

you probably won't have to change that much.

And you'd be able to keep the same public api for people using requests.

[lukesneeringer]

From @jonparrott on July 24, 2017 16:14

inlinecallbacks would help, but this is an enormous undertaking and a lot of our code that deals with IO is non-trivial (resumable-media).

I'm open to doing this, but from my perspective it seems like the tooling isn't quite ready and it would take an enormous amount of engineering effort to get all of our plumbing ready- plus, we have a consider how gRPC plays into all this.

[lukesneeringer]

From @dhermes on July 24, 2017 16:15

FWIW, resumable-media would be the easiest thing of all to add support for because the actual I/O is "isolated" (I took the sans-I/O approach when writing it)

[lukesneeringer]

From @graingert on July 24, 2017 16:16

@jonparrott @dhermes yeah a sans-io approach for this whole library would be the best option!

[lukesneeringer]

From @jonparrott on July 24, 2017 16:16

then it might be a good place to proof-of-concept this?

[lukesneeringer]

From @dhermes on July 24, 2017 16:30

@jonparrott I'm happy to review PRs but I don't have cycles.

As an example, take a look at the very small amount of code in ResumableUpload.transmit_next_chunk:

        method, url, payload, headers = self._prepare_request()
        result = _helpers.http_request(
            transport, method, url, data=payload, headers=headers,
            retry_strategy=self._retry_strategy)
        self._process_response(result, len(payload))
        return result

Every public method is split into "prepare" and "process" stages in the same way.

[lukesneeringer]

I am going to move this to resumable-media.

[graingert]

Hot stuff. Let's drop python 2.7 support

[theacodes]

Hot stuff. Let's drop python 2.7 support

If only our users wouldn't literally murder us for it. :)

[hzlmn]

Hey, any progress on this? big ➕ for asyncio support

[theacodes]

It's likely just not going to be feasible until we can stop supporting Python 2.7.

We can (and will) revisit this in 2020.

[graingert]

What's wrong with using sans-io or txaio? @jonparrott

[theacodes]

This library itself is already mostly sans-io, so if someone wanted to go and write an asyncio interface they can - I'll happily review it. But for us there's no practical benefit to prioritizing that work because our 1st-party downstream clients (google-cloud-storage and google-cloud-bigquery) can not take advantage of asyncio.

[alexpirine]

OMG 2020 I might get hit by a bus before that and never see it :(

[theacodes]

@alexpirine please remember to be respectful in this space, as participating in project discussions mean upholding our code of conduct.

We hear you. We want this to, but we have to do right by our large set of 2.7 users. You may have noticed we announced a depreciation timeline for 2.7 recently. This is the first big step in us being able to adopt new python 3 only features. We move a bit more slowly than some would like, but we do that so that we don't constantly break folks.

[harmony-ek]

2020 is on the horizon; could you share an update on this?

[busunkim96]

Thanks for checking in @harmony-ek.

We'll be dropping Python 2 support across our libraries towards the end of Q1 2020. We are also changing the generator that produces library code from this to https://github.com/googleapis/gapic-generator-python. We're aiming to keep the differences as minimal as possible for the initial transition. That said, we know that users are interested in asyncio support so we will look into doing that work later in 2020.

If any of you reading through this issue have a support contract with Google, please do file a official request. It will help us prioritize the work.

Thanks!

[satels]

2020, lets do supporting asyncio

[graingert]

Or, preferably anyio with httpx

[alexpirine]

2020 has come and I'm alive, let's roll out asyncio support.

[underyx]

@theacodes considering it's 2020, would you mind reopening this issue? 😇

[busunkim96]

Thank you all for the interest! I'll re-open this now for tracking purposes, and keep this issue updated as we develop more definite plans for asyncio this year.

[graingert]

This library should support trio, ~curio~ and asyncio via anyio and httpx

[redraw]

any news? :)

[adriangb]

Hi from 2022. Python 2 has been EOL for years. Uploading files is probably the best bang for the buck out there for async Python.

[ocervell]

Hello from 2024...