boto / botocore

The low-level, core functionality of boto3 and the AWS CLI.
Apache License 2.0
1.51k stars 1.09k forks source link

Support asyncio #458

Open jamesls opened 9 years ago

jamesls commented 9 years ago

This is a tracking issue for the feature request of supporting asyncio in botocore, originally asked about here: https://github.com/boto/botocore/issues/452

There's no definitive timeline on this feature, but feel free to +1 (thumbs up đź‘Ť) this issue if this is something you'd like to see. Also, if you have any additional information about asyncio you'd like to share (even just about your specific use case) feel free to chime in.

mnolet commented 6 years ago

+1000!

mikepashin commented 6 years ago

+1 000 000

aehlke commented 6 years ago

@jamesls can you please edit your issue description to encourage adding a đź‘Ť emoji reaction to your issue instead of "+1 this issue"? Now that github added has since added that feature, it's annoying for people to respond with "+1" replies, pinging all subscribers. The issue text seems to encourage this. Thanks!

ashwiniadiga commented 6 years ago

+1

lsbardel commented 6 years ago

@aehlke this ticket has been open since 2015! Three years!! :joy:

People can put whatever they like to draw some attention. Emoji, plus one or plus whatever. If you don't like to pinged you can aways unsubscribe and swing by from time to time!

https://github.com/aio-libs/aiobotocore is the answer at the moment. Lets rally behind that project!

kadut commented 6 years ago

I'd be happy to be wrong, pls check into it enough to confirm either way. I was going from: https://github.com/boto/botocore/issues/458 which is still open, although there is a project mentioned there that supports some AWS services but but not SSM. Also despite the issue I thought I was safe as long as thread per call, but perhaps not?

mojimi commented 5 years ago

I'm assuming the NodeJS version is naturally async because of the nature of javascript, but we really need this on Python.

As mentioned, you guys could simply build on top of aiobotocore, which shows that it's not that difficult, the hardest part is testing.

Being able to download several S3 Items asynchronously or invoke several lambdas and wait on the returned response asynchronously is more than necessary! In some cases it would execute codes 20x faster.

belegnar commented 5 years ago

+1

atif1996 commented 5 years ago

+1

arairyoto commented 4 years ago

+1

Zeus1984 commented 4 years ago

+1

sharkguto commented 4 years ago

+1

Catastropha commented 4 years ago

+1

patrickwerz commented 4 years ago

+1

why would someone be against a async / non blocking version of boto?

Naddiseo commented 4 years ago

@danketrotzdem we're downvoting people who only add "+1" as a comment, which spams emails to everyone subscribed to this thread without providing any content, when instead they should be using the github reactions on the very first post. We're subscribed to know when this issue is resolved, not to see a constant stream of "+1" emails.

Perhaps it would be time to lock this thread?

Sorry everyone for spamming you.

sharkguto commented 4 years ago

please asyncio support !! we are in 2020 Oo python2 is dead

agronick commented 4 years ago

@jettify there would be no point in using asyncio if all your IO is blocking on a TheadPool. Adding an event loop doesn't make non-io code faster. In fact it makes it slower since you're putting every method on the event loop for no reason. I've tested it. The only way you get the crazy performance benefits from asyncio is to have everything one thread, thus reducing the CPU context switching and eliminating the GIL locking that comes from multiple threads.

graingert commented 4 years ago

@agronick aiobotocore doesn't use a TheadPool

rob-blackbourn commented 4 years ago

I'm looking at deploying a Python 3 asyncio micro-service stack into AWS, but I find I cannot use any of the native Amazon services (which would save me a bunch of money) because there is no asyncio support. None of the other cloud suppliers have asyncio support, so I'll still choose AWS, but without the cool service based products.

It would seem there are two approaches to adapting this library.

The most complicated (but probably the most desirable) would be the state machine approach (like the h11 and h2 libraries have adopted). This would move the sync/async decision to downstream libraries, and keep the core library a clean implementation of the protocol.

A more simple approach would be to remove the explicit sync implementation and to provide sync and async abstract classes. These would then be implemented in downstream libraries. This would require careful design to prevent syntactically invalid code being presented to Python 2 interpreters.

hlongmore commented 4 years ago

I just gave the issue a +1 because trying to use other packages that use botocore along with aiobotocore is painful, due to that project being so tightly coupled with botocore. It is possible with only two packages depending on botocore, or with six, to end up with all but one package that have a compatible set of botocore versions, but aiobotocore's pinning to one specific version means neither pipenv nor poetry can find a match. My solution thus far has been to fork aiobotocore, change the version for botocore, and hope it works, but that is bound to break eventually.

kkozmic commented 4 years ago

One thing that also complicates things is that botocore supports as far back as python 2.6.5, so we'll need to figure out how we can support asyncio and still maintain py2 support. I see that there are asyncio backports to python2, so perhaps something could be done there.

This I assume is no longer a consideration since Python 2 is now obsolete. Is there any official plan/work under way to make the async support happen?

What are the obstacles preventing that from happening now?

graingert commented 4 years ago

You can support anyio and python 2 with one codebase using https://pypi.org/project/unasync/

martindurant commented 4 years ago

Adding my voice here as the maintainer of s3fs, which has recently adopted async functions, and the mismatch between aiobotocore and botocore versioning is hurting us ( https://github.com/dask/s3fs/issues/357 ).

+1 +1 +1

BastianZim commented 4 years ago

Hi, I was just wondering if there is an official statement on whether this is going to be implemented or not, since this issue has been stale for some time now. Thanks!

BastianZim commented 4 years ago

Since @jamesls is not with the boto team anymore: Is there anyone else at AWS that would be able to comment on this or should we consider this stale and try to find a different alternative? I'm not sure who the main maintainer is but issue #764 has an answer by @kyleknap so maybe you're able to comment on this? Thanks!

aaronclong commented 3 years ago

Is there anything the community can do to accelerate or help out with this work?

BastianZim commented 3 years ago

Also pinging @swetashre since you’re listed on most issues here. Is there any comment from AWS?

rafidka commented 3 years ago

I needed the same functionality and ended up creating a simple package, boto3async, to help me with this.

The trick I used is best illustrated with the following example:

async def list_bucktes_async():
    s3_client = boto3.client('s3')
    return await asyncio.to_thread(s3_client.list_buckets)

async def main():
    response = await list_bucktes_async()
    print(json.dumps(response, default=str, indent=2))

asyncio.run(main())

then, to generalize the solution so I don't need to do this for every operation I need, I defined the following method:

def asyncify_client(client):
    """
    Adds async methods to each of the sync methods of a boto3 client.

    Keyword arguments
    client -- The client to add sync methods to. Notice that the client
        will be updated in place, and will also be returned as a return
        value.

    Returns:
    The same client.
    """

    def create_async_func(sync_func):
        async def async_func(*args, **kwargs):
            return await asyncio.to_thread(sync_func, *args, **kwargs)
        return async_func

    for operation in client._service_model.operation_names:
        operation_camelcase = _camel_to_snake(operation)
        sync_func = getattr(client, operation_camelcase)
        async_func = create_async_func(sync_func)
        setattr(client, f'{operation_camelcase}_async', async_func)

    return client

Finally, I defined a method inside boto3async that calls boto3's client() method, followed by calling the above method, to return to the user a boto3 client with _async version for each operation. So, all the user has to do is call boto3async.client instead of boto3.client, passing in the same exact arguments, and then start using the _async methods.

There are two caveats, unfortunately:

Hope some find this useful.

Oppen commented 3 years ago

@rafidka isn't that just wrapping the threaded implementation into asyncio? The idea here, if I understand it correctly, is to make asyncio a first class citizen and ditch threads for IO concurrency, which are much less efficient for a number of reasons.

graingert commented 3 years ago

Ideally trio would be supported as well as asyncio

Oppen commented 3 years ago

I think we're having enough of a drag trying to get it for what's included in the standard library. Is there any special requirement that needs to be addressed for trio support rather than just supporting async/await?

For anyone with experience in the project, how big of an undertaking is this task? I mean, is it just a matter of priorities (which is fine, everyone has limited time) or complexity? If it's the first, is it reasonable for a newcomer with reasonable programming experience (asking for a friend here)?

graingert commented 3 years ago

Mostly it's swapping aiohttp for httpx which already supports trio via anyio in aiobotocore

rafidka commented 3 years ago

@Oppen , yeah, that's right. My implementation is not ideal, but cover most of the use cases. What I wanted is:

It is most probably not as efficient as a native asyncio implementation (I haven't done any benchmark), but most of the time it should be fine, and it gives you the nice language support for the async/await.

graingert commented 3 years ago

For anyone with experience in the project, how big of an undertaking is this task?

I've already reviewed the changes needed, I just need approval to start the work: https://github.com/aio-libs/aiobotocore/issues/749#issuecomment-762150922

Oppen commented 3 years ago

@graingert I think I'm confused. Isn't aiobotocore an independent project? Is it something official or community based? I assumed the latter, kind of like a response to this issue being hanging for 6 years. Is there any intentions to merge it to botocore?

@rafidka I see.

Oppen commented 3 years ago

@graingert oh, I see what happened. What you quoted was meant for updating botocore, not for the trio part. Now that I review my previous comment it was quite unclear that I was talking about two different things.

gainskills commented 2 years ago

+1

GeekEast commented 2 years ago

let's make python javascript AGAIN!

AmitArie commented 2 years ago

Any updates about this issue?

tim-finnigan commented 1 year ago

Hi all, thanks for your patience. I brought up this up for discussion with the team and the consensus was that async support won't be considered until the next major version. Implementing this would require an entire rewrite of botocore, and that effort is currently being allocated to creating a longer term async solution for botocore users.

flying-sheep commented 1 year ago

Implementing this would require an entire rewrite of botocore, and that effort is currently being allocated to creating a longer term async solution for botocore users.

Could you please clarify what you mean by that? Wouldn’t “creating a longer term async solution for botocore users” be basically the same thing as a rewrite of botocore?

stalkerg commented 1 year ago

@tim-finnigan is it means you started rewriting the botocore? Can you collaborate with aiobotocore instead?

Andrew-Chen-Wang commented 1 year ago

Redis corporate took over the entire Python redis ecosystem. If a community botocore was written supporting both sync and async(io), would AWS consider adopting it?

stalkerg commented 1 year ago

@tim-finnigan do you have any update?

takeda commented 1 year ago

@stalkerg this ticket was opened 8 years ago, there's a ticket (https://github.com/aws-cloudformation/cloudformation-coverage-roadmap/issues/196#issuecomment-1669947893) created 4 years ago to add support for stupid tags (something that should maybe take an hour).

I wouldn't hold my breath.

There's aioboto3 and aiobotocore which basically wraps boto3 and botocore in async functions.

flying-sheep commented 1 year ago

We’re interested in first class support because aiobotocore is (partially by necessity) a bad citizen of the Python package ecosystem.

So unless aiobotocore becomes unnecessary because botocore gains async support, we’re stuck with a (as said multiple times necessary) hack. aioboto3 and aiobotocore should become unnecessary one day, and we should look forward to that day.

dacevedo12 commented 1 year ago

Relevant https://github.com/aio-libs/aiobotocore/discussions/1017#discussioncomment-6104364

Funny, he removed it. Basically said that they're not expecting this to happen before 2024, and it will likely be a major version.

stalkerg commented 1 year ago

There's aioboto3 and aiobotocore which basically wraps boto3 and botocore in async functions.

not anymore they rewrote a lot of parts and did async/await drill down. But yes, they still have a dependence on the botocore.

ricardodns commented 1 year ago

+1

stalkerg commented 7 months ago

@tim-finnigan sorry for ping you again, but do you have any updates? Any roadmaps or just strategy? It's literally difficult to do business with AWS predictable right now. As I understand botocore/boto was initially designed for scripts, CLI tools (aws-cli like) and etc and nobody designed it for full and heavy web-services. AsyncIO basically has no sense for such goals and requires a new SDK, but because you have no resources, you just ignore it (and because aiobotocore exists). Even another AWS team tried to use aiobtocore, and it was a shame, from my point of view.
I believe the community and your customers (not small!) want an update about this topic at least once a year.