Open jamesls opened 9 years ago
+1000!
+1 000 000
@jamesls can you please edit your issue description to encourage adding a đź‘Ť emoji reaction to your issue instead of "+1 this issue"? Now that github added has since added that feature, it's annoying for people to respond with "+1" replies, pinging all subscribers. The issue text seems to encourage this. Thanks!
+1
@aehlke this ticket has been open since 2015! Three years!! :joy:
People can put whatever they like to draw some attention. Emoji, plus one or plus whatever. If you don't like to pinged you can aways unsubscribe and swing by from time to time!
https://github.com/aio-libs/aiobotocore is the answer at the moment. Lets rally behind that project!
I'd be happy to be wrong, pls check into it enough to confirm either way. I was going from: https://github.com/boto/botocore/issues/458 which is still open, although there is a project mentioned there that supports some AWS services but but not SSM. Also despite the issue I thought I was safe as long as thread per call, but perhaps not?
I'm assuming the NodeJS version is naturally async because of the nature of javascript, but we really need this on Python.
As mentioned, you guys could simply build on top of aiobotocore, which shows that it's not that difficult, the hardest part is testing.
Being able to download several S3 Items asynchronously or invoke several lambdas and wait on the returned response asynchronously is more than necessary! In some cases it would execute codes 20x faster.
+1
+1
+1
+1
+1
+1
+1
why would someone be against a async / non blocking version of boto?
@danketrotzdem we're downvoting people who only add "+1" as a comment, which spams emails to everyone subscribed to this thread without providing any content, when instead they should be using the github reactions on the very first post. We're subscribed to know when this issue is resolved, not to see a constant stream of "+1" emails.
Perhaps it would be time to lock this thread?
Sorry everyone for spamming you.
please asyncio support !! we are in 2020 Oo python2 is dead
@jettify there would be no point in using asyncio if all your IO is blocking on a TheadPool. Adding an event loop doesn't make non-io code faster. In fact it makes it slower since you're putting every method on the event loop for no reason. I've tested it. The only way you get the crazy performance benefits from asyncio is to have everything one thread, thus reducing the CPU context switching and eliminating the GIL locking that comes from multiple threads.
@agronick aiobotocore doesn't use a TheadPool
I'm looking at deploying a Python 3 asyncio micro-service stack into AWS, but I find I cannot use any of the native Amazon services (which would save me a bunch of money) because there is no asyncio support. None of the other cloud suppliers have asyncio support, so I'll still choose AWS, but without the cool service based products.
It would seem there are two approaches to adapting this library.
The most complicated (but probably the most desirable) would be the state machine approach (like the h11 and h2 libraries have adopted). This would move the sync/async decision to downstream libraries, and keep the core library a clean implementation of the protocol.
A more simple approach would be to remove the explicit sync implementation and to provide sync and async abstract classes. These would then be implemented in downstream libraries. This would require careful design to prevent syntactically invalid code being presented to Python 2 interpreters.
I just gave the issue a +1 because trying to use other packages that use botocore
along with aiobotocore
is painful, due to that project being so tightly coupled with botocore
. It is possible with only two packages depending on botocore
, or with six, to end up with all but one package that have a compatible set of botocore
versions, but aiobotocore
's pinning to one specific version means neither pipenv nor poetry can find a match. My solution thus far has been to fork aiobotocore
, change the version for botocore
, and hope it works, but that is bound to break eventually.
One thing that also complicates things is that botocore supports as far back as python 2.6.5, so we'll need to figure out how we can support asyncio and still maintain py2 support. I see that there are asyncio backports to python2, so perhaps something could be done there.
This I assume is no longer a consideration since Python 2 is now obsolete. Is there any official plan/work under way to make the async support happen?
What are the obstacles preventing that from happening now?
You can support anyio and python 2 with one codebase using https://pypi.org/project/unasync/
Adding my voice here as the maintainer of s3fs
, which has recently adopted async functions, and the mismatch between aiobotocore and botocore versioning is hurting us ( https://github.com/dask/s3fs/issues/357 ).
+1 +1 +1
Hi, I was just wondering if there is an official statement on whether this is going to be implemented or not, since this issue has been stale for some time now. Thanks!
Since @jamesls is not with the boto team anymore: Is there anyone else at AWS that would be able to comment on this or should we consider this stale and try to find a different alternative? I'm not sure who the main maintainer is but issue #764 has an answer by @kyleknap so maybe you're able to comment on this? Thanks!
Is there anything the community can do to accelerate or help out with this work?
Also pinging @swetashre since you’re listed on most issues here. Is there any comment from AWS?
I needed the same functionality and ended up creating a simple package, boto3async, to help me with this.
The trick I used is best illustrated with the following example:
async def list_bucktes_async():
s3_client = boto3.client('s3')
return await asyncio.to_thread(s3_client.list_buckets)
async def main():
response = await list_bucktes_async()
print(json.dumps(response, default=str, indent=2))
asyncio.run(main())
then, to generalize the solution so I don't need to do this for every operation I need, I defined the following method:
def asyncify_client(client):
"""
Adds async methods to each of the sync methods of a boto3 client.
Keyword arguments
client -- The client to add sync methods to. Notice that the client
will be updated in place, and will also be returned as a return
value.
Returns:
The same client.
"""
def create_async_func(sync_func):
async def async_func(*args, **kwargs):
return await asyncio.to_thread(sync_func, *args, **kwargs)
return async_func
for operation in client._service_model.operation_names:
operation_camelcase = _camel_to_snake(operation)
sync_func = getattr(client, operation_camelcase)
async_func = create_async_func(sync_func)
setattr(client, f'{operation_camelcase}_async', async_func)
return client
Finally, I defined a method inside boto3async that calls boto3's client() method, followed by calling the above method, to return to the user a boto3 client with _async
version for each operation. So, all the user has to do is call boto3async.client
instead of boto3.client
, passing in the same exact arguments, and then start using the _async
methods.
There are two caveats, unfortunately:
to_thread
from Python 3.9's source code to make the library work on Python 3.7 and later.Hope some find this useful.
@rafidka isn't that just wrapping the threaded implementation into asyncio
? The idea here, if I understand it correctly, is to make asyncio
a first class citizen and ditch threads for IO concurrency, which are much less efficient for a number of reasons.
Ideally trio would be supported as well as asyncio
I think we're having enough of a drag trying to get it for what's included in the standard library. Is there any special requirement that needs to be addressed for trio support rather than just supporting async
/await
?
For anyone with experience in the project, how big of an undertaking is this task? I mean, is it just a matter of priorities (which is fine, everyone has limited time) or complexity? If it's the first, is it reasonable for a newcomer with reasonable programming experience (asking for a friend here)?
Mostly it's swapping aiohttp for httpx which already supports trio via anyio in aiobotocore
@Oppen , yeah, that's right. My implementation is not ideal, but cover most of the use cases. What I wanted is:
It is most probably not as efficient as a native asyncio implementation (I haven't done any benchmark), but most of the time it should be fine, and it gives you the nice language support for the async/await.
For anyone with experience in the project, how big of an undertaking is this task?
I've already reviewed the changes needed, I just need approval to start the work: https://github.com/aio-libs/aiobotocore/issues/749#issuecomment-762150922
@graingert I think I'm confused. Isn't aiobotocore an independent project? Is it something official or community based? I assumed the latter, kind of like a response to this issue being hanging for 6 years. Is there any intentions to merge it to botocore?
@rafidka I see.
@graingert oh, I see what happened. What you quoted was meant for updating botocore, not for the trio part. Now that I review my previous comment it was quite unclear that I was talking about two different things.
+1
let's make python javascript AGAIN!
Any updates about this issue?
Hi all, thanks for your patience. I brought up this up for discussion with the team and the consensus was that async support won't be considered until the next major version. Implementing this would require an entire rewrite of botocore, and that effort is currently being allocated to creating a longer term async solution for botocore users.
Implementing this would require an entire rewrite of botocore, and that effort is currently being allocated to creating a longer term async solution for botocore users.
Could you please clarify what you mean by that? Wouldn’t “creating a longer term async solution for botocore users” be basically the same thing as a rewrite of botocore?
@tim-finnigan is it means you started rewriting the botocore? Can you collaborate with aiobotocore instead?
Redis corporate took over the entire Python redis ecosystem. If a community botocore was written supporting both sync and async(io), would AWS consider adopting it?
@tim-finnigan do you have any update?
@stalkerg this ticket was opened 8 years ago, there's a ticket (https://github.com/aws-cloudformation/cloudformation-coverage-roadmap/issues/196#issuecomment-1669947893) created 4 years ago to add support for stupid tags (something that should maybe take an hour).
I wouldn't hold my breath.
There's aioboto3 and aiobotocore which basically wraps boto3 and botocore in async functions.
We’re interested in first class support because aiobotocore is (partially by necessity) a bad citizen of the Python package ecosystem.
So unless aiobotocore becomes unnecessary because botocore gains async support, we’re stuck with a (as said multiple times necessary) hack. aioboto3
and aiobotocore
should become unnecessary one day, and we should look forward to that day.
Relevant https://github.com/aio-libs/aiobotocore/discussions/1017#discussioncomment-6104364
Funny, he removed it. Basically said that they're not expecting this to happen before 2024, and it will likely be a major version.
There's aioboto3 and aiobotocore which basically wraps boto3 and botocore in async functions.
not anymore they rewrote a lot of parts and did async/await drill down. But yes, they still have a dependence on the botocore.
+1
@tim-finnigan sorry for ping you again, but do you have any updates? Any roadmaps or just strategy? It's literally difficult to do business with AWS predictable right now.
As I understand botocore/boto was initially designed for scripts, CLI tools (aws-cli like) and etc and nobody designed it for full and heavy web-services. AsyncIO basically has no sense for such goals and requires a new SDK, but because you have no resources, you just ignore it (and because aiobotocore exists). Even another AWS team tried to use aiobtocore, and it was a shame, from my point of view.
I believe the community and your customers (not small!) want an update about this topic at least once a year.
This is a tracking issue for the feature request of supporting asyncio in botocore, originally asked about here: https://github.com/boto/botocore/issues/452
There's no definitive timeline on this feature, but feel free to +1 (thumbs up đź‘Ť) this issue if this is something you'd like to see. Also, if you have any additional information about asyncio you'd like to share (even just about your specific use case) feel free to chime in.