search

googleapis / python-storage

Apache License 2.0
424 stars 150 forks source link

Is there a async storage client, So `async`, `wait` can be used? #478

Open Cedric-Chen opened 3 years ago

Cedric-Chen commented 3 years ago

For google speech to text api, there are two types of client

But I couldn't find one AsyncClient for google storage service. Is there one official implementation?

danielduhh commented 3 years ago

Hi @Cedric-Chen, we don't officially support asyncio in our python client. I'll mark this as a feature request and we'll consider it amongst exiting requests. Thanks!

danielduhh commented 2 years ago

cc: @Breathtender @andrewsg

unforced commented 2 years ago

Hey @Cedric-Chen and anyone else curious about this. We just removed support for Python2 from our client, and that move forward should be supportive for us integrating more async functionality in our official library, and that is something we will be exploring more in this year. Anything that does get put into the official library, we will try to make a note of here.

For now, while there isn't much official support included yet, it is possible to manually create async tasks in such a way that you can get some parallelization going with the storage library. An example of that is included here: https://github.com/googleapis/python-storage/blob/main/samples/snippets/storage_async_upload.py

If there are any questions please feel free to inquire and I can do my best to answer.

raphaelauv commented 2 years ago

In the example of @Breathtender things will run concurrently but not in parallel , cause upload_from_string is not async and you are not using a multiprocessing executor ( multithreading only run 1 thread at the same time with the python GIL ).

If upload_from_string was async then it will not make a blocking network system call and release his own execution to the loop with the keyword await, with that we would have 'optimal' concurrency

adamserafini commented 2 years ago

Yep, I was excited for a moment, but the example isn't a case of 'true' asyncio unfortunately.

Is there any updated timeline for proper first class asyncio upload/download support in this library? 🙏

zoidyzoidzoid commented 2 years ago

This library seems to at least have some asyncio functionality on top of the gcloud library: https://github.com/talkiq/gcloud-aio

praveen-elastic commented 1 year ago

A few functions are skipped by gcloud-aio package mentioned here by @zoidbergwill . This repo provides the support for all the asyncio functionalities which can be achieved using json api. I hope you found it helpful.

mattyoungberg commented 4 months ago

Commenting in 2024, almost three years since this ticket has been opened... Any chance this feature request has made progress? It'd be great for this library to offer official asyncio support like many other official GCP Python clients do. Reading and writing chunks over a network to Cloud Storage is the exact kind of I/O-bound workload that asyncio excels at. That, and asyncio's uptake in the Python world is only increasing, not decreasing.

LEAGUEDORA commented 4 months ago

Hey Google Storage team, our whole architecture is on GCP. And we have an application where the code must be super async (No code must be sync), we are stuck with this package because there is no async support. And the Google-aio-storage thing takes 100ms time to download/upload. I am open to join as a beta user too!