michalc
commented
8 months ago Wow! Thanks for this
I am unsure about the duplication I have to admit. Instead, should there be a layer that allows calling all the existing sync code in stream_zip from the async world? Something like the following: (only briefly tested)
import asyncio
from stream_unzip import stream_unzip
async def async_stream_unzip(chunks, *args, **kwargs):
async def to_async_iterable(sync_iterable):
# to_thread errors if StopIteration raised in it. So we use a sentinel to detect the end
done = object()
it = iter(sync_iterable)
while (value := await asyncio.to_thread(next, it, done)) is not done:
yield value
def to_sync_iterable(async_iterable):
done = object()
async_it = aiter(async_iterable)
while (value := asyncio.run_coroutine_threadsafe(anext(async_it, done), loop).result()) is not done:
yield value
loop = asyncio.get_running_loop()
unzipped_chunks = stream_unzip(to_sync_iterable(chunks), *args, **kwargs)
async for name, size, chunks in to_async_iterable(unzipped_chunks):
yield name, size, to_async_iterable(chunks)Also, I would probably lean to this being in the same module, stream_zip.py, so from the point of view of client code, the imports are always from stream_zip
And finally, running the tests it looks like they don't pass on Python 3.6 and 3.7: looks like since they use IsolatedAsyncioTestCase which I think was only introduced in Python 3.8. I think I'm a fan of continuing to support older versions of Python unless we really can't for some reason.
Edit: simplified the above code a bit
marcell-szabo
Hello, Recently I had to transform a codebase, that had a dependency on this library, to use asyncio. Therefore, I transformed the stream_unzip.py to accept an asynchronous generator as data source, and run the entire unzipping process asynchronously. I would like to contribute the modification back to this original library, so here is my modified code.
Note, that there has not been any modification that concerns the core unzipping/decrypting etc. functionality, only the generators and data passing has become asynchronous. To prove this I have rewritten the tests in test.py to check the new async behavior, the result can be seen here, all of them are passing.