search

Cadair / parfive

An asyncio based parallel file downloader for Python 3.8+
https://parfive.readthedocs.io/
MIT License
50 stars 24 forks source link

Use `asyncio.TaskGroup` for downloading in parallel #128

Open exitflynn opened 1 year ago

exitflynn commented 1 year ago

Task Groups are a new feature introduced in python 3.11 which make it more convenient to create/work with parallel tasks. This change would more or less just be a nice-to-have in the sense that it'd reduce some complexity when looking at the code and make it more easier to maintain/contribute. However there's a requirement of a python version >=3.11, which I'm guessing won't be on most machines right now, so I'm not sure if it's something we can implement in the near-future (i'm unfamiliar with how these things are dealt with usually in projects). I figured I'd raise an issue addressing it.

Cadair commented 1 year ago

Oh that looks super useful, but it's going to be a very long time till we can be 3.11+. I wonder of there is a backport on pypi we could use for <3.11 ?

exitflynn commented 1 year ago

there are! just gave it a look and there's aiotools and taskgroup.

Cadair commented 1 year ago

You got me going down a rabbit hole, this looks interesting as well: https://aiojobs.readthedocs.io/en/stable/intro.html What I can't decide at the moment is at what layer in the stack it would live.

All these things are great ideas for a complete internal rewrite I would like to do to simplify the internal logic, but I can't see that happening any time soon!

exitflynn commented 1 year ago

i seee! i'll love to see that and help out in any way i can!