search

NonProjects / tgbox

Encrypted cloud storage Protocol based on Telegram
https://tgbox.readthedocs.io/
GNU Lesser General Public License v2.1
34 stars 4 forks source link

Try to multiprocess fastelethon.py (upload/download) #16

Closed NotStatilko closed 1 year ago

NotStatilko commented 2 years ago

Because of GIL Python always runs on one core, so event loop too. We can create CPU_COUNT processes and upload files in parallel, but there is two big problems: the first, most obvious, is possible FloodWaitError on Telegram side, the second is pickle, it just doesn't work with coroutines and big classes.

aiomultiprocess.Pool doesn't work, as it rely on pickle. dill instead of pickle.... doesn't work. We need to fully rewrite fastelethon, so it's coros will return parts from which we can reconstruct objects in other Process.

It looks good in theory, but FloodWaitError and GIL... meh, i don't want to do it now. If anyone has ideas - feel free to publish them here.

NotStatilko commented 1 year ago

fastelethon.py functions is fast enough. Sure, we can implement this if we want to obtain a permanent ban by FloodWaitError(s)