FloodWait Exception handling

[ChuckNorrison]

Checklist

• [X] I am sure the error is coming from Pyrogram's code and not elsewhere
• [X] I have searched in the issue tracker for similar bug reports, including closed ones
• [X] I ran pip3 install -U https://github.com/pyrogram/pyrogram/archive/master.zip and reproduced the issue using the latest development version

Description

For a download script i try to respect the telegram rate limits as documented here. This does not work cause my FloodWait exception is never reached.

Steps to reproduce

Call download_media in a loop for more than 200 files to download. Check if the exception get catched with a log and sleep the amount of time from FloodWait value.

Code example

import asyncio
from pyrogram import Client
from pyrogram.errors import FloodWait

[...]

try:
    await app.download_media(message, os.path.join(path,""), progress=progress)
except FloodWait as ex_flood:
    logger.warning("Wait %s seconds to download more media...", ex_flood.value)
    await asyncio.sleep(ex_flood.value+1)

Logs

Telegram says: [420 FLOOD_WAIT_X] - A wait of 1076 seconds is required (caused by "auth.ExportAuthorization")
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/pyrogram/client.py", line 861, in get_file
    exported_auth = await self.invoke(
  File "/usr/local/lib/python3.10/site-packages/pyrogram/methods/advanced/invoke.py", line 79, in invoke
    r = await self.session.invoke(
  File "/usr/local/lib/python3.10/site-packages/pyrogram/session/session.py", line 389, in invoke
    return await self.send(query, timeout=timeout)
  File "/usr/local/lib/python3.10/site-packages/pyrogram/session/session.py", line 357, in send
    RPCError.raise_it(result, type(data))
  File "/usr/local/lib/python3.10/site-packages/pyrogram/errors/rpc_error.py", line 91, in raise_it
    raise getattr(
pyrogram.errors.exceptions.flood_420.FloodWait: Telegram says: [420 FLOOD_WAIT_X] - A wait of 1076 seconds is required (caused by "auth.ExportAuthorization")

[qfewzz]

I found a solution after wasting many days. lets say you have 1000 files to download. sort all the files by dc_id, then start downloading files with the new order. you will not get errors related to Authorization. The reason is let's say first file's dc_id is 4, second file is 1, third file is again 4 and so on. By starting from first file, to download the next file, every time you need to Export and import Authentication data from one server to another. I hope you understand what I said.

[ChuckNorrison]

In my function i walk through get_chat_history and this is sorted by date. Than i call download_media if the message media is MessageMediaType.PHOTO.

[jgardiner68]

I do not see Pyrogram raise an error for any of the messages Telegram may send during a download_media call including Telegram internal errors, FloodWait and so on.

Is this raising an error supported for this call?

[KurimuzonAkuma]

In my function i walk through get_chat_history and this is sorted by date. Than i call download_media if the message media is MessageMediaType.PHOTO.

Better to use search_messages with photo filter.

Also you can try this PR: #1313

[jgardiner68]

I am trying to catch the exception using VS Code: I have "justMyCode": false in my launch.json and I have enabled debugging breakpoints for Raised Exceptions, Uncaught Exceptions and User Uncaught Exceptions so the debugger should stop when there is an issue but I get FloodWait and Internal Telegram errors without the debugger stopping. Any ideas?

[nemec]

The reason this error handling doesn't work is because the exception is trapped and logged within download_media

https://github.com/pyrogram/pyrogram/blob/master/pyrogram/client.py#L1020-L1021

            except Exception as e:
                log.exception(e)

The code should either re-throw the exception or return some details so that we can act on it.

[ChuckNorrison]

This was fixed in #1313, thanks @KurimuzonAkuma

Works in this fork: https://github.com/KurimuzonAkuma/pyrogram/releases/tag/v2.1.16

[jgardiner68]

Thanks for the update. Doesnt the link above need a raise e ?

[porridgexj]

I think it is a bug from pyrogram, we just can't catch the flood error from app.download_media, so I find another way to solve this, when show a FloodWait error, the file downloaded size usually is 0KB, so we can test whether the file size is 0KB.

def is_file_empty(file_path):
    if os.path.exists(file_path):
        return os.path.getsize(file_path) == 0
    else:
        return True

flag = True
while flag:
    file_path = app.download_media(file_id)
    if file_path and not is_file_empty(file_path):  
        flag = False
        # solve success situation
    else:
        time.sleep(10)
        # sleep for 10 seconds and retry

[nemec]

I solve this in an incredibly dumb way but it also respects the throttling delay value sent by Telegram. Basically I set up a Python logging filter on the pyrogram logger and trigger an asyncio.Event to block my downloaders from continuing until after the delay is finished.

import asyncio
from datetime import datetime, timedelta
import logging

from pyrogram.errors.exceptions.flood_420 import FloodWait

from .download_manager import DownloadManager

class RateLimitLogFilter(logging.Filter):
    def __init__(self, dm: DownloadManager):
        super().__init__()
        self.download_manager = dm

    def filter(self, record: logging.LogRecord) -> bool:
        if record.exc_info is None:
            return True
        _, exc, _ = record.exc_info

        if isinstance(exc, FloodWait):
            exc: FloodWait
            duration = timedelta(seconds=exc.value)
            wake_time = datetime.now() + duration
            print(f'Encountered rate limit - need to sleep for {duration.total_seconds()} seconds, until {wake_time}')
            asyncio.create_task(self.download_manager.pause_downloads_for(duration))

            return False
        return True

    async def pause_downloads_for(self, duration: timedelta):
        self.rate_limit_event.clear()
        self.console.print("Starting sleep")
        try:
            await asyncio.sleep(duration.total_seconds())
        except asyncio.CancelledError:
            self.console.print("Sleep cancelled")
        finally:
            self.rate_limit_event.set()
            self.console.print("Finished sleep")

    pyrogram_logger = logging.getLogger('pyrogram.client')
    pyrogram_logger.addFilter(RateLimitLogFilter(dm))

Like you mentioned, if file size is 0 you can assume the download failed and just retry after the delay.