Wauplin
commented
1 day ago Hi @Skylion007, thanks for opening this topic!
Would you be interested in adding such a feature in ./src/huggingface_hub/utils/_http.py. I would add it directly to the _default_backend_factory by defining a custom adapter that retry on 429 errors (with a warning and according to "Retry-After" header). In practice, you could define a single adapter HfDefaultAdapter that does the current job of UniqueRequestIdAdapter, OfflineAdapter, and the retry on rate-limit logic. WDYT?
Is your feature request related to a problem? Please describe. A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] It surprises me that the http-after does not seem to respect Retry-After headers if I am reading the codebase properly. It's pretty easy to add directly into the requests library using an adapter and the Requests Retry object like so: https://github.com/fsspec/filesystem_spec/blob/262f664574e091228251b467ac92b2a6c327034b/fsspec/implementations/dbfs.py#L54 or by parsing the proper HTTP field. This would reduce the amount of errors encountered I find when trying to stream data lots of data from the Hub and would also implement nice effective client-side rate limiting.
Describe the solution you'd like A clear and concise description of what you want to happen. Respect Retry-After headers, especially on 429 status codes.
Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.
Additional context Add any other context or screenshots about the feature request here.