search

dol-sen / ssl-fetch

A convenience python library for verified ssl connection downloads using headers such as User-Agent and If-Modified-Since.
GNU General Public License v2.0
5 stars 5 forks source link

connections.py: Adds time limit enforcer to downloads #7

Closed twitch153 closed 8 years ago

twitch153 commented 8 years ago

To lighten the load on downloading from Infra's servers, ssl-fetch has implemented an enforcer which limits how often a program that uses it can download the same file from the server.

dwfreed commented 8 years ago

This should not have a default of 60 minutes; ssl-fetch is a generic library that has uses outside of Gentoo

dol-sen commented 8 years ago

This current patch, is incorrect for other reasons. But it's been requested by infra for a cycle rate limit of 1 hour. Doing it here with a new defaulted parameter will take care of all 3 api consumers without the need of them being bumped. With a fast stabilization, it'll be in effect quickly.

We can then work on updating the api consumers to handle it properly and reduce the cycle limit or disable it by default.

dol-sen commented 8 years ago

there is a group of consumers that have layman hitting up api.gentoo.org every 5 seconds. taking up a lot of bandwidth without actual transfers most of the time.

twitch153 commented 8 years ago

The commit has been rebased to make the proper changes which we originally had in mind. Which was to check against the mtime of the file and not the timestamp actually inside the timestamp file.

twitch153 commented 8 years ago

Which default are we changing to 0? climit? Do you want me to make that change too?

twitch153 commented 8 years ago

So should we go ahead and merge this pull request? :+1: