PrivacyDevel / nitter

Alternative Twitter front-end
https://nitter.net
GNU Affero General Public License v3.0
151 stars 29 forks source link

Nitter reports throttling on bad token before attempting to refill the pool and serve the request #25

Closed bbappserver closed 1 year ago

bbappserver commented 1 year ago

@PrivacyDevel If the token pool is stale nitter will report that it is throttled instead of first attempting to discard all stale tokens and refill the pool as much as it is able, and then try to serve the request.

This seems to happen intermittently, so I'm not sure if it's when a single token or several are stale resulting in a request failure. It definitely happens when restarting nitter after a while as the tokens still live in redis but are all stale.

bbappserver commented 1 year ago

The correct algorithm would be something like

def onRequest(url): 

 while poolNotEmpty:
   tok=getOldestTokenFromPool()
   r=requestToTwitter(url,tok)
   if r.failed():
    discardToken(tok)
   else:
     return r

#pool was completely stale
refillPool() #refill some or all of tokens until you get service unavailable or reach the count i nthe ocnfig
tok=getOldestTokenFromPool()
r=requestToTwitter(url,tok)     

if r.failed():
  return ThrottledError() #Ok we're out of tokens and we also aren't getting any new ones
else:
  return r #Managed to get a working token even though the whole pool was stale before.
PrivacyDevel commented 1 year ago

I strongly assume that with the use of guest accounts this should no longer be an issue for now. Please reopen if I am wrong.