Random error 502

[DV777]

Hello Botometer,

I keep on receiving random 502 errors while checking lists of accounts.

Is this situation circumstancial or is there something I can do to improve the request rate / day ?

It sharply fell from around 15k-17k to less than 3k per day + I need to manually adapt and restart the requests a dozen times / day.

Thanks in advance for your help, DV7

Error format:

(...)HTTPError (http_error_msg, response=self)
HTTPError: 502 Server Error: Bad Gateway for url: https://botometer-pro.p.rapidapi.com/2/check_account

Original code:

low_memory=False 
import nest_asyncio 
nest_asyncio.apply() 
import pandas as pd 
import os 
import glob 
import csv 
import botometer 
import json 
import time 

column_usernames_corrected = pd.read_csv('column_usernames_corrected.csv') 
column_usernames_corrected = column_usernames_corrected.astype(str) 
error_accounts = ['error accounts'] 

botometer_api_url = 'https://botometer-pro.p.rapidapi.com' 
twitter_app_auth = {         
        'consumer_key': 'XXX', 
        'consumer_secret': 'XXX', 
        'access_token': 'XXX', 
        'access_token_secret': 'XXX', 
} 
rapidapi_key = 'XXX' 
bom = botometer.Botometer(botometer_api_url=botometer_api_url, 
                          rapidapi_key=rapidapi_key, 
                          wait_on_ratelimit=True, 
                          **twitter_app_auth) 
print(time.time()) 
print("Starting...") 
for i in range(len(column_usernames_corrected)): 
    try: 
        result = bom.check_account('@' + column_usernames_corrected.loc[i, 'username']) 
        with open(column_usernames_corrected.loc[i, 'username'] +'.json', 'a', encoding='utf-8') as js: 
            json.dump(result, js, ensure_ascii=False, indent=4) 
    except botometer.TweepError: 
        error_accounts.append(column_usernames_corrected.loc[i, 'username']) 
        error_accounts_csv = pd.DataFrame(error_accounts, columns=['error accounts']) 
        error_accounts_csv.to_csv('error_accounts.csv', index=False, encoding='utf-8-sig') 
        pass 
print("Step 1 - Done") 
print(time.time()) 

[bitscott1102]

same. It keeps saying that requests.exceptions.HTTPError: 502 Server Error:. How did you solve it?

[DV777]

Unfortunately I think it's related to their servers being overloaded at the moment. So there is very little we can do. I start checking accounts from where it stops each time... I tried setting some automatic relaunch but when the random 502 error takes place it tends to stick for some time before you can relaunch - however it's a bit different each time.

[clayadavis]

Indeed, it's due to overcapacity. They are working on it as fast as possible given limited resources, even called me out of retirement to consult 😉

[DV777]

Thanks for the update Mr Davis 👍

[yang3kc]

The situation should be improved now.