search

MeltanoLabs / tap-facebook

Singer tap for extracting data from the Facebook Marketing API
https://pypi.org/project/meltano-tap-facebook/
Other
2 stars 14 forks source link

Rate limit errors #88

Open ezeev opened 1 year ago

ezeev commented 1 year ago

Facebook's rate limits are quite low for apps in "Development" mode and getting approved to come out of development mode is not trivial. This makes it difficult to use Meltano with a new app.

Would it be possible to implement a rate limit backoff? It is possible to see how close you are to the rate limit using a header that FB provides in responses from the API:

def check_limit(account_number, access_token):

    check=requests.get('https://graph.facebook.com/v5.0/'+account_number+'/insights?access_token='+access_token)
    usage=float(find_between(check.headers['x-business-use-case-usage'],'"total_time":',','))
    print('\tRate limit for account %s threshold: %d%%' % (account_number, usage))
    return usage
ezeev commented 1 year ago

Also it looks like the Meltano SDK supports implementing a rate limit backoff? Custom Backoff

I'm brand new to Meltano so my assumptions may be wrong about how or where to implement the rate limit back off.

pnadolny13 commented 1 year ago

cc @edgarrmondragon @kgpayne I think you'd know best how to implement this

edgarrmondragon commented 1 year ago

The tap does try to handle rate limits, but harsher Development tier limits may not be correctly handled: https://github.com/MeltanoLabs/tap-facebook/blob/f8f329588299175733974a51312ea013ef169708/tap_facebook/client.py#L118-L170

If the API provides a header lets the user now when the rate limit will be refreshed, the tap could leverage that in a custom implementation of backoff_runtime.