xdevplatform / twitter-python-ads-sdk

A Twitter supported and maintained Ads API SDK for Python.
https://twitterdev.github.io/twitter-python-ads-sdk/
MIT License
189 stars 106 forks source link

'handle_rate_limit' incorrect behaviour #300

Open pep4eto1211 opened 2 years ago

pep4eto1211 commented 2 years ago

Description:

When creating an instance of Client (found in twitter_ads.client) you can pass a dict with additional options, using the options parameter. One of this options is handle_rate_limit. When passed, the library is supposed to look for the rate limit headers returned by the API and sleep the execution when such a limit is encountered. However, it would appear that the wait time is insufficient, as the next request after the wait period fails with a rate limit error.

How to replicate:

client = Client(**config, options={'handle_rate_limit': True})
...
for chunk_ids in split_list(ids, 20):
    analytics_data: List[dict] = Campaign.all_stats(
        account,
        chunk_ids,
        [METRIC_GROUP.ENGAGEMENT, METRIC_GROUP.BILLING],
        granularity=GRANULARITY.TOTAL,
        start_time=export_range.start_date,
        end_time=export_range.end_date
    )

The above code is ran for an export range of one day. It is repeated for multiple accounts. This is the output when a rate limit is reached:

time=2022-07-26 17:22:50 name=twitter_ads.http level=WARNING message=Request reached Rate Limit: resume in 592 seconds
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "*********/twitter-ads-env/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "*********/twitter-ads-env/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "*********/twitter-ads-env/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "*********/twitter-ads-env/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "*********/utils/backfill.py", line 110, in run_backfill
    for entity_data in results:
  File "*********/utils/backfill.py", line 74, in _download_data
    analytics_data: List[dict] = Campaign.all_stats(
  File "*********/twitter-ads-env/lib/python3.8/site-packages/twitter_ads/analytics.py", line 78, in all_stats
    response = Request(account.client, 'get', resource, params=params).perform()
  File "*********/twitter-ads-env/lib/python3.8/site-packages/twitter_ads/http.py", line 70, in perform
    raise Error.from_response(response)
twitter_ads.error.RateLimit: <RateLimit object at 0x7f2b79a75520 code=429 details=None>

Expected behaviour:

The library should wait the appropriate amount of time before attempting to call the API again to completely avoid RateLimit errors.

Observed behavior:

At the current state, the handle_rate_limit option serves no purpose. Despite waiting a certain amount of time, the rate limit error is encountered anyway.

tushdante commented 2 years ago

@pep4eto1211 i believe this is this related to #298. can you confirm?