freezing-sync is hitting Strava rate limits even before competition starts - causing new registrations to fail

[obscurerichard]

The calls that freezing-sync makes are triggering the Strava API rate limiter. This is a big problem as the competition has not even started for 2021, and if freezing-sync makes us run out of rate limit quota, registration won't work either.

We were seeing this the week of Thanksgiving too but resetting the database and giving the system a 24 hour break temporarily gave us a reprieve.

Logs for freezing-sync look like:

freezing-sync        | INFO     [freezing.sync.subscribe] Fetching detailed activity athlete_id=4571114, activity_id=6346487196
freezing-sync        | INFO     [freezing.sync.data] access token for athlete 4571114 is still valid
freezing-sync        | INFO     [freezing.sync.utils.cache.CachingActivityFetcher] [CACHE-MISS] Fetching activity detail for 6346487196
freezing-sync        | INFO     [stravalib.protocol.ApiV3] GET 'https://www.strava.com/api/v3/activities/6346487196' with params {'include_all_efforts': True}
freezing-sync        | ERROR    [freezing.sync.subscribe] Error fetching/writing activity detail 6346487196, athlete 4571114
freezing-sync        | Traceback (most recent call last):
freezing-sync        |   File "/usr/local/lib/python3.6/dist-packages/freezing/sync/data/activity.py", line 377, in fetch_and_store_activity_detail
freezing-sync        |     use_cache=use_cache,
freezing-sync        |   File "/usr/local/lib/python3.6/dist-packages/freezing/sync/utils/cache.py", line 205, in fetch
freezing-sync        |     only_cache=only_cache,
freezing-sync        |   File "/usr/local/lib/python3.6/dist-packages/freezing/sync/utils/cache.py", line 142, in retrieve_object_json
freezing-sync        |     athlete_id=athlete_id, object_id=object_id
freezing-sync        |   File "/usr/local/lib/python3.6/dist-packages/freezing/sync/utils/cache.py", line 181, in download_object_json
freezing-sync        |     "/activities/{id}", id=object_id, include_all_efforts=True
freezing-sync        |   File "/usr/local/lib/python3.6/dist-packages/stravalib/protocol.py", line 299, in get
freezing-sync        |     return self._request(url, params=params, check_for_errors=check_for_errors, use_webhook_server=use_webhook_server)
freezing-sync        |   File "/usr/local/lib/python3.6/dist-packages/stravalib/protocol.py", line 218, in _request
freezing-sync        |     self.rate_limiter(raw.headers)
freezing-sync        |   File "/usr/local/lib/python3.6/dist-packages/stravalib/util/limiter.py", line 254, in __call__
freezing-sync        |     r(args)
freezing-sync        |   File "/usr/local/lib/python3.6/dist-packages/stravalib/util/limiter.py", line 108, in __call__
freezing-sync        |     self._check_limit_rates(limit)
freezing-sync        |   File "/usr/local/lib/python3.6/dist-packages/stravalib/util/limiter.py", line 126, in _check_limit_rates
freezing-sync        |     self._raise_rate_limit_exception(limit['limit'], limit['time'])
freezing-sync        |   File "/usr/local/lib/python3.6/dist-packages/stravalib/util/limiter.py", line 141, in _raise_rate_limit_exception
freezing-sync        |     limit=limit_rate, timeout=timeout)
freezing-sync        | stravalib.exc.RateLimitExceeded
freezing-sync        | ERROR    [freezing.sync.subscribe] Error procesing message, will requeue w/ delay.

Right now we don't have great visibility on the logs or what is causing the high volume of API calls, the logs are rolling out of visibility too fast. We may need to re-enable log shipping to a service such as Papertrail (which is configured but not working now) or to CloudWatch logs (which would be the easiest replacement since we already have this running in an AWS account) to get better visibility on the logs. See https://github.com/freezingsaddles/freezing-compose/issues/18 for that issue.