Gracefully handle stravalib/Strava API rate limiting.

[aaron-schroeder]

Currently, this can happen if a user is trying to add more activities at once than the rate limit allows, and generally if they are doing a lot of interacting with the app.

Update 4/7/23: A partial mitigation strategy is now in place on a branch - each api json response is saved in its entirety to the database on the first request.

In dev mode, I like the idea of just displaying dummy data (probably from the mock stravalib).

In production, I suppose I should handle things more gracefully: just messages when data is missing; potentially info about the rate limiting itself; a countdown on the page showing how many more requests are remaining.

Sub task: #55

Strava API request limits:

• My production app
  • 600 requests every 15 minutes
  • 30,000 per day
  • Note: it is feasible that I could get ~600 activities fully imported in just two batches. If I started backfilling activities right before a quarter hour, I would hit the limit which would subsequently reset.
    I'm thinking each activity import hits the strava api twice, meaning 300 imports per batch. Also I am ignoring the initial requests to get summary data in big batches.
  • My development app
  • 100 requests every 15 minutes
  • 1,000 requests per day

Stravalib seems to keep track of some rate limits (re stravalib.exc.RateLimitExceeded). I want verify for myself that they work accurately and reflect the rate limits on my account.

I want to see if there is specific, actionable information about rate limits in Stravalib's exceptions that I can use to make my retry scheme smarter. I just need to wait to tinker until I'm rate-limited.

I handle this in the very specific case of the batch-add-activity task. But it exists in lots of places where I haven't dealt with it.

Stravalib points of contact

• strava_api.manage: Each account uses stravalib to display basic summary info
  • Improve handling: models.StravaAccount returns dummy information if it finds itself rate limited. Don't love it.
• pages.strava_table: At least one stravalib request per page view
  • Improve handling: preventupdate if rate limited.
  • Desired:
  • show something in the able to indicate status and hide the batch-import stuff
  • OR hide the table and indicate status where the batch-import stuff would go
  • Wrinkle: the table sends at least one new request each time it is sorted, paged, (and soon filtered?)
• pages.strava_activity
  • client.get_activity_streams() in layout()
  • client.get_activity() in layout()
  • But generally, if someone is above their rate limit, what would the value be of showing a dashboard at all? It's 100% strava data.
• pages.saved_activity
  • client.get_activity_streams() in layout()
  • Does not use get_activity for scalar data - relies on saved activity
  • Handled: shows available DB summary info, with a message indicating rate limiting where the figures would go.
• tasks.async_save_activity: does it multiple times per task

Notes

Successful response example

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Date: Tue, 10 Oct 2020 20:11:01 GMT
X-Ratelimit-Limit: 600,30000
X-Ratelimit-Usage: 314,27536

Rate-limited response example

HTTP/1.1 429 Too Many Requests
Content-Type: application/json; charset=utf-8
Date: Tue, 10 Oct 2020 20:11:05 GMT
X-Ratelimit-Limit: 600,30000
X-Ratelimit-Usage: 692,29300

Rate-limited json response b'{"message":"Rate Limit Exceeded","errors":[{"resource":"Application","field":"overall rate limit","code":"exceeded"}]}'

[aaron-schroeder]

Notes about strava batch-save

One interesting wrinkle is figuring out whether I'll regain access in ~15 minutes or the next day, and planning retries accordingly. Currently, my retry scheme will definitely fail if the daily rate limit is hit, with a possibility of giving up on even a 15-minute rate limit. I don't necessarily mind giving up, but if it happens, I want it to happen loudly so the user knows their activities weren't added.

[aaron-schroeder]

Create a flask endpoint that shows rate-limit status

I'm a visual person and want to debug that way.

Start off with an endpoint that answers "is this app rate-limited?"

Looking under the hood of stravalib, it seems that:

• TLDR: to access the state of the rate limits:

  >>> client.protocol.rate_limiter.rules[0].rate_limits
  {
    'short': {
        'usageFieldIndex': 0,
        'usage': 1,
        'limit': 100,
        'time': 900,
        'lastExceeded': None
    }, 
    'long': {
        'usageFieldIndex': 1,
        'usage': 581, 
        'limit': 1000,
        'time': 86400,
        'lastExceeded': None
    }
  }

  • Note: this gives the status as of the most recent request the client performed. If the client hasn't performed any requests yet, the rule will be in the state it was initiated in, which shows a higher limit than my dev app actually allows.
• a RateLimiter is instantiated with some rules, then passed to the low-level class ApiV3.
• ApiV3 calls the RateLimiter with the raw response headers as args
• The RateLimiter calls each of its rules using the raw headers as args.
• The rules as defined by DefaultRateLimiter consist of one XRateLimitRule for both the short- and long-term rate limits. Its default limits are correct for my production app, but higher than the limits for my dev app. I don't think this really matters, as the limiters and rules just react to whatever response headers they get.

[aaron-schroeder]

I guess the question now that I can check the rate limit status of the response in multiple ways...what do I want to do w that info? Here are some scenarios that I may encounter during a large backfill of strava activities:

https://github.com/aaron-schroeder/distilling-flask/blob/e0f7df51c0851a0b4b034fb1fb911b76bc03d551/distilling_flask/io_storages/strava/models.py#L83-#L102

• models.StravaImportStorage.iterkeys()
  • in a loop until an empty list is returned as the response json:

    resp = client.get('/athlete/activities', page=page, per_page=200)
    if len(activity_list := resp.json()) and isinstance(activity_list, list):
    for activity_summary in activity_list:
        yield activity_summary['id']
    page += 1

  • Not very request-heavy: num_requests = math.ceil(num_activities / 200)
• models.StravaImportStorage.get_data()
  • for each newly-discovered key from iterkeys:

  • summary_resp = client.get(f'/activities/{key}', include_all_efforts=True)
    stream_data = stream_resp.json()
    # data['streams'] = json.dumps(stream_data).encode('utf-8')
    data['streams_compressed'] = zlib.compress(json.dumps(stream_data).encode('utf-8'))

  • stream_resp = client.get(
      f'/activities/{key}/streams/time,latlng,distance,altitude,'
      f'velocity_smooth,heartrate,cadence,watts,temp,moving,grade_smooth')
    stream_data = stream_resp.json()
    # data['streams'] = json.dumps(stream_data).encode('utf-8')
    data['streams_compressed'] = zlib.compress(json.dumps(stream_data).encode('utf-8'))

• models.StravaImportStorage.validate_connection()

  • # TODO GH48: Handle rate limiting here and wherever else it occurs.
    # logger.debug(f'Test connection to bucket {self.bucket} with prefix {self.prefix}')
    client.get('/athlete')
    # client.get_activities(limit=5)

  • This is currently a checkpoint; nothing is done with the response. The expectation is that the StravaRateLimitMonitor instance created within StravaApiClient.get() will raise errors.
• views.show_strava_status

  • @strava.route('/status')
    def show_strava_status():
    # TODO: Find a way to RL display status without burning through
    # a dummy request every time. I'm thinking it can be a database 
    # or redis entry.
    if flag_set('ff_rename'):
          from distilling_flask.io_storages.strava.util import StravaRateLimitMonitor
          s = db.session.scalars(db.select(StravaImportStorage)).first()
          response = s.get('/athlete')
          monitor = StravaRateLimitMonitor(response)
          # TODO: Finish

• I think that seriously might be it at the moment.