Timeout on souffleing top 100

[zhammer]

    result = handler(input['event'], context)
  File "./playlist_souffle/delivery/aws_lambda/souffle_playlist.py", line 67, in handler

    souffled_playlist_uri = souffle_playlist(spotify, playlist_uri, shuffle_by)
  File "./playlist_souffle/use_cases/souffle_playlist.py", line 29, in souffle_playlist

    related_by=shuffle_by
  File "./playlist_souffle/gateways/spotify_util.py", line 30, in inner

    return func(*args, **kwargs)   

  File "./playlist_souffle/gateways/spotify.py", line 80, in fetch_related_tracks_by_track

    tracks
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/concurrent/futures/_base.py", line 556, in result_iterator

    yield future.result()

  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/concurrent/futures/_base.py", line 398, in result

    return self.__get_result()

  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/concurrent/futures/_base.py", line 357, in __get_result

    raise self._exception                                          

  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/concurrent/futures/thread.py", line 55, in run

    result = self.fn(*self.args, **self.kwargs)
  File "./playlist_souffle/gateways/spotify.py", line 79, in <lambda>

    lambda track: (track, self.fetch_related_tracks(track, related_by)),
  File "./playlist_souffle/gateways/spotify_util.py", line 30, in inner

    return func(*args, **kwargs)
  File "./playlist_souffle/gateways/spotify.py", line 62, in fetch_related_tracks

    related_track_data = self._spotify.album_tracks(track.album)['items']
  File "/Users/zhammer/code/playlist-souffle/venv/lib/python3.6/site-packages/spotipy/client.py", line 316, in album_tracks

    offset=offset)
  File "/Users/zhammer/code/playlist-souffle/venv/lib/python3.6/site-packages/spotipy/client.py", line 146, in _get

    return self._internal_call('GET', url, payload, kwargs)                                                                                                                                            

  File "/Users/zhammer/code/playlist-souffle/venv/lib/python3.6/site-packages/spotipy/client.py", line 108, in _internal_call

    r = self._session.request(method, url, headers=headers, proxies=self.proxies, **args)

  File "/Users/zhammer/code/playlist-souffle/venv/lib/python3.6/site-packages/requests/api.py", line 58, in request

    return session.request(method=method, url=url, **kwargs)                                                                  
  File "/Users/zhammer/code/playlist-souffle/venv/lib/python3.6/site-packages/requests/sessions.py", line 508, in request

    resp = self.send(prep, **send_kwargs)                                                                                   
  File "/Users/zhammer/code/playlist-souffle/venv/lib/python3.6/site-packages/requests/sessions.py", line 618, in send

    r = adapter.send(request, **kwargs)

  File "/Users/zhammer/code/playlist-souffle/venv/lib/python3.6/site-packages/requests/adapters.py", line 508, in send                     

    raise ConnectionError(e, request=request)                      

requests.exceptions.ConnectionError: HTTPSConnectionPool(host='api.spotify.com', port=443): Max retries exceeded with url: /v1/albums/6Eo5EkmdLvZrONzi046iC2/tracks/?limit=50&offset=0 (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x10a4272e8>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',))

[zhammer]

(venv) Zachs-MBP:playlist-souffle zhammer$ sls invoke local -f souffle-playlist -d "{\"headers\": {\"Authorization\": \"Bearer $ACCESS\"}, \"body\": \"playlistUri=spotify%3Auser%3Azachthehammer%3Aplaylist%3A5UwmRyJAt4LyitmWXITjJw&shuffleBy=album\"}"
Fetch user id: 0.6363968849182129

Fetch playlist: 1.521245002746582

Fetch Related tracks: 4.676896095275879

Souffle function: 0.0032722949981689453

{
    "statusCode": 500,
    "body": "{\"message\": \"Encountered Spotify api error. Message: \\\"https://api.spotify.com/v1/users/zachthehammer/playlists:\\n API rate limit exceeded\\\".\"}"
}

[zhammer]

I can get several albums with one request https://developer.spotify.com/documentation/web-api/reference/albums/get-several-albums/ but can't get the top tracks of several aritsts with one request unfortunately.

[zhammer]

Update: this isn't actually timing out, basically using concurrent request threads w/ enough requests causes rate limit to be exceeded, which throws an exception. (spotipy already has some built in retry logic).

this could be resolved by not doing concurrent fetches, but that'd be super slow. will keep looking into solutions