Yahoo Fantasy API Potential Rate Limits

[RyanCaldwell360]

@uberfastman I'm wondering if you ever ran into any issues with the number of requests you could make during your development and use of this repo. I haven't been able to find any documentation with Yahoo on rate limits, so I'm just curious if you had any issues as well.

An example where this might be a problem is when I'm trying to collect matchup data for each season, and for each week within those seasons. I'll repeatedly make a call to query.get_league_scoreboard_by_week(wk). If I try to iterate over 2 seasons or so it will result in the following connection error:

http.client.RemoteDisconnected: Remote end closed connection without response

[uberfastman]

@RyanCaldwell360 yes I have. Unfortunately Yahoo has always had some fairly stringent rate limits (at least compared to other fantasy platform APIs) for their fantasy sports API, although I do not know precisely what the limits are. In YFPY I've attempted to mitigate it by catching response code 999, which is what Yahoo returns when you exceed their rate limit. I've typically found that once I hit that I can wait around 10-15 minutes and then I'm fine again.

That being said, the 999 rate limit response code is handled, I've never run into the above error you mentioned before. Could you post the full stack trace of the error and the code snippet that causes it?

[RyanCaldwell360]

Here is the code snippet:

def induce_connection_error(secrets_path, league_id, game_code, game_id):
    # initialize Yahoo Query for this season
    query = YahooFantasySportsQuery(auth_dir=secrets_path,
                                    league_id=league_id,
                                    game_code=game_code,
                                    game_id=game_id,
                                    browser_callback=False)

    for wk in [x for x in range(1,17)]:
        scoreboard = query.get_league_scoreboard_by_week(wk)

induce_connection_error(secrets_path, league_id, game_code, game_id)

Here is the stack trace:

Traceback (most recent call last): File "/home/ryancaldwell/miniconda3/envs/myenv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 790, in urlopen response = self._make_request( File "/home/ryancaldwell/miniconda3/envs/myenv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 536, in _make_request response = conn.getresponse() File "/home/ryancaldwell/miniconda3/envs/myenv/lib/python3.10/site-packages/urllib3/connection.py", line 461, in getresponse httplib_response = super().getresponse() File "/home/ryancaldwell/miniconda3/envs/myenv/lib/python3.10/http/client.py", line 1375, in getresponse response.begin() File "/home/ryancaldwell/miniconda3/envs/myenv/lib/python3.10/http/client.py", line 318, in begin version, status, reason = self._read_status() File "/home/ryancaldwell/miniconda3/envs/myenv/lib/python3.10/http/client.py", line 287, in _read_status raise RemoteDisconnected("Remote end closed connection without" http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/ryancaldwell/miniconda3/envs/myenv/lib/python3.10/site-packages/requests/adapters.py", line 486, in send resp = conn.urlopen( File "/home/ryancaldwell/miniconda3/envs/myenv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 844, in urlopen retries = retries.increment( File "/home/ryancaldwell/miniconda3/envs/myenv/lib/python3.10/site-packages/urllib3/util/retry.py", line 470, in increment raise reraise(type(error), error, _stacktrace) File "/home/ryancaldwell/miniconda3/envs/myenv/lib/python3.10/site-packages/urllib3/util/util.py", line 38, in reraise raise value.with_traceback(tb) File "/home/ryancaldwell/miniconda3/envs/myenv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 790, in urlopen response = self._make_request( File "/home/ryancaldwell/miniconda3/envs/myenv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 536, in _make_request response = conn.getresponse() File "/home/ryancaldwell/miniconda3/envs/myenv/lib/python3.10/site-packages/urllib3/connection.py", line 461, in getresponse httplib_response = super().getresponse() File "/home/ryancaldwell/miniconda3/envs/myenv/lib/python3.10/http/client.py", line 1375, in getresponse response.begin() File "/home/ryancaldwell/miniconda3/envs/myenv/lib/python3.10/http/client.py", line 318, in begin version, status, reason = self._read_status() File "/home/ryancaldwell/miniconda3/envs/myenv/lib/python3.10/http/client.py", line 287, in _read_status raise RemoteDisconnected("Remote end closed connection without" urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/mnt/c/Users/ryanc/Documents/Projects/Fantasy_Football/fantasy_football_app/src/backend/data_extraction/yahoo_data_script.py", line 186, in induce_connection_error("/mnt/c/Users/ryanc/Documents/Projects/Fantasy_Football", "527783", "nfl", "331") File "/mnt/c/Users/ryanc/Documents/Projects/Fantasy_Football/fantasy_football_app/src/backend/data_extraction/yahoo_data_script.py", line 171, in induce_connection_error scoreboard = query.get_league_scoreboard_by_week(wk) File "/home/ryancaldwell/miniconda3/envs/myenv/lib/python3.10/site-packages/yfpy/query.py", line 1591, in get_league_scoreboard_by_week f"https://fantasysports.yahooapis.com/fantasy/v2/league/{self.get_league_key()}/scoreboard;" File "/home/ryancaldwell/miniconda3/envs/myenv/lib/python3.10/site-packages/yfpy/query.py", line 775, in get_league_key return f"{self.get_game_metadata_by_game_id(self.game_id).game_key}.l.{self.league_id}" File "/home/ryancaldwell/miniconda3/envs/myenv/lib/python3.10/site-packages/yfpy/query.py", line 604, in get_game_metadata_by_game_id return self.query( File "/home/ryancaldwell/miniconda3/envs/myenv/lib/python3.10/site-packages/yfpy/query.py", line 263, in query response = self.get_response(url) File "/home/ryancaldwell/miniconda3/envs/myenv/lib/python3.10/site-packages/yfpy/query.py", line 181, in get_response response: Response = self.oauth.session.get(url, params={"format": "json"}) File "/home/ryancaldwell/miniconda3/envs/myenv/lib/python3.10/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, kwargs) File "/home/ryancaldwell/miniconda3/envs/myenv/lib/python3.10/site-packages/rauth/session.py", line 358, in request return super(OAuth2Session, self).request(method, url, req_kwargs) File "/home/ryancaldwell/miniconda3/envs/myenv/lib/python3.10/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, send_kwargs) File "/home/ryancaldwell/miniconda3/envs/myenv/lib/python3.10/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, kwargs) File "/home/ryancaldwell/miniconda3/envs/myenv/lib/python3.10/site-packages/requests/adapters.py", line 501, in send raise ConnectionError(err, request=request) requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

[uberfastman]

Thanks for the added info @RyanCaldwell360. I just tested out your code snippet (for a different league obviously) and it runs repeatedly and promptly without any issues, so I can't seem to reproduce that error you're getting with the code you provided.

That being said... could you run pip freeze in your command line and post the output of your installed packages? Looking through the stack trace, it seems that you are using a virtual environment managed by miniconda... historically in my career I've run into the occasional issue with miniconda/anaconda where the packages and/or Python distribution built into them are not identical to the plain ones you might get if you installed/managed them yourself, so it's possible that there is some dependency mismatch in a transitive dependency, in particular I'm curious about the versions of requests, urllib3, and rauth.

It seems that the issue is originating from one of those dependencies, not YFPY itself, so if one of those is somehow set differently then that might cause some issue. Did you install all the YFPY dependencies by running pip install -r requirements.txt in your virtualenv?

[RyanCaldwell360]

@uberfastman here is the list of packages from pip freeze within my virtual environment:

bandit==1.7.5 certifi==2023.11.17 charset-normalizer==3.3.2 coverage==7.3.2 exceptiongroup==1.2.0 flake8==6.1.0 gitdb==4.0.11 GitPython==3.1.40 httplib2==0.22.0 idna==3.6 iniconfig==2.0.0 markdown-it-py==3.0.0 mccabe==0.7.0 mdurl==0.1.2 numpy==1.26.2 oauth2==1.9.0.post1 packaging==23.2 pandas==2.1.4 pbr==6.0.0 pluggy==1.3.0 pyaml==23.9.7 pycodestyle==2.11.1 pyflakes==3.1.0 Pygments==2.17.2 pyparsing==3.1.1 pytest==7.4.3 python-dateutil==2.8.2 python-dotenv==1.0.0 pytz==2023.3.post1 PyYAML==6.0.1 rauth==0.7.3 requests==2.31.0 rich==13.7.0 ruamel.yaml==0.18.5 ruamel.yaml.clib==0.2.8 six==1.16.0 smmap==5.0.1 stevedore==5.1.0 stringcase==1.2.0 tomli==2.0.1 tzdata==2023.3 urllib3==2.1.0 yahoo-oauth==2.0 yfpy==15.0.3

[RyanCaldwell360]

Did you install all the YFPY dependencies by running pip install -r requirements.txt in your virtualenv?

I did, and it now seems to have resolved this problem!

[uberfastman]

Hooray, thanks for confirming @RyanCaldwell360! Glad it's sorted!