NBA API: timeout for /stats endpoints

sallareznov commented 1 year ago

Hello,

First, I cannot thank you enough for this wonderful library.

I believe this has already been addressed in previous issues, but still shooting my shot. I'm having an issue with the NBA API for an app deployed on Heroku:

the requests to /stats endpoints time out in production, while working locally
the requests to /live endpoints work fine

I'm aware that the NBA block their API for Heroku instances, but I wanted to know if you were able to find a long-term workaround for this.

I'm using the version 1.1.13 of the library.

Thanks in advance :)

zbleszczak commented 1 year ago

Hi, basically i have the same issue but not with /stats but with 'PlayerCareerStats' endpoint. I heard that upgrading versions to the newest would help, but i already have this done. I made request timeout longer to 90s but its still Read timed out situation every 9/10 situations.

sallareznov commented 1 year ago

Hi, basically i have the same issue but not with /stats but with 'PlayerCareerStats' endpoint. I heard that upgrading versions to the newest would help, but i already have this done. I made request timeout longer to 90s but its still Read timed out situation every 9/10 situations.

Yeah exactly :/

akacaio commented 1 year ago

Same here. I'm trying to access the traditional boxscore endpoint and this happened to me.

pauldevos commented 1 year ago

I believe the header requirements by nba.com/stats has changed. The parameters are often the same with a few different API changes, e.g. boxscoreadvancedv2 => boxscoreadvancedv3 (version 3)

https://stats.nba.com/stats/boxscoreadvancedv3?
    GameID=0022200701&LeagueID=00&endPeriod=0&endRange=28800&rangeType=0&startPeriod=0&startRange=0

But the parameters are the same, but doesn't work, gets a timeout issue. Which likely suggests the request header is an issue. I've tried 12 different endpoints and all get the same error. I'm using my laptop, so not a chunk of blocked AWS or Azure resources (which is a common issue with scraping).

When visiting the webpage directly and using the inspect element, I find the API and the Header. I then put those into code. But still having issues. Makes me wonder if need to capture the Response Cookie??

Here's an endpoint that is taken right from the inspector:

https://stats.nba.com/stats/leaguedashplayerptshot?CloseDefDistRange=&College=&Conference=&Country=&DateFrom=&DateTo=&Division=&DraftPick=&DraftYear=&DribbleRange=&GameScope=&GameSegment=&GeneralRange=Overall&Height=&LastNGames=0&LeagueID=00&Location=&Month=0&OpponentTeamID=0&Outcome=&PORound=0&PaceAdjust=N&PerMode=Totals&Period=0&PlayerExperience=&PlayerPosition=&PlusMinus=N&Rank=N&Season=2022-23&SeasonSegment=&SeasonType=Regular%20Season&ShotClockRange=&ShotDistRange=&StarterBench=&TeamID=0&TouchTimeRange=&VsConference=&VsDivision=&Weight=

Put in header form:

headers = {
'Host': 'stats.nba.com',
'Connection': 'keep-alive',
'Accept': 'application/json, text/plain, */*',
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36',
'Origin': 'https://www.nba.com',
'Referer': 'https://www.nba.com/',
'Accept-Encoding': 'gzip, deflate, br',
'Accept-Language': 'en-US,en;q=0.5',
'Sec-Fetch-Dest': 'empty',
'Sec-Fetch-Mode': 'cors',
'Sec-Fetch-Site': 'same-site',
'Sec-GPC': '1'
}

This is the error message:

JSONDecodeError: Expecting value: line 1 column 1 (char 0)
During handling of the above exception, another exception occurred:

JSONDecodeError                           Traceback (most recent call last)
Input In [13], in <cell line: 22>()
     50 url = f"{BASE_URL}{API_ENDPOINT}?{urlencode(params)}"
     51 print(url)
---> 52 r = requests.get(url, headers=headers).json()
     54 # names directory for file, placed file on given features and then checks for/creates director
     55 PATH = NBA_DIR / f"{API_ENDPOINT}/{Season}/{SeasonType}/"

File ~/.pyenv/versions/3.10.2/lib/python3.10/site-packages/requests/models.py:917, in Response.json(self, **kwargs)
    915     raise RequestsJSONDecodeError(e.message)
    916 else:
--> 917     raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)

JSONDecodeError: [Errno Expecting value] upstream request timeout: 0

Anyone else had success?

pauldevos commented 1 year ago

I've also seen a 504 response, again this is using local [Mac], Jupyter Notebook Python requests package. But in actually going to the nba.com website, seeing the API endpoint in the Network traffic, it's the same. So something to do with a cookie and/or header is my guess.

Here I get a few alternate 200 and 504s.

7 of 751 games
https://stats.nba.com/stats/boxscoretraditionalv2?EndPeriod=10&EndRange=43800&GameID=0022200294&RangeType=0&Season=2022-23&SeasonType=Regular+Season&StartPeriod=1&StartRange=0
200
<Response [200]>
Slept: 1.76 seconds

8 of 751 games
https://stats.nba.com/stats/boxscoretraditionalv2?EndPeriod=10&EndRange=43800&GameID=0022200142&RangeType=0&Season=2022-23&SeasonType=Regular+Season&StartPeriod=1&StartRange=0
504
<Response [504]>

Then interrupting and trying again, I get a 504 and kills it right away:

Games: 751
1 of 751 games
https://stats.nba.com/stats/boxscoretraditionalv2?EndPeriod=10&EndRange=43800&GameID=0022200657&RangeType=0&Season=2022-23&SeasonType=Regular+Season&StartPeriod=1&StartRange=0
504
---------------------------------------------------------------------------
JSONDecodeError                           Traceback (most recent call last)
File ~/.pyenv/versions/3.10.2/lib/python3.10/site-packages/requests/models.py:910, in Response.json(self, **kwargs)
    909 try:
--> 910     return complexjson.loads(self.text, **kwargs)
    911 except JSONDecodeError as e:
    912     # Catch JSON-related errors and raise as requests.JSONDecodeError
    913     # This aliases json.JSONDecodeError and simplejson.JSONDecodeError

File ~/.pyenv/versions/3.10.2/lib/python3.10/json/__init__.py:346, in loads(s, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
    343 if (cls is None and object_hook is None and
    344         parse_int is None and parse_float is None and
    345         parse_constant is None and object_pairs_hook is None and not kw):
--> 346     return _default_decoder.decode(s)
    347 if cls is None:

File ~/.pyenv/versions/3.10.2/lib/python3.10/json/decoder.py:337, in JSONDecoder.decode(self, s, _w)
    333 """Return the Python representation of ``s`` (a ``str`` instance
    334 containing a JSON document).
    335 
    336 """
--> 337 obj, end = self.raw_decode(s, idx=_w(s, 0).end())
    338 end = _w(s, end).end()

File ~/.pyenv/versions/3.10.2/lib/python3.10/json/decoder.py:355, in JSONDecoder.raw_decode(self, s, idx)
    354 except StopIteration as err:
--> 355     raise JSONDecodeError("Expecting value", s, err.value) from None
    356 return obj, end

JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Edit/Update:

I put a sleeper of > 3.5 seconds and made sure to have the latest endpoint, e.g. v3 for boxscoretraditionalv3 and it seems to be working perfectly.

Games: 751
1 of 751 games
https://stats.nba.com/stats/boxscoretraditionalv3?EndPeriod=10&EndRange=43800&GameID=0022200657&RangeType=0&Season=2022-23&SeasonType=Regular+Season&StartPeriod=1&StartRange=0
200
Slept: 3.91 seconds

2 of 751 games
https://stats.nba.com/stats/boxscoretraditionalv3?EndPeriod=10&EndRange=43800&GameID=0022200641&RangeType=0&Season=2022-23&SeasonType=Regular+Season&StartPeriod=1&StartRange=0
200
Slept: 3.52 seconds

3 of 751 games
https://stats.nba.com/stats/boxscoretraditionalv3?EndPeriod=10&EndRange=43800&GameID=0022200240&RangeType=0&Season=2022-23&SeasonType=Regular+Season&StartPeriod=1&StartRange=0
200
Slept: 3.66 seconds

4 of 751 games
https://stats.nba.com/stats/boxscoretraditionalv3?EndPeriod=10&EndRange=43800&GameID=0022200102&RangeType=0&Season=2022-23&SeasonType=Regular+Season&StartPeriod=1&StartRange=0
200
Slept: 3.82 seconds

5 of 751 games
https://stats.nba.com/stats/boxscoretraditionalv3?EndPeriod=10&EndRange=43800&GameID=0022200358&RangeType=0&Season=2022-23&SeasonType=Regular+Season&StartPeriod=1&StartRange=0
200
Slept: 3.57 seconds

6 of 751 games
https://stats.nba.com/stats/boxscoretraditionalv3?EndPeriod=10&EndRange=43800&GameID=0022200308&RangeType=0&Season=2022-23&SeasonType=Regular+Season&StartPeriod=1&StartRange=0
200
Slept: 3.56 seconds

7 of 751 games
https://stats.nba.com/stats/boxscoretraditionalv3?EndPeriod=10&EndRange=43800&GameID=0022200294&RangeType=0&Season=2022-23&SeasonType=Regular+Season&StartPeriod=1&StartRange=0
200

efiul commented 1 year ago

Hi, I have the same issue.

the requests to /stats endpoints (for example 'https://stats.nba.com/stats/commonteamroster') time out in production (AWS), while working locally.
the requests to /live endpoints work fine (for example 'https://cdn.nba.com/static/json/liveData/scoreboard/todaysScoreboard_00.json')

I am using version 1.1.14 of the library. I tried to use different headers with no success.

Has anyone resolved this issue? Thanks.

Justin19960919 commented 1 year ago

Same issue here as @efiul

OnerInce commented 1 year ago

Since AWS/GCP/Heroku (maybe others too) IP addresses are blocked by NBA, a solution doesn't seem possible using Headers. Only possible solution as far as I know is using proxies while making requests. If you already have a VPN provider, you just need to dig some documentation to fetch proxy endpoint information. That way I was able to use /stats endpoints on AWS Lambda.

JoKuebler commented 1 year ago

Some endpoints were working for me today but stopped after a bit. I turned on my VPN and it worked again. Seems like my IP gets blocked after a while which is weird since I didn't use it extensively.

Justin19960919 commented 1 year ago

does anyone know how to use proxies? I looked up free proxies yesterday but none of them worked. I passed in "{proxy}:{port}" in the proxy param for the call but connection still failed

OnerInce commented 1 year ago

does anyone know how to use proxies? I looked up free proxies yesterday but none of them worked. I passed in "{proxy}:{port}" in the proxy param for the call but connection still failed

Its so difficult to find a free working proxy from the web. You can check out paid proxy services

caseta commented 11 months ago

Anyone figure this out with Heroku?

ckim16 commented 10 months ago

has anyone resolved with streamlit by any chance? it works fine on local but it errors out when i try to deploy

requests.exceptions.ReadTimeout: This app has encountered an error. The original error message is redacted to prevent data leaks. Full error details have been recorded in the logs (if you're on Streamlit Cloud, click on 'Manage app' in the lower right of your app).
Traceback:
File "/home/adminuser/venv/lib/python3.9/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 541, in _run_script
    exec(code, module.__dict__)
File "/mount/src/nba_stat/main.py", line 39, in <module>
    career_stat =  playercareerstats.PlayerCareerStats(player_id=player_id).get_data_frames()
File "/home/adminuser/venv/lib/python3.9/site-packages/nba_api/stats/endpoints/playercareerstats.py", line 34, in __init__
    self.get_request()
File "/home/adminuser/venv/lib/python3.9/site-packages/nba_api/stats/endpoints/playercareerstats.py", line 37, in get_request
    self.nba_response = NBAStatsHTTP().send_api_request(
File "/home/adminuser/venv/lib/python3.9/site-packages/nba_api/library/http.py", line 130, in send_api_request
    response = requests.get(url=base_url, params=parameters, headers=request_headers, proxies=proxies, timeout=timeout)
File "/home/adminuser/venv/lib/python3.9/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
File "/home/adminuser/venv/lib/python3.9/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
File "/home/adminuser/venv/lib/python3.9/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
File "/home/adminuser/venv/lib/python3.9/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
File "/home/adminuser/venv/lib/python3.9/site-packages/requests/adapters.py", line 532, in send
    raise ReadTimeout(e, request=request)

jacasta2 commented 6 months ago

I had a mix experience with Streamlit, sometimes it works, but some others it doesn't. I work mainly with the endpoints leaguegamefinder.LeagueGameFinder() and boxscoretraditionalv2.BoxScoreTraditionalV2().

Now, I believe I'm facing this issue with GitHub Actions. I have a scheduled action to fetch recent games using the endpoint leaguegamefinder.LeagueGameFinder(). I'm testing it to fetch info from only 6 games (a 6-row dataframe), but I'm getting the following error:

raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='stats.nba.com', port=443): Read timed out. (read timeout=30)

This is the full trace:

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/urllib3/connectionpool.py", line 467, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/opt/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/urllib3/connectionpool.py", line 462, in _make_request
    httplib_response = conn.getresponse()
  File "/opt/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/http/client.py", line 1375, in getresponse
    response.begin()
  File "/opt/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
  File "/opt/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/http/client.py", line 279, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/opt/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/socket.py", line 705, in readinto
    return self._sock.recv_into(b)
  File "/opt/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/ssl.py", line 1274, in recv_into
    return self.read(nbytes, buffer)
  File "/opt/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/ssl.py", line 1130, in read
    return self._sslobj.read(len, buffer)
TimeoutError: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
  File "/opt/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/urllib3/connectionpool.py", line 799, in urlopen
    retries = retries.increment(
  File "/opt/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/urllib3/util/retry.py", line 550, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/opt/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/urllib3/packages/six.py", line 770, in reraise
    raise value
  File "/opt/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/urllib3/connectionpool.py", line 715, in urlopen
    httplib_response = self._make_request(
  File "/opt/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/urllib3/connectionpool.py", line 469, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/opt/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/urllib3/connectionpool.py", line 358, in _raise_timeout
    raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='stats.nba.com', port=443): Read timed out. (read timeout=30)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/runner/work/***/***/src/fetch_data.py", line 326, in <module>
    fetch_recent_games()
  File "/home/runner/work/***/***/src/fetch_data.py", line 260, in fetch_recent_games
    new_data_regular_season = leaguegamefinder.LeagueGameFinder(
  File "/opt/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/nba_api/stats/endpoints/leaguegamefinder.py", line [20](https://github.com/jacasta2/nba_analysis/actions/runs/8010834757/job/21882723941#step:5:21)4, in __init__
    self.get_request()
  File "/opt/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/nba_api/stats/endpoints/leaguegamefinder.py", line 207, in get_request
    self.nba_response = NBAStatsHTTP().send_api_request(
  File "/opt/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/nba_api/library/http.py", line 1[30](https://github.com/jacasta2/nba_analysis/actions/runs/8010834757/job/21882723941#step:5:31), in send_api_request
    response = requests.get(url=base_url, params=parameters, headers=request_headers, proxies=proxies, timeout=timeout)
  File "/opt/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
  File "/opt/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/opt/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/opt/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/requests/adapters.py", line 5[32](https://github.com/jacasta2/nba_analysis/actions/runs/8010834757/job/21882723941#step:5:33), in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='stats.nba.com', port=4[43](https://github.com/jacasta2/nba_analysis/actions/runs/8010834757/job/21882723941#step:5:44)): Read timed out. (read timeout=30)

What a buzzkiller!

swar / nba_api

NBA API: timeout for /stats endpoints #320