search

mpope9 / nba-sql

:basketball: An application to build an NBA database backed by MySQL, Postgres, or SQLite
Apache License 2.0
175 stars 20 forks source link

Larger calls fail even with time-between-requests #103

Open magic8bull opened 10 months ago

magic8bull commented 10 months ago

I get the error below anytime I try to pull a larger dataset. I tried to use the time-between-requests to address, but it still fails at random points. I tried to pull the data in chucks but then run into issues with duplicate keys.

Traceback (most recent call last):---------------------| 13.0% File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/requests/models.py", line 971, in json return complexjson.loads(self.text, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/json/init.py", line 346, in loads return _default_decoder.decode(s) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/Users/steve/Projects/nba-sql/stats/nba_sql.py", line 398, in main(args, False) File "/Users/steve/Projects/nba-sql/stats/nba_sql.py", line 333, in main default_mode(settings, create_schema, request_gap, seasons, skip_tables, quiet or from_gui) File "/Users/steve/Projects/nba-sql/stats/nba_sql.py", line 164, in default_mode new_rows = play_by_play_requester.fetch_game(game_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/steve/Projects/nba-sql/stats/play_by_play.py", line 54, in fetch_game response = requests.get(url=self.url, headers=headers, params=params_str).json() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/requests/models.py", line 975, in json raise RequestsJSONDecodeError(e.msg, e.doc, e.pos) requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

mpope9 commented 8 months ago

Do you still get this error? Sometimes the NBA's API fails and only returns partial data, I'm not sure if this is a form of rate limiting or not.