search

mpope9 / nba-sql

:basketball: An application to build an NBA database backed by MySQL, Postgres, or SQLite
Apache License 2.0
171 stars 20 forks source link

Currently cannot build anything #91

Closed jakesjordan00 closed 1 year ago

jakesjordan00 commented 1 year ago

Hello Mpope, my sincere appreciation and thanks for developing this application. I've been using it daily for a few months now. I am currently running Windows 10. Honestly, the first two issues are the ones giving me the most trouble, I don't care too much about SCD, but listing all issues so that you are aware. Thank you again!

When building a DB for the 2022-23 season and skipping the play_by_play, shot_chart_detail and event_message_type tables, I receive this error image

When building a DB for the 2022-23 season and skipping all tables except for play_by_play and event_message_type, I receive this error. image

When building the shot_chart_detail table, I get a successful result but no rows are populated.

Please let me know if I can provide any more details that would help you out, thank you so much!

mpope9 commented 1 year ago

Hello! Thank you for the kind words and the bug report. I haven't ran the app in a few days, I'll fire it up and see if I get the same issue.

mpope9 commented 1 year ago

Are you using the default mode or current season mode? Are you skipping all the 'basetables'? Like player, game, team...

jakesjordan00 commented 1 year ago

Hi, absolutely, this is some awesome work you've done here!

In the first screenshot, my configuration was as follows: SQLite, default mode, 2022-23 season. I skipped the play_by_play, shot_chart_detail, and Event_message_type tables. That config has consistently run without issues since November, I believe.

In the second screenshot, it was SQLite, default mode, 2022-23 season. Here, I built only the play_by_play and Event_message_type tables, while skipping every other table, including the base tables. This strategy has been consistently working for a week or so until yesterday.

Before when trying to include play_by_play, I had been building the entire database, just skipping shot_chart_detail, but I would receive similar errors to those pictured. That lead to me to trying to build it with only play by play.

I also updated my python files with the code update you posted the other day and I am still receiving the same error messages.

Thank you for your quick response! I hope this was a concise answer.

mpope9 commented 1 year ago

Error handing isn't a strong suit of this code so I'm trying to debug it manually.

So I can reproduce the issue with play_by_play. But it isn't consistent for me. I am starting to wonder if the nba is getting more tight with their API. There is an issue on the nba_api https://github.com/swar/nba_api/issues/320 which we don't use but rely on the same endpoints.

In the meantime try bumping up the time between requests variable and see if that helps ya.

mpope9 commented 1 year ago

I understand the issue now. The nba-api is returning an incomplete response, it is truncated and it is not a valid JSON object, and we hit a decode error. Currently we fetch play_by_play data using the game_id. In the play_by_play documentation we can see that there is no other filter we can iterate over.

This is maybe a bug introduced by the NBA for the play_by_play endpoint. Sadly one of the more interesting endpoints. I'm not sure there is much we can do besides introduce some flag that says 'ignore errors' which will build an incomplete database. Missing games, etc. Unless there is some form of pagination I'm missing.

Under the whims of a public API.

jakesjordan00 commented 1 year ago

Hey, thank you for your time and thoughtful response. I tried it again on my work computer and was able to successfully build it, bizarre because not sure what in my current environment could be causing this.

Sorry for the waste of your time, sir, but please do keep up the great work. Thank you!

mpope9 commented 1 year ago

Hi, no worries. Glad it works again. Probably a bug on the NBA's side. Lemme know if anything comes up again.