search

mpope9 / nba-sql

:basketball: An application to build an NBA database backed by MySQL, Postgres, or SQLite
Apache License 2.0
178 stars 23 forks source link

Distribute season loading (or rewrite in Elixir) #11

Open mpope9 opened 3 years ago

mpope9 commented 3 years ago

This is to get past the significant slowness due to rate limiting prevention. Distributing by season_id would probably be best. Can add a flask sever and use a single REST endpoint. Or, we could re-write in Elixir, but that would make the barrier to entry for contributions higher.

mpope9 commented 3 years ago

Argument for this: the NBA's rate limiting is getting more aggressive. We've had to bump the time between requests to .5 to .7. See this issue: https://github.com/mpope9/nba-sql/issues/19

ddresselhaus commented 3 years ago

if you need any elixir contributions, hmu 👍

mpope9 commented 3 years ago

@ddresselhaus Forsure! I'm on the fence about it at the moment, but will let you know if some type of 'control' layer is in the works.