add college data per player

aatherton / team_project

team project for data science class

0 stars 0 forks source link

add college data per player #3

Closed aatherton closed 6 years ago

aatherton commented 6 years ago

no idea how we're going to do this. possibilities:

automated make API key, then make a ton of calls with it? -- use as last resort. highly likely we'll get detected by IP address and shut down --- we should NOT have to bypass someone's security for a school project :C -- maybe use contacts at sportsradar? if they're willing to just dump a json of the data we'd be querying in our laps, that'd be k
url-hack/webscrape from http://www.espn.com/nfl/college/_/letter/c -- use as next-to-last resort. dunno all the details of webscraping. ask TAs for help?
use https://api.nfl.com/docs/global/endpoints/index.html -- no idea how this API works. documentation is ass. if we can figure it out, it'll be perfect
use http://api.fantasy.nfl.com/v1/docs/service?serviceName=playersStats -- this one requires a player ID unique to their api. so, if we can find a way to get that, it'll be two calls per player.

Flaw1331 commented 6 years ago

I've got the college's covered. I've got my api keys slightly modified by my friend to allow me to do some extra polling. Additionally, I have extra keys set up so that we won't have to worry about our limits. It sounds like Sportradar doesn't monitor their api keys all that closely. As long as you don't make the account and api key on the same IP, they won't care how many accounts you ultimately have. But if you make it from the same IP, it'll hard-lock your second account/key automatically. Basically, they operate under the thinking, "we'll give you as much data as you want, for free, just don't be a dick and abuse the system and ruin it for everyone." They just don't want people making trials keys every 3 months to get around the time limit when building a real-time updating app for sports. But a one time dump of some info, they could care less.

aatherton commented 6 years ago

coolcool. trying to think of best way to format it to mesh with what I'm doing... I'm thinking two-column either playerID, college or player_name, college? I might just init college as NA while I'm doing my thing... then we just replace with the relevant values. either way, assigning to you, since you're on it

Flaw1331 commented 6 years ago

I was thinking of just appending all the college's to the all_players.csv after I pull it all. Then we have all the data in one big csv/DF and can use as needed.

aatherton commented 6 years ago

that'll work. ...I'll have to write a script to get back to where we are now with it, though. that's fine- it just means starting from a different spot. I'll write my thing to start from all_players, cut all non-offense, drop duplicates, and change their values to their sum

aatherton commented 6 years ago

actually, it's probably better if you append to unique_players.csv- reduces the api calls.

Flaw1331 commented 6 years ago

My bad, I mispoke - I meant the unique_players list. No need to do ~35k calls for all players. Forgot we made that list.