Newbie foolishness

[rjhaz]

I am trying to get the stats for week 1 of 2012... getting this error. Never programed python until this test program, am pretty good PHP programmer though.. I know this is something I am doing wrong, I just cannot figure it out.

Ray

all_stats Traceback (most recent call last): File "./testnfl.py", line 23, in nflgame.combine(nflgame.games(2012, 1)).csv('Week1-2012.csv') File "/usr/lib/python2.7/site-packages/nflgame/seq.py", line 301, in csv for category, stats in p.all_stats().iteritems(): File "/usr/lib/python2.7/site-packages/nflgame/player.py", line 219, in getattr raise AttributeError AttributeError

[BurntSushi]

Thanks for this report. Turns out that I needed to update the csv method to reflect some changes to the API that I made recently. I've made a new release that includes this fix.

I really need to add some unit tests to this project.

[rjhaz]

I am learning as I go… I as it turns out, I installed the new package (I did not see an uninstall for the old package) and when I ran my program that dumps stats to a file, it gets errors with 1.1.2 that it did not with 1.1.1.

[root@AZtest ~]# ./testnfl.py

Traceback (most recent call last):

File "./testnfl.py", line 2, in

import nflgame

File "/usr/lib/python2.7/site-packages/nflgame/init.py", line 185, in

import nflgame.live

File "/usr/lib/python2.7/site-packages/nflgame/live.py", line 84, in

_EASTERN_TZ = pytz.timezone('US/Eastern')

NameError: name 'pytz' is not defined

AND here is the testnfl.py script (I know it is ugly, I am a newbe):

!/usr/bin/env python2.7

import nflgame

games = nflgame.games(2012, 1)

players = nflgame.combine(games)

players = nflgame.combine_max_stats(games)

rushers = players.rushing()

fo = open("rush.txt", "wb")

for p in rushers.filter(rushing_att=lambda x: x > 0).sort("rushing_yds"):

# print "Rusher: ", p, "Attempts: ", p.rushing_att, "Yards: ", p.rushing_yds, "Rush TDs: " ,p.rushing_tds

# print "Rusher: ", p, p.formatted_stats()

junk = "Rusher: ", p.name, ",", p.formatted_stats()

for xx in junk: fo.write(xx)

fo.write("\n")

fo.close()

passers = players.passing()

fo = open("pass.txt", "wb")

for p in passers.filter(passing_att=lambda x: x > 0).sort("passing_yds"):

# print "Passer: ", p, "Attempts: ", p.passing_att, "Yards: ", p.passing_yds, "Pass TDs: ", p.passing_tds

#print "Passer: ", p, p.formatted_stats()

pjunk = "Passer: ", p.name, ",", p.formatted_stats()

for xx in pjunk: fo.write(xx)

fo.write("\n")

fo.close()

receivers = players.receiving()

fo = open("receivers.txt", "wb")

for p in receivers.sort("receiving_yds"):

# print "Receivers: ",  p, "Yards: ", p.receiving_yds, "Receiving TDs: ", p.receiving_tds

#print "Receiver: ", p, p.formatted_stats()

rjunk = "Receiver: ", p.name, ",", p.formatted_stats()

for xx in rjunk: fo.write(xx)

fo.write("\n")

fo.close()

From: Andrew Gallant [mailto:notifications@github.com] Sent: Saturday, September 15, 2012 11:26 AM To: BurntSushi/nflgame Cc: rjhaz Subject: Re: [nflgame] Newbie foolishness (#9)

Thanks for this report. Turns out that I needed to update the csv method to reflect some changes to the API that I made recently.

I really need to add some unit tests to this project.

— Reply to this email directly or view it on GitHub https://github.com/BurntSushi/nflgame/issues/9#issuecomment-8587373 .

https://github.com/notifications/beacon/J6T91GIPIyhU-8ti4GCGP7d8-tM1nkdFqGeejP0wbnzRHLk6PmsHBEWNssotYQ0W.gif

[rjhaz]

Installed pytz and that solved the problem.. sorry.

[BurntSushi]

Yes, installing pytz would fix this problem, but you've actually uncovered another bug that I introduced in the last release. pytz is an optional dependency and should not be required for anything you're doing. (It's only used in the nflgame.live module for dealing with game times.)

The bug was that the main nflgame module was requiring pytz when it shouldn't be.

If you aren't going to use nflgame.live, you should feel free to uninstall pytz if you update to nflgame 1.1.3.

(I did not see an uninstall for the old package)

I'm not sure which operating system you're using, but if it's Windows, I'd imagine you'd be able to uninstall the old version through the standard Add/Remove interface. But it's been many moons since I've used Windows.

If you're on Mac or Linux and you installed it using pip, you should be able to uninstall it with pip too: pip uninstall nflgame. Failing that, you'll have to find your Python site-packages directory (usually something like /usr/lib/python2.7/site-packages) and remove the nflgame directory.

[rjhaz]

No problem… installing the pytz package was a simple matter once I realized it was a package… I am a little slow sometimes… J

I am using Linux (Fedora 16 for Dev, Fedora 10 on my prod site)… funny, prod is a little FFL site for me and 11 friends. I have run it for many, many years. The last 15 years I have been using data from Quickstats, LLC- but they were bought out this year and the new owner unexpectedly shutdown after one week. So I am scrambling for data (weekly static for now but I really love the .live part and will get to it as time permits… )

As for the uninstall, I was worried that something from 1.1.1 package crossed the 1.1.2 package and caused the pytz requirement and thought maybe I should have run “pythin2 setup.py uninstall “ for 1.1.1 (but I found there was no uninstall option) before I ran the install of 1.1.2.

I have all my stats in a MySQL database updated after each game (with the old QuickStats data). I have three issues I am working on overcoming with a combination of code to combine the stats, one is to get all the defense data for a single team in one bucket (for each team) so I can use it- I think that s just getting the right call to you classes, another is actually pulling the player info (I noticed it is not yet in your objects, so I am going to use the players.json file) so I can map the old QS player IDs to the NFL player IDs (I need team, position, along with name and NFL ID to do that) and last is to get the TD distance information (just a correct class call for the data again) for each score.

But the good news is you have this great package and I am having fun learning Python and merging the data into a format that I can put into my existing database and keep going this year. When I get around to the .live stuff, I am not sure I need more than a static updated once every 30 minutes or so since we are a small and simple league, so I will be a little more careful with how I implement that- based on what I learn now.

Thanks again for all your hard work and making nflgame available.

Ray

[BurntSushi]

prod is a little FFL site for me and 11 friends. I have run it for many, many years. The last 15 years I have been using data from Quickstats, LLC- but they were bought out this year and the new owner unexpectedly shutdown after one week. So I am scrambling for data (weekly static for now but I really love the .live part and will get to it as time permits… )

Ah wow, nice! I didn't know about Quickstats. I found some of its history. It's amazing how some of this data is so hard to find using conventional search methods. I can't tell you how much I've searched for machine readable data before writing this library, and QuickStats never came up.

Out of curiosity, why don't you use a hosted web site like Yahoo or ESPN?

one is to get all the defense data for a single team in one bucket (for each team) so I can use it- I think that s just getting the right call to you classes

I have done some work on that already. It isn't technically released yet, but I've put it up on github just now for you. The DefensePlayer class might be relevant to you. I'd recommend that you rip that code out rather than trying to use the whole package at this point. (It's not polished, rough and liable to change.) Most of it is pretty easy once you get the hang of it, but the Points Allowed bit is a little hellish. (But that's only if you're scoring it like Yahoo.) If you're interested in what else is going on there, see the README.

another is actually pulling the player info (I noticed it is not yet in your objects, so I am going to use the players.json file)

You're certainly free to use the players.json file (or perhaps the nflgame.players variable), but the player meta data is actually hooked into the PlayerStats objects already:

>>> import nflgame
>>> brady = nflgame.one(2012, 1, 'NE', 'NE').max_player_stats().name('T.Brady')
>>> print brady.player.name, brady.player.number, brady.player.college, brady.player.birthdate, brady.player.profile_url
Tom Brady 12 Michigan 8/3/1977 http://www.nfl.com/player/tombrady/2504211/profile

If for some reason you need to scrape for more data, see download-player-data for ideas. It uses BeautifulSoup. But note that the player objects contain the NFL profile URL already, which contains an identifier. (Whether it's the one you need or not, I'm not sure. NFL.com uses several different sets of player identifiers.)

and last is to get the TD distance information (just a correct class call for the data again) for each score.

Yup. I believe there is always a *_yds field associated with every touchdown. i.e., for rushing_tds there would be a rushing_yds.

But the good news is you have this great package and I am having fun learning Python and merging the data into a format that I can put into my existing database and keep going this year. When I get around to the .live stuff, I am not sure I need more than a static updated once every 30 minutes or so since we are a small and simple league, so I will be a little more careful with how I implement that- based on what I learn now.

Good luck and please do come back if you get stuck!

[rjhaz]

Sorry, this just seemed like the easiest way to say something... In using your program download-player-data, I was consistently getting errors on 20 to 30 of the 1800+ player profiles. I added a 1 second delay in the get_html routine and if it faild, a 5 second delay and then a retry of the failure. This resulted in 5 failed requests, all of which succeeded on the retry... the only down side is it takes 58 minutes to run... the UP side nfl.com gets a break and I get ALL players on one run.

[BurntSushi]

Hmm. I'm not sure what's going on. Sounds like a problem with the connection (either on your end or on NFL.com's end). I haven't had any problems with it (yet). I've actually just updated the player data in the repository a day or two ago.

Also, please open new issues when you have a new question :-). I am liable to miss these. It's much easier to see an open issue that doesn't have a response!

[rjhaz]

  Also, please open new issues when you have a new question :-)

Well, when it is not really an issue, I am reluctant to open one… this case is probably what I expected, and you confirmed, a connection issue (which has nothing to do with nflgame, and everything to do with the internet connection of the person running the screen scraper. But I can… perhaps next time I will open a “Open Questions” issue that can stay open and be used as a sounding board to determine if there is an issue or a user error… I am bug on user errors… old guys are like that… J

[BurntSushi]

No worries. Issues should be open for just about anything—including questions about how to use nflgame, or even problems with NFL.com data. There should be a pretty low threshold to opening one.

Basically, it comes down to information organization. If each question has its own issue, then they are easier to find in the future when someone else has the same question. And like I said, they are less likely to be forgotten about by me :-)

But I can… perhaps next time I will open a “Open Questions” issue that can stay open and be used as a sounding board to determine if there is an issue or a user error… I am bug on user errors… old guys are like that… J

Sounds good! :P

[rjhaz]

Sorry to bother you… I was wondering if you have any idea how long it takes the NFL to update their rosters for each team. I see the individual team web sites have updated theirs but nfl.com has not.

Ray

[ochawkeye]

I'm really surprised they're not updated yet. Teams had until 6:00 last night to submit their final 53-man rosters. Can't imagine that there isn't anyone manning the nfl.com offices one week before the season kicks off, even for a holiday weekend.

Ourlads looks like it is doing a better job than the head honchos.

[BurntSushi]

@rjhaz, @ochawkeye: So far, I have not been impressed with the timeliness of roster updates on NFL.com. I am strongly hoping for it to change once the season starts. As far as I'm concerned, if they can update them at least weekly, I'll be pretty happy.

If they don't, I'm not sure there is much I can do about it anyway. I need to use those roster pages for identifier mapping. Otherwise, I need to do name matching which is completely and utterly unreliable.

[rjhaz]

Thank you. I noticed that at least New England was updated on NFL.com this morning. Glad to see you are still doing well.. I am still using your stuff, thanks again… yes, I updated to the latest stuff as well.

Ray

From: Andrew Gallant [mailto:notifications@github.com] Sent: Sunday, September 01, 2013 6:50 PM To: BurntSushi/nflgame Cc: rjhaz Subject: Re: [nflgame] Newbie foolishness (#9)

@rjhaz https://github.com/rjhaz , @ochawkeye https://github.com/ochawkeye : So far, I have not been impressed with the timeliness of roster updates on NFL.com. I am strongly hoping for it to change once the season starts. As far as I'm concerned, if they can update them at least weekly, I'll be pretty happy.

If they don't, I'm not sure there is much I can do about it anyway. I need to use those roster pages for identifier mapping. Otherwise, I need to do name matching which is completely and utterly unreliable.

— Reply to this email directly or view it on GitHub https://github.com/BurntSushi/nflgame/issues/9#issuecomment-23637685 . https://github.com/notifications/beacon/u1jC4ix2CwPdEBRq26MnuLMZnd3-kMfXbYgXb92Y4AdtVIeTvqvTbXBjMUyaDqI2.gif

[rjhaz]

I was getting a little frustrated with the program that gets all the player stats… it would fail 9 times out of 10 getting the 1800+ players. So, I hacked (sorry, my python skills are very limited) a fix.. FYI…

def get_html(url):

try:

    # perr('\nURL: %s ' % url)

    return urllib2.urlopen(url,  timeout=180).read()

    time.sleep(5)

    return urllib2.urlopen(url,  timeout=180).read()

    time.sleep(5)

    return urllib2.urlopen(url,  timeout=180).read()

except urllib2.HTTPError:

    return None

[BurntSushi]

@rjhaz Which script are you using? You should be using nflgame-update-players, but it looks like you might be using download-player-data, which is now deprecated. I probably should remove it from the repo.

[rjhaz]

Yep, I did not get the new program. I will look for it now.

From: Andrew Gallant [mailto:notifications@github.com] Sent: Sunday, September 15, 2013 1:39 PM To: BurntSushi/nflgame Cc: rjhaz Subject: Re: [nflgame] Newbie foolishness (#9)

@rjhaz https://github.com/rjhaz Which script are you using? You should be using nflgame-update-players, but it looks like you might be using download-player-data, which is now deprecated. I probably should remove it from the repo.

— Reply to this email directly or view it on GitHub https://github.com/BurntSushi/nflgame/issues/9#issuecomment-24479675 . https://github.com/notifications/beacon/u1jC4ix2CwPdEBRq26MnuLMZnd3-kMfXbYgXb92Y4AdtVIeTvqvTbXBjMUyaDqI2.gif

[aqsmith08]

Hi - Most important, thanks for creating nflgame. Pretty cool tool. I'm (slowly) figuring it out but pretty excited to learn more.

Here's my issue - My end goal is to pull NFL player first_name, last_name, birthdate, gsis_id, player_id, profile_id, height, weight and years_pro.

Per the comments in this issue chain, I should be able to use BeautifulSoup (?). However, when I try to follow the link, I receive a 404 error. Is there a way I can pull the player stats directly from nflgame? Thanks for your time.

You're certainly free to use the players.json file (or perhaps the nflgame.players variable), but the player meta data is actually hooked into the PlayerStats objects already:

import nflgame brady = nflgame.one(2012, 1, 'NE', 'NE').max_player_stats().name('T.Brady') print brady.player.name, brady.player.number, brady.player.college, brady.player.birthdate, brady.player.profile_url Tom Brady 12 Michigan 8/3/1977 http://www.nfl.com/player/tombrady/2504211/profile

If for some reason you need to scrape for more data, see download-player-data for ideas. It uses BeautifulSoup. But note that the player objects contain the NFL profile URL already, which contains an identifier. (Whether it's the one you need or not, I'm not sure. NFL.com uses several different sets of player identifiers.)

[ochawkeye]

@aqsmith08 What exactly are you having trouble with? From your post, maybe you are trying to input a player's name and get all of that information back? In order to do that, you might be better served using nfldb with its fuzzy matching capabilities.

On the other hand, if you're just trying to get all of the information in question for every player, that shouldn't be a problem using nflgame - no need for BeautifulSoup knowledge, the data is all right there.

The following would return name, birthdate, id, height, weight, and years pro for every player that registered a stat in week 6, 2013.

import nflgame

year, week = nflgame.live.current_year_and_week()
season_type = 'REG'

games = nflgame.games(year, week=week, kind=season_type)
players = nflgame.combine_game_stats(games)

for player in players:
    print (player.player.name, player.player.birthdate, player.playerid,
           player.player.height, player.player.weight,
           player.player.years_pro)

(u'Jay Cutler', u'4/29/1983', u'00-0024226', u'6\'3"', u'220', 8)
(u'Alshon Jeffery', u'2/14/1990', u'00-0029137', u'6\'3"', u'216', 2)
(u'Matt Forte', u'12/10/1985', u'00-0026184', u'6\'2"', u'218', 6)
(...)a few hundred more...766 to be exact