BurntSushi / nflgame

An API to retrieve and read NFL Game Center JSON data. It can work with real-time data, which can be used for fantasy football.
http://pdoc.burntsushi.net/nflgame
The Unlicense
1.28k stars 413 forks source link

Updating Players #316

Open ToddAtWSU opened 7 years ago

ToddAtWSU commented 7 years ago

So I remembered I need to update the player information before getting stats for rookies and the such. So I successfully ran pip to install the upgrade to nflgame. I couldn't get nflgame-update-players to run through Powershell or the command window in Windows 10, but I did load the script in Idle with Python 2.7. I hit Run to execute the script and got the following output:

============ RESTART: C:\Python27\Scripts\nflgame-update-players ============ Loading games for REG 2017 week 1 Downloading team rosters...

1/32 complete. (3.12%) 2/32 complete. (6.25%) 3/32 complete. (9.38%) 4/32 complete. (12.50%) 5/32 complete. (15.62%) 6/32 complete. (18.75%) 7/32 complete. (21.88%) 8/32 complete. (25.00%) 9/32 complete. (28.12%) 10/32 complete. (31.25%) 11/32 complete. (34.38%) 12/32 complete. (37.50%) 13/32 complete. (40.62%) 14/32 complete. (43.75%) 15/32 complete. (46.88%) 16/32 complete. (50.00%) 17/32 complete. (53.12%) 18/32 complete. (56.25%) 19/32 complete. (59.38%) 20/32 complete. (62.50%) 21/32 complete. (65.62%) 22/32 complete. (68.75%) 23/32 complete. (71.88%) 24/32 complete. (75.00%) 25/32 complete. (78.12%) 26/32 complete. (81.25%) 27/32 complete. (84.38%) 28/32 complete. (87.50%) 29/32 complete. (90.62%) 30/32 complete. (93.75%) 31/32 complete. (96.88%) 32/32 complete. (100.00%) Done! Fetching GSIS identifiers for players not in nflgame...

1/428 complete. (0.23%) 2/428 complete. (0.47%) 3/428 complete. (0.70%) 4/428 complete. (0.93%) 5/428 complete. (1.17%) 6/428 complete. (1.40%) 7/428 complete. (1.64%) 8/428 complete. (1.87%) 9/428 complete. (2.10%) 10/428 complete. (2.34%) 11/428 complete. (2.57%) 12/428 complete. (2.80%) 13/428 complete. (3.04%) 14/428 complete. (3.27%) 15/428 complete. (3.50%) 16/428 complete. (3.74%) 17/428 complete. (3.97%) 18/428 complete. (4.21%) 19/428 complete. (4.44%) 20/428 complete. (4.67%) 21/428 complete. (4.91%) 22/428 complete. (5.14%) 23/428 complete. (5.37%) 24/428 complete. (5.61%) 25/428 complete. (5.84%) 26/428 complete. (6.07%) 27/428 complete. (6.31%) 28/428 complete. (6.54%) 29/428 complete. (6.78%) 30/428 complete. (7.01%) 31/428 complete. (7.24%) 32/428 complete. (7.48%) 33/428 complete. (7.71%) 34/428 complete. (7.94%) 35/428 complete. (8.18%) 36/428 complete. (8.41%) 37/428 complete. (8.64%) 38/428 complete. (8.88%) 39/428 complete. (9.11%) 40/428 complete. (9.35%) 41/428 complete. (9.58%) 42/428 complete. (9.81%) 43/428 complete. (10.05%) 44/428 complete. (10.28%) 45/428 complete. (10.51%) 46/428 complete. (10.75%) 47/428 complete. (10.98%) 48/428 complete. (11.21%) 49/428 complete. (11.45%) 50/428 complete. (11.68%) 51/428 complete. (11.92%) 52/428 complete. (12.15%) 53/428 complete. (12.38%) 54/428 complete. (12.62%) 55/428 complete. (12.85%) 56/428 complete. (13.08%) 57/428 complete. (13.32%) 58/428 complete. (13.55%) 59/428 complete. (13.79%) 60/428 complete. (14.02%) 61/428 complete. (14.25%) 62/428 complete. (14.49%) 63/428 complete. (14.72%) 64/428 complete. (14.95%) 65/428 complete. (15.19%) 66/428 complete. (15.42%) 67/428 complete. (15.65%) 68/428 complete. (15.89%) 69/428 complete. (16.12%) 70/428 complete. (16.36%) 71/428 complete. (16.59%) 72/428 complete. (16.82%) 73/428 complete. (17.06%) 74/428 complete. (17.29%) 75/428 complete. (17.52%) 76/428 complete. (17.76%) 77/428 complete. (17.99%) 78/428 complete. (18.22%) 79/428 complete. (18.46%) 80/428 complete. (18.69%) 81/428 complete. (18.93%) 82/428 complete. (19.16%) 83/428 complete. (19.39%) 84/428 complete. (19.63%) 85/428 complete. (19.86%) 86/428 complete. (20.09%) 87/428 complete. (20.33%) 88/428 complete. (20.56%) 89/428 complete. (20.79%) 90/428 complete. (21.03%) 91/428 complete. (21.26%) 92/428 complete. (21.50%) 93/428 complete. (21.73%) 94/428 complete. (21.96%) 95/428 complete. (22.20%) 96/428 complete. (22.43%) 97/428 complete. (22.66%) 98/428 complete. (22.90%) 99/428 complete. (23.13%) 100/428 complete. (23.36%) 101/428 complete. (23.60%) 102/428 complete. (23.83%) 103/428 complete. (24.07%) 104/428 complete. (24.30%) 105/428 complete. (24.53%) 106/428 complete. (24.77%) 107/428 complete. (25.00%) 108/428 complete. (25.23%) 109/428 complete. (25.47%) 110/428 complete. (25.70%) 111/428 complete. (25.93%) 112/428 complete. (26.17%) 113/428 complete. (26.40%) 114/428 complete. (26.64%) 115/428 complete. (26.87%) 116/428 complete. (27.10%) 117/428 complete. (27.34%) 118/428 complete. (27.57%) 119/428 complete. (27.80%) 120/428 complete. (28.04%) 121/428 complete. (28.27%) 122/428 complete. (28.50%) 123/428 complete. (28.74%) 124/428 complete. (28.97%) 125/428 complete. (29.21%) 126/428 complete. (29.44%) 127/428 complete. (29.67%) 128/428 complete. (29.91%) 129/428 complete. (30.14%) 130/428 complete. (30.37%) 131/428 complete. (30.61%) 132/428 complete. (30.84%) 133/428 complete. (31.07%) 134/428 complete. (31.31%) 135/428 complete. (31.54%) 136/428 complete. (31.78%) 137/428 complete. (32.01%) 138/428 complete. (32.24%) 139/428 complete. (32.48%) 140/428 complete. (32.71%) 141/428 complete. (32.94%) 142/428 complete. (33.18%) 143/428 complete. (33.41%) 144/428 complete. (33.64%) 145/428 complete. (33.88%) 146/428 complete. (34.11%) 147/428 complete. (34.35%) 148/428 complete. (34.58%) 149/428 complete. (34.81%) 150/428 complete. (35.05%) 151/428 complete. (35.28%) 152/428 complete. (35.51%) 153/428 complete. (35.75%) 154/428 complete. (35.98%) 155/428 complete. (36.21%) 156/428 complete. (36.45%) 157/428 complete. (36.68%) 158/428 complete. (36.92%) 159/428 complete. (37.15%) 160/428 complete. (37.38%) 161/428 complete. (37.62%) 162/428 complete. (37.85%) 163/428 complete. (38.08%) 164/428 complete. (38.32%) 165/428 complete. (38.55%) 166/428 complete. (38.79%) 167/428 complete. (39.02%) 168/428 complete. (39.25%) 169/428 complete. (39.49%) 170/428 complete. (39.72%) 171/428 complete. (39.95%) 172/428 complete. (40.19%) 173/428 complete. (40.42%) 174/428 complete. (40.65%) 175/428 complete. (40.89%) 176/428 complete. (41.12%) 177/428 complete. (41.36%) 178/428 complete. (41.59%) 179/428 complete. (41.82%) 180/428 complete. (42.06%) 181/428 complete. (42.29%) 182/428 complete. (42.52%) 183/428 complete. (42.76%) 184/428 complete. (42.99%) 185/428 complete. (43.22%) 186/428 complete. (43.46%) 187/428 complete. (43.69%) 188/428 complete. (43.93%) 189/428 complete. (44.16%) 190/428 complete. (44.39%) 191/428 complete. (44.63%) 192/428 complete. (44.86%) 193/428 complete. (45.09%) 194/428 complete. (45.33%) 195/428 complete. (45.56%) 196/428 complete. (45.79%) 197/428 complete. (46.03%) 198/428 complete. (46.26%) 199/428 complete. (46.50%) 200/428 complete. (46.73%) 201/428 complete. (46.96%) 202/428 complete. (47.20%) 203/428 complete. (47.43%) 204/428 complete. (47.66%) 205/428 complete. (47.90%) 206/428 complete. (48.13%) 207/428 complete. (48.36%) 208/428 complete. (48.60%) 209/428 complete. (48.83%) 210/428 complete. (49.07%) 211/428 complete. (49.30%) 212/428 complete. (49.53%) 213/428 complete. (49.77%) 214/428 complete. (50.00%) 215/428 complete. (50.23%) 216/428 complete. (50.47%) 217/428 complete. (50.70%) 218/428 complete. (50.93%) 219/428 complete. (51.17%) 220/428 complete. (51.40%) 221/428 complete. (51.64%) 222/428 complete. (51.87%) 223/428 complete. (52.10%) 224/428 complete. (52.34%) 225/428 complete. (52.57%) 226/428 complete. (52.80%) Traceback (most recent call last): File "C:\Python27\Scripts\nflgame-update-players", line 4, in nflgame.update_players.run() File "C:\Python27\lib\site-packages\nflgame\update_players.py", line 436, in run for i, (purl, gid) in enumerate(pool.imap(fetch, purls), 1): File "C:\Python27\lib\multiprocessing\pool.py", line 668, in next raise value timeout: timed out

What do I need to do to get this to run completely? Or am I doing something incorrectly? Thanks again for your help!

ToddAtWSU commented 7 years ago

Okay, so I tried running it a couple more times and got this far. Any idea how to fix this error?:

418/428 complete. (97.66%) 419/428 complete. (97.90%) 420/428 complete. (98.13%) 421/428 complete. (98.36%) 422/428 complete. (98.60%) 423/428 complete. (98.83%) 424/428 complete. (99.07%) 425/428 complete. (99.30%) 426/428 complete. (99.53%) 427/428 complete. (99.77%) 428/428 complete. (100.00%) Done!

There were some errors during the download. Usually this is a result of an HTTP request timing out, which means the resulting "players.json" file is probably missing some data. An appropriate solution is to re-run the script until there are no more errors (or when the errors are problems on NFL.com side.)

Could not get player info from roster row:

72 Leno, Charles, Jr. T ACT 6'3" 305 10/9/1991 4 Boise State

Exception:

Traceback (most recent call last): File "C:\Python27\lib\site-packages\nflgame\update_players.py", line 419, in run roster.append(meta_from_soup_row(team, row)) File "C:\Python27\lib\site-packages\nflgame\update_players.py", line 179, in meta_from_soup_row last_name, first_name = map(lambda s: s.strip(), name.split(',')) ValueError: too many values to unpack

rjhaz commented 7 years ago

Exact same error I get, no matter how many times I run it.

chedder123 commented 7 years ago

This is a bandaid fix... but will let you get past the problem.

Edit update_players.py

After line 175 add these two lines: if name.count(',') == 2: name = name.strip(', Jr.')

Make sure to match the spacing...

Before: def meta_from_soup_row(team, soup_row): tds, data = [], [] for td in soup_row.find_all('td'): tds.append(td) data.append(td.get_text().strip()) profile_url = 'http://www.nfl.com%s' % tds[1].a['href']

name = tds[1].a.get_text().strip()
if ',' not in name:
    last_name, first_name = name, ''
else:
    last_name, first_name = map(lambda s: s.strip(), name.split(','))

return {
    'team': team,
    'profile_id': profile_id_from_url(profile_url),
    'profile_url': profile_url,
    'number': try_int(data[0]),

After:

def meta_from_soup_row(team, soup_row): tds, data = [], [] for td in soup_row.find_all('td'): tds.append(td) data.append(td.get_text().strip()) profile_url = 'http://www.nfl.com%s' % tds[1].a['href']

name = tds[1].a.get_text().strip()
if name.count(',') == 2:
    name = name.strip(', Jr.')
if ',' not in name:
    last_name, first_name = name, ''
else:
    last_name, first_name = map(lambda s: s.strip(), name.split(','))

return {
    'team': team,
    'profile_id': profile_id_from_url(profile_url),
    'profile_url': profile_url,
    'number': try_int(data[0]),
jaydeluca commented 7 years ago

Not sure if this will help you, but I found that installing nflgame via github solved the problem for me:

pip uninstall nflgame
pip install git+https://github.com/BurntSushi/nflgame.git

And then running the player update script finally worked without error. I've done this successfully on mac and linux but not PC, so not sure if helps.

HvyOnEz commented 7 years ago

chedder123 I did the bandaid fix to get around the Jr. issue and it seems to have worked. However, I received errors opening the file with gedit (see below) - did this cause a major issue? Should I uninstall/reinstall?

[root@blackhouse nflgame]# gedit update_player.py (gedit:6498): WARNING : Error when getting information for file '/usr/lib/python2.7/site-packages/nflgame/update_player.py': No such file or directory

(gedit:6498): WARNING : Set document metadata failed: Setting attribute metadata::gedit-position not supported [root@blackhouse nflgame]# gedit update_players.py

(gedit:6514): Gtk-WARNING **: Calling Inhibit failed: GDBus.Error:org.freedesktop.DBus.Error.ServiceUnknown: The name org.gnome.SessionManager was not provided by any .service files

(gedit:6514): Gtk-WARNING **: Calling Inhibit failed: GDBus.Error:org.freedesktop.DBus.Error.ServiceUnknown: The name org.gnome.SessionManager was not provided by any .service files

(gedit:6514): Gtk-WARNING **: Calling Inhibit failed: GDBus.Error:org.freedesktop.DBus.Error.ServiceUnknown: The name org.gnome.SessionManager was not provided by any .service files

(gedit:6514): WARNING : Set document metadata failed: Setting attribute metadata::gedit-spell-language not supported

(gedit:6514): WARNING : Set document metadata failed: Setting attribute metadata::gedit-encoding not supported

(gedit:6514): Gtk-WARNING **: Calling Inhibit failed: GDBus.Error:org.freedesktop.DBus.Error.ServiceUnknown: The name org.gnome.SessionManager was not provided by any .service files

(gedit:6514): WARNING : Set document metadata failed: Setting attribute metadata::gedit-spell-language not supported

(gedit:6514): WARNING : Set document metadata failed: Setting attribute metadata::gedit-encoding not supported ^C [root@blackhouse nflgame]# python update_players.py Loading games for POST 2016 week 5 Downloading team rosters... 32/32 complete. (100.00%) Done!

jbird23 commented 7 years ago

Ran into this issue with player Charles Leno Jr.

I read this post after "fixing" the issue myself.

Instead of using if name.count(',') == 2: name = name.strip(', Jr.')

I used if ", Jr." in name: name = name.replace(", Jr.", "")

same result either way.... it's the extra comma in there that is the problem.

Why doesn't any other NFL player have a Jr. in their name on nfl.com like Odell?