BurntSushi / nfldb

A library to manage and update NFL data in a relational database.
The Unlicense
1.08k stars 264 forks source link

nfldb-update not working, Multiple errors #216

Open theSanchize3 opened 7 years ago

theSanchize3 commented 7 years ago

I ran nfldb-update and got multiple errors. Here is the output:

STARTING NFLDB UPDATE AT 2016-12-09 15:42:51.333000
Connecting to nfldb... done.
Setting timezone to UTC... done.
Updating player JSON database... (last update was 2016-12-07 00:00:56.272000+00:00)
Loading games for REG 2016 week 14
Finding (profile id -> gsis id) mapping for players...
1/1 complete. (100.00%)
Done!
Downloading team rosters...
32/32 complete. (100.00%)
Done!
Fetching GSIS identifiers for players not in nflgame...
58/58 complete. (100.00%)
Done!

There were some errors during the download. Usually this is a
result of an HTTP request timing out, which means the
resulting "players.json" file is probably missing some data.
An appropriate solution is to re-run the script until there
are no more errors (or when the errors are problems on
NFL.com side.)
-------------------------------------------------------------------------------
Could not get profile URL for (00-0032820, T.Smith)
-------------------------------------------------------------------------------
Could not get player info from roster row:

<tr class="even">
<td>72</td>
<td style="text-align:left"> <a href="/player/charleslenojr./2550169/profile">Leno, Charles, Jr.</a></td>
<td>T</td>
<td>ACT</td>
<td> 6'3"</td>
<td>305</td>
<td>10/9/1991</td>
<td>3</td>
<td>Boise State</td></tr>

Exception:

Traceback (most recent call last):
  File "C:\Python27\lib\site-packages\nflgame\update_players.py", line 419, in run
    roster.append(meta_from_soup_row(team, row))
  File "C:\Python27\lib\site-packages\nflgame\update_players.py", line 179, in meta_from_soup_row
    last_name, first_name = map(lambda s: s.strip(), name.split(','))
ValueError: too many values to unpack

done.
Locking player table...
Updating 6827 players... done.
Locking write access to tables... done.
Updating season phase, year and week... done.
Adding schedule data for 321 games... done.
Bulk inserting data for 258 games...
        Sending batch of data to database.
        Sending batch of data to database.
        Sending batch of data to database.
        Sending batch of data to database.
        Sending batch of data to database.
        Sending batch of data to database.
        Sending batch of data to database.
        Sending batch of data to database.
        Sending batch of data to database.
        Sending batch of data to database.
        Sending batch of data to database.
        Sending batch of data to database.
        Sending batch of data to database.
        Sending batch of data to database.
        Sending batch of data to database.
        Sending batch of data to database.
        Sending batch of data to database.
        Sending batch of data to database.
        Sending batch of data to database.
Traceback (most recent call last):
  File "nfldb-update", line 39, in <module>
    nfldb.update.run(**vars(args))
  File "C:\Python27\lib\site-packages\nfldb\update.py", line 535, in run
    doit()
  File "C:\Python27\lib\site-packages\nfldb\update.py", line 525, in doit
    update_games(db, batch_size=batch_size)
  File "C:\Python27\lib\site-packages\nfldb\update.py", line 397, in update_games
    bulk_insert_game_data(cursor, scheduled, batch_size=batch_size)
  File "C:\Python27\lib\site-packages\nfldb\update.py", line 195, in bulk_insert_game_data
    do()
  File "C:\Python27\lib\site-packages\nfldb\update.py", line 188, in do
    nfldb.db._big_insert(cursor, table, bulk[table])
  File "C:\Python27\lib\site-packages\nfldb\db.py", line 356, in _big_insert
    % (table, insert_fields, values))
  File "C:\Python27\lib\site-packages\psycopg2\extras.py", line 223, in execute
    return super(RealDictCursor, self).execute(query, vars)
psycopg2.IntegrityError: insert or update on table "play" violates foreign key constraint "play_pos_team_fkey"
DETAIL:  Key (pos_team)=(JAX) is not present in table "team".

I upgraded nfldb and nflgame but that did not work. I saw online someone suggested (to a different post) to user python nfldb-update --play-interval 0, but that did not work either.

Thanks in advance, Sanchez

andr3w321 commented 7 years ago

These are both known and open issues. The most critical one is 194 which after you apply the hack should update just fine skipping over the player with multiple names. https://github.com/BurntSushi/nfldb/issues/194 https://github.com/BurntSushi/nfldb/issues/214