Closed mesee298 closed 9 years ago
In one case you are counting the number of fumbles_lost
that appear in plays that involve Matt Jones. In the other, you are counting the number of plays that Matt Jones appeared in where a fumble was lost. Those two should be the same number and I think only @BurntSushi can best answer this one, but this might be a bug due to the "strange" nature of that particular play.
(WAS, NYG 7, Q4, 4 and 1) (9:42) M.Jones left end to NYG 2 for 5 yards (T.Wade). FUMBLES (T.Wade), ball out of bounds in End Zone, Touchback.
The play.data
includes this information for M.Jones:
u'00-0031939': [{u'playerName': u'M.Jones', u'clubcode': u'WAS', u'yards': 5, u'statId': 10, u'sequence': 2},
{u'playerName': u'M.Jones', u'clubcode': u'WAS', u'yards': 0, u'statId': 52, u'sequence': 4},
{u'playerName': u'M.Jones', u'clubcode': u'WAS', u'yards': 0, u'statId': 54, u'sequence': 5},
{u'playerName': u'M.Jones', u'clubcode': u'WAS', u'yards': 0, u'statId': 106, u'sequence': 6}]
statID: 10
must have to do with with rushing yards and/or attempts
statId: 52
signifies that a fumble was forced
statID: 54
signifies that the ball was fumbled out of bounds
statID: 106
signifies that the fumble was lost
By definition, a ball being fumbled out of bounds is not a turnover so there should only be a single fumble_lost
there.
Looking at the other players involved in the play:
u'00-0029314': [{u'playerName': u'T.Wade', u'clubcode': u'NYG', u'yards': 0, u'statId': 79, u'sequence': 3},
{u'playerName': u'T.Wade', u'clubcode': u'NYG', u'yards': 0, u'statId': 91, u'sequence': 7}]
Everything on the up and up there with old T.Wade
.
statID: 91
is a defensive forced fumble.
statID: 79
most likely has to do with being involved in a tackle.
Next:
u'0': [{u'playerName': u'', u'clubcode': u'WAS', u'yards': 0, u'statId': 9, u'sequence': 1}]
Now, it is quite common for "someone" like that to show in the JSON (that "player" appears 94 times in this game alone). It must be how team stats are awarded. Where I have a question is with statID: 9
. Looking at other JSON examples of lost fumbles, something like that does not appear in the play data. And that particular stat ID doesn't appear anywhere else in this game or any of the other games I glanced at. If it represents a fumble_lost
for the team, then it could be leading to a double counting of fumbles_lost
.
OMG - all that typing and it was in front of me the whole time:
9: {
'cat': 'team',
'fields': ['fourth_down_att', 'fourth_down_failed'],
'yds': '',
'desc': '4th down attempt failed',
'long': '4th down play did not result in a first down or touchdown.',
},
So it was a 4th down. So much for that theory!
I need to change my answer. Please put me down for "Huh...how about that. You got me!"
combine_max_stats
combines both game level statistics and play level statistics by taking the max
of each statistical category. combine_plays
just takes sums the values at the play level and ignores game level statistics. They are, by definition, different. The discrepancy is probably due to an inaccuracy in the game level data?
The reason why combine_max_stats
exists is because I thought it was a good heuristic to get the most accurate stats. But it doesn't always work.
But can you explain why he's getting credited with 2 fumbles on a single play?
import nflgame
games = nflgame.games(year=2015, week=3, home='WAS', away='WAS')
for play in nflgame.combine_plays(games):
if play.has_player('00-0031939') and play.fumbles_lost:
print play
for player in play.players:
print player.player, player.fumbles_lost, player.fumbles_tot
(WAS, NYG 7, Q4, 4 and 1) (9:42) M.Jones left end to NYG 2 for 5 yards (T.Wade). FUMBLES (T.Wade), ball out of bounds in End Zone, Touchback.
Trevin Wade (DB, NYG) 0 0
Matt Jones (RB, WAS) 1 2
Any ideas? I can force my program for the anomaly, but would prefer a fix.
The source JSON data includes an event for "fumble forced" and "fumble went out of bounds." The statistical mapping considers these mutually exclusively events when tallying the fumbles_tot
field. In this play, both events happened which explains why fumbles_tot
is set to 2
.
I'm not sure why the mapping considers these events exclusive. In any case, I've fixed that. To get the fix, upgrade nflgame: pip install --upgrade nflgame
.
If you're running nfldb
, then you can use this program to get all games with an erroneous play:
import nfldb
db = nfldb.connect()
q = nfldb.Query(db)
for pp in q.play_player(fumbles_tot__ge=2).as_play_players():
print pp.gsis_id
There are 366 of them!
This will delete all such games from your DB:
for gsis_id in $(python2 150.py | sort -u); do printf "psql nfldb -c \"DELETE FROM game WHERE gsis_id = '%s'\"\n" $gsis_id; done | sh
And then re-insert them with corrected stats (AFTER upgrading nflgame
):
nfldb-update --batch-size 150
If your computer is low on RAM (<=8GB), then you might want to use smaller batch sizes, but it will take longer.
Alternatively, wait for me to finish running nfldb-update
. I'll upload a correct DB and you can just re-import.
OK, I've updated the database dump, which should have corrected this particular issue: http://burntsushi.net/stuff/nfldb.sql.zip
And this is cute. If I run the above program that detects errant plays and prints them---after I fixed the DB---I get this output:
2015011100 {'fumbles_tot': 2, 'fumbles_forced': 1, 'fumbles_lost': 1, 'fumbles_rec': 1, 'fumbles_notforced': 1, 'defense_tkl': 1, 'fumbles_rec_yds': -9, 'rushing_att': 1}
(GB, OPP 27, Q2, 3 and 7) (11:21) (No Huddle, Shotgun) A.Rodgers FUMBLES (Aborted) at DAL 32, and recovers at DAL 32. A.Rodgers to DAL 34 for -2 yards (J.Mincey). FUMBLES (J.Mincey), RECOVERED by DAL-J.Mincey at DAL 36. J.Mincey to DAL 36 for no gain (A.Rodgers).
-------------------------------------------------------------------------------
2014092113 {'fumbles_tot': 2, 'fumbles_forced': 1, 'fumbles_lost': 1, 'fumbles_rec': 1, 'fumbles_notforced': 1, 'puntret_tot': 1, 'puntret_yds': -10}
(PIT, OWN 29, Q4, 4 and 1) (11:17) B.Wing punts 59 yards to CAR 12, Center-G.Warren. C.Brown MUFFS catch, and recovers at CAR 13. C.Brown to CAR 10 for -3 yards (S.Thomas). FUMBLES (S.Thomas), touched at CAR 3, RECOVERED by PIT-R.Golden at CAR -4. TOUCHDOWN.
-------------------------------------------------------------------------------
2014091412 {'fumbles_tot': 2, 'fumbles_forced': 2, 'fumbles_lost': 1, 'receiving_yac_yds': 19, 'fumbles_rec': 1, 'receiving_rec': 1, 'receiving_yds': 42, 'receiving_tar': 1}
(OAK, OWN 45, Q2, 2 and 4) (1:07) (No Huddle, Shotgun) D.Carr pass deep right to J.Jones to HOU 29 for 26 yards (K.Lewis). FUMBLES (K.Lewis), and recovers at HOU 28. J.Jones to HOU 13 for 15 yards (J.Joseph). FUMBLES (J.Joseph), RECOVERED by HOU-D.Swearinger at HOU 3. D.Swearinger to HOU 3 for no gain (R.Streater).
-------------------------------------------------------------------------------
2013112411 {'fumbles_tot': 2, 'fumbles_forced': 1, 'fumbles_rec': 1, 'fumbles_notforced': 1, 'puntret_tot': 1, 'puntret_yds': 8}
(DEN, OWN 42, Q2, 4 and 16) (13:00) B.Colquitt punts 49 yards to NE 9, Center-A.Brewer. J.Edelman MUFFS catch, and recovers at NE 9. J.Edelman to NE 17 for 8 yards (D.Trevathan). FUMBLES (D.Trevathan), recovered by NE-M.Buchanan at NE 21. M.Buchanan to NE 21 for no gain (S.Johnson).
-------------------------------------------------------------------------------
2013101309 {'fumbles_tot': 2, 'fumbles_forced': 1, 'fumbles_lost': 1, 'fumbles_rec': 1, 'fumbles_notforced': 1, 'fumbles_rec_yds': -19, 'rushing_att': 1}
(SEA, OPP 4, Q2, 3 and 4) (:02) (Field Goal formation) C.Maragos FUMBLES (Aborted) at TEN 12, and recovers at TEN 12. C.Maragos to TEN 16 for -4 yards (M.Griffin). FUMBLES (M.Griffin), RECOVERED by TEN-J.McCourty at TEN 23. J.McCourty for 77 yards, TOUCHDOWN. Play Challenged by Replay Assistant and Upheld.
-------------------------------------------------------------------------------
2012122310 {'passing_sk': 1, 'fumbles_tot': 2, 'fumbles_forced': 1, 'fumbles_lost': 1, 'fumbles_rec': 1, 'fumbles_notforced': 1}
(HOU, OWN 40, Q4, 3 and 9) (3:23) (Shotgun) T.Yates FUMBLES (Aborted) at HOU 40, and recovers at HOU 32. T.Yates sacked at HOU 36 for -4 yards (H.Smith). FUMBLES (H.Smith) [H.Smith], RECOVERED by MIN-E.Griffen at HOU 41. E.Griffen to HOU 36 for 5 yards (D.Newton).
-------------------------------------------------------------------------------
2012102810 {'fumbles_tot': 2, 'fumbles_forced': 1, 'fumbles_lost': 1, 'fumbles_rec': 1, 'fumbles_notforced': 1, 'puntret_tot': 1, 'puntret_yds': -6}
(NYG, OWN 20, Q1, 4 and 5) (1:08) (Punt formation) S.Weatherford punts 53 yards to DAL 27, Center-Z.DeOssie. D.Bryant MUFFS catch, and recovers at DAL 21. D.Bryant to DAL 20 for -1 yards (Z.DeOssie). FUMBLES (Z.DeOssie), RECOVERED by NYG-M.Coe at DAL 15.
-------------------------------------------------------------------------------
2012093007 {'fumbles_tot': 2, 'fumbles_forced': 1, 'fumbles_rec': 2, 'fumbles_notforced': 1, 'puntret_tot': 1, 'puntret_yds': 8}
(MIA, OWN 19, Q4, 4 and 20) (14:22) B.Fields punts 47 yards to ARI 34, Center-J.Denney. P.Peterson MUFFS catch, touched at ARI 34, and recovers at ARI 34. P.Peterson to ARI 42 for 8 yards (K.Misi). FUMBLES (K.Misi), and recovers at ARI 42. P.Peterson to ARI 42 for no gain (J.Denney).
-------------------------------------------------------------------------------
2011112007 {'fumbles_tot': 2, 'fumbles_oob': 1, 'fumbles_rec': 1, 'fumbles_notforced': 2, 'rushing_yds': 6, 'rushing_att': 1}
(GB, OWN 21, Q1, 4 and 1) (10:04) (Punt formation) T.Masthay right end to GB 12 for -9 yards. FUMBLES, and recovers at GB 18. T.Masthay to GB 27 for 9 yards. FUMBLES, ball out of bounds at GB 27.
-------------------------------------------------------------------------------
2011110605 {'fumbles_tot': 2, 'fumbles_forced': 1, 'fumbles_rec': 2, 'fumbles_notforced': 1, 'puntret_tot': 1, 'puntret_yds': 2}
(NO, OWN 22, Q4, 4 and 8) (9:51) T.Morstead punts 57 yards to TB 21, Center-J.Drescher. P.Parker MUFFS catch, and recovers at TB 21. P.Parker to TB 23 for 2 yards (J.Amaya). FUMBLES (J.Amaya), and recovers at TB 23. P.Parker to TB 23 for no gain (J.Amaya).
-------------------------------------------------------------------------------
2010112804 {'passing_sk': 1, 'fumbles_tot': 2, 'passing_sk_yds': -15, 'fumbles_forced': 1, 'fumbles_rec': 2, 'fumbles_notforced': 1}
(PHI, OPP 3, Q2, 3 and 3) (8:34) M.Vick sacked at CHI 17 for -14 yards (J.Peppers). FUMBLES (J.Peppers), and recovers at CHI 18. M.Vick to CHI 18 for no gain. FUMBLES, and recovers at CHI 18. M.Vick to CHI 18 for no gain (H.Melton).
-------------------------------------------------------------------------------
2010110710 {'fumbles_tot': 2, 'fumbles_forced': 1, 'fumbles_lost': 1, 'fumbles_rec': 1, 'fumbles_notforced': 1, 'defense_tkl': 1, 'puntret_tot': 1, 'puntret_yds': 5}
(KC, OWN 15, Q4, 4 and 10) (9:02) D.Colquitt punts 53 yards to OAK 32, Center-T.Gafford. N.Miller MUFFS catch, and recovers at OAK 25. N.Miller to OAK 30 for 5 yards (D.Williams). FUMBLES (D.Williams), RECOVERED by KC-V.Tucker at OAK 30. V.Tucker to OAK 30 for no gain (N.Miller).
-------------------------------------------------------------------------------
2010102411 {'fumbles_tot': 2, 'fumbles_forced': 1, 'fumbles_rec': 1, 'fumbles_notforced': 1, 'puntret_tot': 1, 'puntret_yds': 10}
(OAK, OWN 35, Q2, 4 and 14) (5:29) S.Lechler punts 46 yards to DEN 19, Center-J.Condo. S.Thompson MUFFS catch, and recovers at DEN 19. S.Thompson to DEN 29 for 10 yards (J.Condo). FUMBLES (J.Condo), recovered by DEN-D.Bruton at DEN 32. D.Bruton to DEN 32 for no gain (R.Cartwright).
-------------------------------------------------------------------------------
2010101012 {'fumbles_tot': 2, 'fumbles_forced': 1, 'fumbles_lost': 1, 'fumbles_notforced': 1, 'rushing_yds': 3, 'rushing_att': 1}
(SF, OWN 42, Q1, 1 and 10) (:23) F.Gore right guard to SF 45 for 3 yards (M.Fokou). FUMBLES (M.Fokou), RECOVERED by PHI-E.Sims at SF 45. E.Sims to SF 45 for no gain (J.Staley).
-------------------------------------------------------------------------------
2010101009 {'passing_sk': 1, 'fumbles_tot': 2, 'fumbles_notforced': 1, 'fumbles_forced': 1, 'rushing_att': 1}
(ARI, OPP 1, Q2, 3 and 1) (2:31) (Shotgun) M.Hall sacked at NO 2 for -1 yards (T.Hargrove). FUMBLES (T.Hargrove) [T.Hargrove], recovered by ARI-L.Brown at NO 2. L.Brown for 2 yards, TOUCHDOWN. New Orleans challenged the fumble ruling, and the play was Upheld. (Timeout #3 at 02:19.)
-------------------------------------------------------------------------------
2010101007 {'fumbles_tot': 2, 'fumbles_forced': 1, 'fumbles_lost': 1, 'fumbles_notforced': 1, 'rushing_yds': 1, 'rushing_att': 1}
(CLE, OWN 43, Q1, 2 and 7) (14:01) P.Hillis left guard to CLE 44 for 1 yard (J.Babineaux). FUMBLES (J.Babineaux), RECOVERED by ATL-W.Moore at CLE 47. W.Moore to CLE 47 for no gain (J.Thomas).
-------------------------------------------------------------------------------
2010101004 {'fumbles_tot': 2, 'fumbles_forced': 1, 'fumbles_notforced': 1, 'defense_frec_yds': 9, 'defense_frec': 1}
(STL, OPP 9, Q1, 1 and 9) (8:01) (Shotgun) S.Bradford pass short left to D.Amendola to DET 6 for 3 yards (C.Williams). FUMBLES (C.Williams), RECOVERED by DET-A.Smith at DET 4. A.Smith to DET 13 for 9 yards (M.Gilyard). FUMBLES (M.Gilyard), recovered by DET-J.Wade at DET 17. J.Wade to DET 27 for 10 yards (B.Gibson). Play Challenged by STL and Upheld. (Timeout #1 at 07:45.)
-------------------------------------------------------------------------------
2010101004 {'fumbles_tot': 2, 'fumbles_forced': 1, 'fumbles_lost': 1, 'receiving_yac_yds': 1, 'fumbles_notforced': 1, 'receiving_rec': 1, 'receiving_yds': 3, 'receiving_tar': 1}
(STL, OPP 9, Q1, 1 and 9) (8:01) (Shotgun) S.Bradford pass short left to D.Amendola to DET 6 for 3 yards (C.Williams). FUMBLES (C.Williams), RECOVERED by DET-A.Smith at DET 4. A.Smith to DET 13 for 9 yards (M.Gilyard). FUMBLES (M.Gilyard), recovered by DET-J.Wade at DET 17. J.Wade to DET 27 for 10 yards (B.Gibson). Play Challenged by STL and Upheld. (Timeout #1 at 07:45.)
-------------------------------------------------------------------------------
2010092614 {'fumbles_tot': 2, 'fumbles_forced': 1, 'fumbles_rec': 1, 'fumbles_notforced': 1, 'rushing_yds': 3, 'rushing_att': 1}
(MIA, OPP 38, Q2, 1 and 10) (11:35) R.Williams right guard to NYJ 35 for 3 yards (H.Green, K.Wilson). FUMBLES (H.Green), and recovers at NYJ 35.
-------------------------------------------------------------------------------
2010091907 {'fumbles_tot': 2, 'fumbles_forced': 1, 'fumbles_lost': 1, 'fumbles_notforced': 1, 'rushing_yds': 4, 'rushing_att': 1}
(ATL, OWN 44, Q2, 1 and 10) (:39) (Shotgun) M.Ryan scrambles up the middle to ATL 48 for 4 yards (J.Porter). FUMBLES (J.Porter), RECOVERED by ARI-G.Toler at ARI 46. G.Toler to ARI 46 for no gain (J.Blalock).
-------------------------------------------------------------------------------
2010091208 {'fumbles_tot': 2, 'fumbles_forced': 1, 'fumbles_lost': 1, 'defense_frec': 1, 'fumbles_notforced': 1, 'defense_frec_yds': 18}
(ARI, OWN 45, Q3, 1 and 14) (7:02) (Shotgun) D.Anderson sacked at ARI 36 for -9 yards (C.Ah You). FUMBLES (C.Ah You), RECOVERED by SL-C.Ryan at ARI 23. C.Ryan to ARI 5 for 18 yards (S.Breaston). FUMBLES (S.Breaston), RECOVERED by ARI-L.Sendlein at ARI 0. Touchback.
-------------------------------------------------------------------------------
2010091208 {'fumbles_tot': 2, 'fumbles_forced': 1, 'receiving_yac_yds': -8, 'fumbles_notforced': 1, 'receiving_rec': 1, 'receiving_yds': -5, 'receiving_tar': 1}
(STL, OPP 23, Q2, 2 and 8) (6:51) S.Bradford pass short right to B.Bajema to ARI 19 for 4 yards (G.Toler). FUMBLES (G.Toler), recovered by SL-S.Bradford at ARI 28. S.Bradford to ARI 28 for no gain (D.Dockett).
-------------------------------------------------------------------------------
2010090266 {'fumbles_tot': 2, 'fumbles_forced': 1, 'fumbles_lost': 1, 'fumbles_rec': 1, 'fumbles_notforced': 1, 'puntret_tot': 1, 'puntret_yds': 7}
(WAS, OWN 26, Q4, 4 and 9) (1:09) J.Bidwell punts 48 yards to ARI 26, Center-N.Sundberg. J.Miller MUFFS catch, and recovers at ARI 27. J.Miller to ARI 34 for 7 yards (R.Torain). FUMBLES (R.Torain), RECOVERED by WAS-R.Robinson at ARI 37. R.Robinson to ARI 37 for no gain (M.Washington).
-------------------------------------------------------------------------------
2010082051 {'fumbles_tot': 2, 'fumbles_forced': 1, 'receiving_yac_yds': 5, 'fumbles_notforced': 1, 'receiving_rec': 1, 'receiving_yds': 22, 'receiving_tar': 1}
(CIN, OWN 39, Q3, 1 and 10) (4:28) J.O'Sullivan pass deep middle to J.Simpson to PHI 39 for 22 yards (D.Patterson). FUMBLES (D.Patterson), recovered by CIN-C.Coffman at PHI 37. C.Coffman to PHI 37 for no gain (Q.Demps).
-------------------------------------------------------------------------------
2010082051 {'fumbles_tot': 2, 'fumbles_forced': 1, 'fumbles_lost': 1, 'receiving_yac_yds': -1, 'fumbles_notforced': 1, 'receiving_rec': 1, 'receiving_yds': 8, 'receiving_tar': 1}
(PHI, OPP 36, Q1, 3 and 9) (1:24) (Shotgun) K.Kolb pass short left to J.Maclin to CIN 28 for 8 yards (Dh.Jones). FUMBLES (Dh.Jones), RECOVERED by CIN-J.Fanene at CIN 25. J.Fanene to CIN 25 for no gain (J.Peters).
-------------------------------------------------------------------------------
2010081454 {'fumbles_tot': 2, 'fumbles_notforced': 1, 'fumbles_forced': 1, 'rushing_att': 1, 'fumbles_lost': 1}
(HOU, OPP 1, Q2, 3 and 1) (2:28) S.Slaton up the middle to ARI 1 for no gain (P.Togafau). FUMBLES (P.Togafau), RECOVERED by ARI-D.Washington at ARI 0. Touchback.
-------------------------------------------------------------------------------
2010081352 {'fumbles_tot': 2, 'fumbles_forced': 1, 'fumbles_rec': 1, 'fumbles_notforced': 1, 'rushing_yds': 1, 'rushing_att': 1}
(PHI, OWN 35, Q3, 3 and 6) (12:05) (Shotgun) M.Vick to PHI 35 for no gain (L.Douzable). FUMBLES (L.Douzable), and recovers at PHI 35. M.Vick to PHI 36 for 1 yard.
-------------------------------------------------------------------------------
2009121302 {'fumbles_tot': 2, 'fumbles_forced': 1, 'fumbles_lost': 1, 'fumbles_rec': 1, 'fumbles_notforced': 1, 'rushing_yds': -12, 'rushing_att': 1}
(GB, OPP 29, Q3, 1 and 10) (12:56) A.Rodgers to CHI 36 for -7 yards. FUMBLES, and recovers at CHI 36. A.Rodgers to CHI 38 for -2 yards (H.Hillenmeyer). FUMBLES (H.Hillenmeyer), RECOVERED by CHI-A.Adams at CHI 41. A.Adams to CHI 44 for 3 yards (S.Wells).
-------------------------------------------------------------------------------
Some of the plays are actually instances of a player fumbling twice.
It looks like the rest are actually errors in the source data that we won't fix. Namely, a single fumble occurs, but it is simultaneously reported as both a forced fumble and not a forced fumble. So either the data is in error, or there is more nuance that isn't captured by the data.
The plays where fumbling happened twice turn out to be cool candidates to watch! http://fantasy.burntsushi.net/query?refresh=0&my_players=0&limit=500&sort=-game_date&sort=-play_id&play_fumbles_tot__ge=2&entity=play
Man, you can locate some pretty wacky plays very easily.
The shell command to delete the relevant games from the database was prompting me for the password for every game so I used the following script instead.
import nfldb
import psycopg2
try:
db_pass = nfldb.db.config()[0]['password']
conn = psycopg2.connect(database="nfldb", user="nfldb", password=db_pass)
except:
print "I am unable to connect to the database."
cur = conn.cursor()
db = nfldb.connect()
q = nfldb.Query(db)
for pp in q.play_player(fumbles_tot__ge=2).as_play_players():
print pp.gsis_id
SQL = """DELETE FROM game WHERE gsis_id = %s;"""
data = (pp.gsis_id,)
cur.execute(SQL, data)
conn.commit()
cur.close()
conn.close()
@andr3w321 FYI, you can remove the password prompt with an appropriately configured $HOME/.pgpass
. Docs: http://www.postgresql.org/docs/9.3/static/libpq-pgpass.html
I'm trying to figure out why there is a difference between nflgame.combine_plays and nflgame.combine_max_stats. Looking at Matt Jones from Washington's game last night, and max shows as 2, but combine_plays only shows 1 for fumbles.
Here is the code that shows the difference:
import nflgame games = nflgame.games(year=2015, week=3, home='WAS', away='WAS') playsCP = nflgame.combine_plays(games) playsCMS = nflgame.combine_max_stats(games) Player = "M.Jones"
for play in playsCMS: if play.name == Player: print "Combine Plays:", play.name, "Fumble Total:", play.fumbles_tot
for playCP in playsCP: for player in playCP.players.filter(fumbles_tot__ge=1): if player.name == Player: print "Combine Max Stat:", playCP