[Review] [Turing Data Story] Baseball story

[crangelsmith]

Story Review:

Story Name: Baseball story

Submitting Author: @edaub (Eric Daub)

Pull Request: #126

Reviewers: @jack89roberts @nbarlowATI

Reviewer instructions & questions

@jack89roberts, @nbarlowATI, please carry out your review in this issue by updating the checklist below (you can copy it in a comment and fill it individually), and writing new comments in case you have any questions. If you cannot edit the checklist please:

• Make sure you're logged in to your GitHub account
• Be sure to accept the invite at this URL: https://github.com/alan-turing-institute/TuringDataStories/settings/access

Any questions, concerns or suggestions regarding the review process please let @crangelsmith, @DavidBeavan or @samvanstroud know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Review Checklist

Code of conduct

• [ ] I confirm that I read and will adhere to the Turing Data Stories code of conduct.

General checks

• [ ] Notebook: Is the source code for this data story available as a notebook in the linked pull request?
• [ ] Contribution and authorship: Are the authors clearly listed? Does the author list seem appropriate and complete?
• [ ] Scope and eligibility: Does the submission contain an original and complete analysis of open data? Is the story aligned with the Turing Data Stories vision statement?

Reproducibility

• [ ] Does the notebook run in a local environment?
• [ ] Does the notebook build and run in binder?
• [ ] Are all data sources openly accessible and properly cited with a link?
• [ ] Are the data open, and do they have an explicit licence, provenance and attribution?

Pedagogy

• [ ] Does the story demonstrate some specific data analysis or visualisation techniques?
• [ ] Are these techniques well motivated?
• [ ] Are these techniques well implemented?
• [ ] Is the notebook well documented, using both markdown cells and comments in code cells?
• [ ] Does the notebook has a introduction section motivating the story?
• [ ] Does the notebook has a conclusion section discussing the main insight from the stories?
• [ ] Is the paper well written (it does not require editing for structure, language, or writing quality)?

Context

• [ ] Does the story give an insight into some societal issue?
• [ ] Is the context around this issue well referenced (newspaper articles, scientific papers, etc.)?

Ethical

• [ ] Is any linkage of datasets in the story unlikely to lead to an increased risk of the personal identification of individuals?
• [ ] Is the Story truthful and clear about any limitations of the analysis (and potential biases in data)?
• [ ] Is the Story unlikely to lead to negative social outcomes, such as (but not limited to) increasing discrimination or injustice?

AOB

[jack89roberts]

Code of conduct

• [x] I confirm that I read and will adhere to the Turing Data Stories code of conduct.

General checks

• [x] Notebook: Is the source code for this data story available as a notebook in the linked pull request?
• [x] Contribution and authorship: Are the authors clearly listed? Does the author list seem appropriate and complete?
• [x] Scope and eligibility: Does the submission contain an original and complete analysis of open data? Is the story aligned with the Turing Data Stories vision statement?

Reproducibility

• [x] Does the notebook run in a local environment?
• [ ] Does the notebook build and run in binder?
  • sklearn should be scikit-learn in environment.yml, suggested the change in the PR
• [x] Are all data sources openly accessible and properly cited with a link?
• [x] Are the data open, and do they have an explicit licence, provenance and attribution?

Pedagogy

• [x] Does the story demonstrate some specific data analysis or visualisation techniques?
• [x] Are these techniques well motivated?
• [x] Are these techniques well implemented?
• [x] Is the notebook well documented, using both markdown cells and comments in code cells?
• [x] Does the notebook has a introduction section motivating the story?
• [x] Does the notebook has a conclusion section discussing the main insight from the stories?
• [ ] Is the paper well written (it does not require editing for structure, language, or writing quality)?
  • Very well written, just one cut off sentence that needs to be fixed (see separate comment)

Context

• [x] Does the story give an insight into some societal issue?
  • Not sure whether baseball counts, but even if not it's nicely linked to marginal utility in economics and other fields!
• [ ] Is the context around this issue well referenced (newspaper articles, scientific papers, etc.)?
  • A few more references might be nice, see below.

Ethical

• [x] Is any linkage of datasets in the story unlikely to lead to an increased risk of the personal identification of individuals?
  • Player names and data are already public.
• [x] Is the Story truthful and clear about any limitations of the analysis (and potential biases in data)?
• [x] Is the Story unlikely to lead to negative social outcomes, such as (but not limited to) increasing discrimination or injustice?

AOB

My answer for all of these is yes 😄 , but I'll post some more detailed comments and thoughts below that could be integrated.

• [x] Baseball is not as well known in the UK as in the US, Latin America, and Japan. Did I clarify enough about the game and what the statistics that I used mean?
• [x] Are the SQL queries to get the data not too hard to understand for a novice?
• [x] Is my approach reasonable? There are a few numbers I tweaked to get the stats to line up well with the standard estimates (percentiles used to quantify a "full season" and how many standard deviations below the average in the final estimate), so if these seem like egregiously unreasonable numbers to use then please let me know.
• [x] Did I cover the Bayesian methods adequately?
• [x] Is my coding style reasonable and understandable? I try to write things mostly in functions out of habit, which isn't always straightforward to understand in a literate tool like a notebook.
• [x] Everything looks ok?

[jack89roberts]

Great job @edaub, I really enjoyed reading the story! In response to your specific questions I believe the approach looks reasonable and you definitely do a good job explaining baseball, the queries and the Bayesian methods. I have written some comments/thoughts below on a few bits and pieces I either struggled to get my head around or thought could be expanded upon, should you wish to do so (sorry it got quite long!) Nice work!

• Baseball in general

  • A basic diagram of a baseball field/diamond may be helpful (annotated with terms used in the analysis like S, 2B, 3B, HR where possible)?
  • The definitions are in the text and the data dictionary, but at times I would have found a glossary of terms helpful, mostly of things like AB, BB, HBP, SH, SF, PA etc. that appear either in queries or data frames (maybe it could go in the data section at the beginning).

• SQL queries

  • I think you do a good job of explaining what the queries are doing. The one thing I find a bit tricky is the calculation of fields in rc_base_query, e.g. it may not be obvious why O = AB + SH - SF - H.

• Runs created:

  • I'm not sure I fully understand why this is a good metric/the way it is defined. Naively, I'd expect it to be TB / 4, or TB / 4*PA if it was a rate stat rather than a counting stat. What's the purpose of the success rate term?
  • My instinct was that TB counts only the number of bases the batter advanced from their initial hits (not bases advanced by teammates at the same time, or bases subsequently gained during teammates' hits). Is that correct? What made me unsure were notes like the ones below, as I'm not sure how they influence RC unless TB is incorporating more than I thought:

  • hits tend to advance any teammates already on the bases a different number of bases

  • Extending this to be more accurate requires factoring in base running

  • being more precise in how the runs each team scores can be attributed to each player (i.e. the "a player cannot be on base for themselves" effect).

    • Edit: On reflection, is one nuance that a hitter might have slower teammates on bases in front of them, which limits how many bases they can advance?

• Did I cover the Bayesian methods adequately?

  • At the start of "Multinomial Bayesian Interface" is there maybe a more layman's terms way the posterior could be introduced/defined (currently: "and seek to compute the posterior probability conditioned on the data")?
  • Is it worth discussing why the prior you've selected is reasonable a bit more (or revisiting that later when your results show it was reasonable)?

• Simulating League Seasons

  • Could you add a legend to the plot in "Simulating League Seasons" comparing the simulated and empirical distributions?
  • Theres a sentence that's not finished:

  • Others within that peak near zero have not provided evidence that they are good, so these are not useful in evaluating if

• Gaussian mixture model:

  • At first I didn't fully comprehend why the GMM was needed; my instinct was you could have just calculated the 2.5% quantile directly from the sample. What I missed is that you're calculating the 2.5% quantile of the second Gaussian only, not the full distribution. Maybe this could be emphasised a bit more in the text?

• Replacement runs created:

  • In the summary at the end, perhaps you could comment on how replacement RC could be used by a team, e.g. to identify over-/undervalued players, I think? Maybe out of scope for the story, but could you pick out a player that's much better/worse than their reputation suggests, for example?

• Is my coding style reasonable and understandable?

  • In runs_created should line 12 df.ndim == 1 be season_stats.ndim == 1 (or if not should df be an argument to that function rather than relying on using the variable defined earlier in the notebook)?
  • In project_player_season you call an argument prior instead of priors which is used in other places/functions. I think the prior in project_player_seasons is identical to the priors used elsewhere, so maybe it could be changed to priors as well to be consistent?

• References/links:

  • It might be nice to add a few more links to wider context around baseball and baseball stats, including to things mentioned in the text like "Moneyball by Michael Lewis", "legendary baseball statistician Bill James", "dead ball" era prior to 1920", "the late 1990's shows a spike during the 'steriod' era", and also to terms related to the analysis like "Beta distribution", "Dirichlet distribution" etc.

[edaub]

Thanks for the helpful comments, @jack89roberts. I won't get back to this for a while, but I'll be sure to address these in May when I return to make revisions!

[nbarlowATI]

Excellent work @edaub - I really enjoyed reading this (even if I disagree with the initial croissant premise - see below! :) ) and found it very well explained, interesting and informative!

Review Checklist

Code of conduct

• [x] I confirm that I read and will adhere to the Turing Data Stories code of conduct.

General checks

• [x] Notebook: Is the source code for this data story available as a notebook in the linked pull request?
• [x] Contribution and authorship: Are the authors clearly listed? Does the author list seem appropriate and complete?
• [x] Scope and eligibility: Does the submission contain an original and complete analysis of open data? Is the story aligned with the Turing Data Stories vision statement?

Reproducibility

• [x] Does the notebook run in a local environment?
• Yes, though I had to conda install nb_conda_kernels to get jupyter notebook to use my fresh conda environment (maybe this is a basic thing I ought to have already known, but I was lucky enough that I never came across it before) - without this it was picking up an older sklearn that didn't have `sklearn.utils._testing.
• [ ] Does the notebook build and run in binder?
• I got the following failure message (I think from repo2docker) when trying to build this branch in mybinder.org:

  
  Looking for: ['python=3.7', 'requests', 'numpy', 'pandas', 'geopandas', 'descartes', 'scipy', 'matplotlib', 'seaborn', 'openpyxl', 'jupyter', 'autopep8', 'conda-forge::jupyter_contrib_nbextensions', 'pymc3', 'sklearn']

Encountered problems while solving. Problem: nothing provides requested sklearn

- [x] Are all data sources openly accessible and properly cited with a link?
- [x] Are the data [open](https://opendatahandbook.org/guide/en/what-is-open-data/), and do they have an explicit licence, provenance and attribution?

#### Pedagogy
- [x] Does the story demonstrate some specific data analysis or visualisation techniques?
- [x] Are these techniques well motivated?
- [x] Are these techniques well implemented?
- [x] Is the notebook well documented, using both markdown cells and comments in code cells?
- [x] Does the notebook has a introduction section motivating the story?
- [x] Does the notebook has a conclusion section discussing the main insight from the stories?
- [x] Is the paper well written (it does not require editing for structure, language, or writing quality)?

#### Context
- [x] Does the story give an insight into some societal issue?
- I think it is reasonable to describe a popular professional sport as a societal issue.
- [x] Is the context around this issue well referenced (newspaper articles, scientific papers, etc.)?
- Not so many references, but I think it is well enough explained and motivated.

#### Ethical 

- [x] Is any linkage of datasets in the story unlikely to lead to an increased risk of the personal identification of individuals?
- [x] Is the Story truthful and clear about any limitations of the analysis (and potential biases in data)?
- [x] Is the Story unlikely to lead to negative social outcomes, such as (but not limited to) increasing discrimination or injustice?

## AOB
- Baseball is not as well known in the UK as in the US, Latin America, and Japan. Did I clarify enough about the game and what the statistics that I used mean?

This is tricky...  I can definitely see the risk of giving more detail, and burying the reader in too much information.  However, leaving out some detail does risk things becoming confusing.  For example, I think the difference between "At Bat" and "Plate Appearance" is not really obvious here without looking at the SQL query.  

I would be a bit tempted to go all-in on the basic explanation.  Something like:
 - Every time a batter comes to the plate to face the pitcher, this is a "Plate Appearance"
 - The pitcher will throw the ball, and one of the following will happen:
    - Batter hit by pitch (HBP) - advances to first base.
    - Batter doesn't swing at the ball, ball is outside the "strike zone" - this is a "ball".  Pitcher will throw again. If there are four balls in a Plate Appearance, this is a "Walk" (or "base-on-balls") (BB), batter advances to first base.
    - Batter doesn't swing at the ball, ball is inside the "strike zone", OR the batter swings at the ball and misses - this is a "Strike".  Pitcher will throw again.   If there are three Strikes in a Plate Appearance, the batter is out.
etc. etc.   Could also then mention "batting average", as this is, as far as I can see, the main motivation for having "At Bat" as a separate stat from "Plate Appearance" ?

Maybe this is too much information though - it's definitely a matter of opinion, so I wouldn't argue the case too strongly.

I also didn't quite understand the point about the best players being overvalued because they cannot get on base for themselves...   Is this because although they might get to e.g. 2B, the batters behind them will be worse, so this is less likely to result in a Run?
I would have thought that in this case it's not the level of the player relative to the rest of his team that's important, but relative to the whole league?  (And couldn't this be countered by the fact that the best teams have lots of good players, so good players are more likely to have good team-mates who can bat them in?)

In general though I think the explanations are good, but it might be nice to tell the reader in advance that they are coming - e.g. just before Cell 4, a quick sentence saying that all the acronyms will be explained in the following section.

- [x] Are the SQL queries to get the data not too hard to understand for a novice?
The only tricky one is Cell 5, with the "coalesce" commands - the explanation is good, but I'm torn as to whether this should go before or after the cell itself.    It might also be good to have a sentence here explaining why this command filters out the pitchers - I'm not sure I quite understand that.

- [x] Is my approach reasonable? There are a few numbers I tweaked to get the stats to line up well with the standard estimates (percentiles used to quantify a "full season" and how many standard deviations below the average in the final estimate), so if these seem like egregiously unreasonable numbers to use then please let me know.
- [x] Did I cover the Bayesian methods adequately?
- Yes, really good explanations I think (and useful code snippets!)
- [x] Is my coding style reasonable and understandable? I try to write things mostly in functions out of habit, which isn't always straightforward to understand in a literate tool like a notebook.
- Yes, I think this is excellent, and having everything in functions is useful for any reader who might want to transplant bits and pieces into their own code!
- [x] Everything looks ok?

- I think the plot that is the output of Cell 13, comparing the actual and simulated runs created over a whole season, would benefit from a legend.
- There's a missing half-sentence just after this: "so these are not useful in evaluating if".

## AOAOB
I actually think that Pret A Manger croissants (the basic "all butter" ones) are better than pretty much any other croissants one can find outside of France!

[edaub]

@nbarlowATI Thanks for the helpful comments -- and agreed that if Pret crossaints are replacement level then we're doing pretty well here 😄

[crangelsmith]

I think this story is almost ready to be published. @edaub, @billfinnegan let you some final comments after a proof read he did on it. If you are implementing any more changes based on the comment let us know when is ready and we'll do the rest.

[billfinnegan]

Sorry - put my notes in #126 - pasted again below...

Really great data story - the writing style is clear and accessible, and I like how the intro sets up the concept outside of the details of baseball. Here are a few notes based on a proofread:

• In the Replacement Level section you mention the 80% stat and referred to it on the Wikipedia page for WAR. I scanned the page and couldn't find any reference to replacement being 80% above average - is that based on a calculation of the relative runs in the WAR calc? If so, might be worth explaining, or finding another link, as this rule of thumb is the starting point for your analysis.
• In the Data section, 2nd para, 2nd sentence - I'd change "choose" to "chose". I haven't done a close read of tense, and it might be hard to be consistent (sometimes you are looking back on previous data/analysis, sometimes you are describing what you are doing, other times explaining what you are about to do) but that particular verb jumped out at me.
• End of the data section, you've got two paragraphs starting with "However" which I find a little confusing. Can you drop the first however and start with "Most of the time..."?
• Last para in the data section - I think it might be worth defining "position players" in a brief parenthetical just to make sure that is crystal clear, maybe something like: (the eight defensive fielding positions other than the pitcher). I assume you are ignoring the DH, or at least not getting into that particular quirk.
• At the beginning of the Runs Created section, I might be getting too caught up in the language, but is "Outs occur when a player fails to hit the ball" totally accurate? "Outs occur when the player fails to get a hit" feels like it might be better, distinguishing between generally making contact and being credited with a hit. Or maybe the slightly longer: "Outs occur when a player fails to hit the ball and advance to the bases."
• Small punctuation thing: at the end of Runs Created, I think "etc)." should be "etc.)." Although, at that point we've just got a HR left, so is that worth mentioning?
• In Runs Created, after the plot, in the bulleted caveats, I think the language in the last bullet is still a little confusing: "because they cannot get on base for themselves" - is it the same to say "because they cannot already be on base while they are batting"?
• In the Bayesian intro, 2nd para, I think you need a comma after "or the probability"
• In Simulating full seasons, before the histogram, missing a word in: "variation in the value this hypothetical player" - of this hypothetical player, or this hypothetical player has?
• In Approx the Dist, first sentence, replace "might we expect" with "we might expect" ?
• In the Mixture Models section, after the plot, when describing the different distributions, can you refer to the labels in the plot?
• In the code block that calcs the estimated and heuristic replacement level, can you also calc and output the mean runs created/outs so that is handy for understanding the % below league average?
• When you run the model for every year in baseball history, does that start in 1900? The Lahman data looks like it starts in 1871, but I'm not sure when MLB formally began. Might be worth a mention of the start point and logic for that here or when you first reference the data source.
• 2 para below the baseball history plot, 3rd sentence - "Additional thing to consider" probably needs to start with An or One...
• Towards the end, I think it would be good to add a "Discussion" or "Conclusion" header, maybe starting just below the history plot.
• Penultimate paragraph - extra space between their and hits in 3rd sentence.
• Penultimate paragraph - "is" should be "are" or "teams that produce" changed to "a team that produces"
• At the end, I really wanted you to go back to the % below average stat and what the Bayesian model calculates that to be for each year (unless I'm misunderstanding something about how this is working). Is it possible to create a plot of % below average over time as well (again, if I'm understanding this, the heuristic would just be a line at 80% and the bayesian would range from about 60-75%)? Or maybe just in the text for the conclusion you could reiterate the range of runs created/outs and how that reflects % below league average as these are the key #s at the top of the story.

Let me know if you need me to clarify any of this, and feel free to ignore anything!