Probability and Statistics Reading List

A compilation of books for learning probability and statistics from scratch.

** Some notes

This repository contains an analysis of the commonly recommended books for learning probability and statistics from around the web.

One thing to keep in mind is that advanced mathematics in areas like analysis and measure theory might become more important as we delve deeper into topics. Or may be even before that since things are deeply intertwined all across math. Some of the close adjacent areas to explore after this seems to be information theory, and statistical mechanics/thermodynamics as they can be seen commonly sprucing up in books addressing the topics of probability that I have examined here. I have attempt to put the link to resources directly to those that are freely available and included Amazon affiliate links to others. I have also attempt to link to the places where the resource was recommended from.

TODO: Tag the resources that are free using an icon/tag

** On the Categorization

The books have been organized into 3 sections, that are ordered as: Introductory → Elementary → Rigorous. The various books I have sampled are categorized into these three sections and within these sections, the books are unordered, which means, I don’t recommend any sequence of reading them inside these broader categories.

Apart from these, as adjuncts, I have also added some history books and topics that are adjacent to probability and statistics one might enjoy learning.

Within the Elementary section, I have tried to categorize the books along 2 axes, { logical, empirical } × { visual, symbolic }.

By a symbolic vs. visual approach, I intend to cover those which take a rigorous algebraic approach vs. ones that lean in on geometric/structural intuition of the concepts at play. These are not mutually exclusive categories and books are situated in a spectrum. With logical vs. empirical approach, I intend to locate those books that have an inclination towards describing the underlying formal patterns as deriving from a rational understanding of the space vs. a more data oriented one where case studies from real world are presented and inferences are inducted from this space. Whenever there was an inclination towards either poles, I have done a rounding off and put it off to align with one of the poles. So even when reading a book classified as say, logical/visual, there is good chance that it will have a good amount of symbolic knowledge to teach you.

For statistics, I have tried to mention whether the books take a frequentist or Bayesian approach with the same caveat of mutual inclusion of ideas as above.

Even though, this is a simplistic division and fails to capture the actual complexity of concepts in probability and statistics, it is the best provisional axes I could supply to make sense of these fields. Once my comprehension of the landscape improves, I will try to adapt/revise the bases to have better span and decomposition properties accordingly.

TODO: May be include a 2 × 2 of the book distribution?

TODO: After listing this out, I have felt that may be a better categorization would be { conceptual, computational } × { probability, statistics }. This would have the books that give a conceptual grounding primarily vs. one that has a lot of exercises to solve which would help the reader arrive at an understanding for the relations at play. Try to do this after you put up the concept lattice for the chapters in the second or third pass.

Index

Introductory
[[#probability-theory-first-steps][Probability Theory: First Steps]]
[[#lady-luck][Lady Luck]]
[[#the-lady-tasting-tea][The Lady Tasting Tea]]
[[#the-theory-that-would-not-die][The Theory that Would not Die]]
[[#the-art-of-statistics][The Art of Statistics]]
[[#naked-statistics][Naked Statistics]]

| Logical/Visual | Logical/Symbolic | Empirical/Visual | Empirical/Symbolic | |--------------------------------+-----------------------------------+----------------------------------+------------------------------------| | [[#probability-theory-for-scientists-and-engineers][Probability Theory (for Scientists and Engineers)]] | [[#probability-theory-the-logic-of-science][Probability Theory: The Logic of Science]] | [[#the-world-is-built-on-probability][The World is Built on Probability]] | [[#understanding-uncertainty][Understanding Uncertainty]] | | [[#probability-theory-first-steps][Probability Theory: First Steps]] | [[#probability-for-an-enthusiastic-beginner][Probability Theory for An Enthusiastic Beginner]] | [[#peter-norvig-python-notebooks][Peter Norvig Python Notebooks]] | [[#probability-via-expectation][Probability via Expectation]] | | [[#conditional-probability-explained-visually][Conditional Probability Explained Visually]] | [[#introduction-to-probability][Introduction to Probability]] | [[#bayes-theorem-a-visual-introduction-for-beginners][Bayes Theorem: A Visual Introduction for Beginners]] | [[#principles-of-statistical-inference][Principles of Statistical Inference]] | | | [[#an-introduction-to-probability-and-random-processes][An Introduction to Probability and Random Processes]] | [[#seeing-theory][Seeing Theory]] | [[#all-of-statistics-a-course-in-statistical-inference][All of Statistics]] | | | [[#causal-inference-a-primer][Causal Inference: A Primer]] | | [[#theory-of-probability-a-critical-introductory-treatment][Theory of Probability]] | | | [[#discrete-probability][Discrete Probability]] | | [[#statistical-rethinking][Statistical Rethinking]] | | | | | [[#think-stats][Think Stats]] | | | | | [[#openintro-statistics][OpenIntro Statistics]] | | | | | [[#an-introduction-to-statistical-learning-with-applications-in-r][An Introduction to Statistical Learning: With Applications in R]] | | | | | [[#the-elements-of-statistical-learning][The Elements of Statistical Learning]] | | | | | [[#statistics][Statistics]] | | | | | [[#introduction-to-probability-and-statistics-for-engineers-and-scientists][Introduction to Probability and Statistics for Engineers and Scientists]] | | | | | [[#grinstead-and-snells-introduction-to-probability][Grinstead and Snell’s Introduction to Probability]] | | | | | [[#probability-and-statistics][Probability and Statistics]] | | | | | [[#think-bayes][Think Bayes]] | | | | | [[#understanding-probability][Understanding Probability]] | | | | | [[#blitzstein-and-hwangs-introduction-to-probability][Blitzstein and Hwang’s Introduction to Probability]] |

Rigorous works
[[#introduction-to-probability-theory][Introduction to Probability Theory]]
[[#probability-and-measure][Probability and Measure]]
[[#probability][Probability]]
[[#a-course-in-probability-theory][A Course in Probability Theory]]
[[#a-probability-path][A Probability Path]]
[[#measure-theory-and-probability-theory][Measure Theory and Probability Theory]]
[[#probability-with-martingales][Probability with Martingales]]
[[#probability-theory-and-examples][Probability: Theory and Examples]]
[[#the-foundations-of-statistics][The Foundations of Statistics]]
[[#probability-theory-and-stochastic-processes-with-applications][Probability Theory and Stochastic Processes With Applications]]
[[#probability-theory-a-comprehensive-course][Probability Theory: A Comprehensive Course]]
[[#probability-and-random-processes][Probability and Random Processes]]

History
[[#taming-of-chance][Taming of Chance]]
[[#the-empire-of-chance][The Empire of Chance]]
[[#the-rise-of-statistical-thinking-1820-1900][The Rise of Statistical Thinking - 1820 – 1900]]
[[#chance-logic-and-intuition]]

Additional reads
[[#against-the-gods][Against the Gods]]
[[#the-misbehaviour-of-markets][The (Mis)Behaviour of Markets]]
[[#information-theory-inference-and-learning-algorithms][Information Theory, Inference, and Learning Algorithms]]
[[#advanced-data-analytics-from-an-elementary-point-of-view][Advanced Data Analytics from an Elementary Point of View]]

** Overview of topics covered

TODO: Add in later by codifying the data on how different books treat different topics.

Introductory Works

These are the introductory works into the topic. The books here are popular introductory works into probability which doesn’t get down into the nitty gritty and as such reading these should be supplemented with more rigorous works. They are added here so that an interested reader who is totally new to this domain can build context and familiarize themselves with the central ideas of this field. This also includes some computational notebooks by Peter Norvig which could be of great help in trying to get hands dirty with the algorithms that one uses when trying to navigate the landscape of probability and statistics.

** [[https://amzn.to/3nyM3v1][Lady Luck]]

+BEGIN_HTML

+END_HTML

Warren Weaver 1982

400 pages

An introduction to probability emphasizing the history of the subject.

** [[https://en.wikipedia.org/wiki/The_Lady_Tasting_Tea][The Lady Tasting Tea]]

+BEGIN_HTML

+END_HTML

*** David Salsburg

April 1, 2001
352 pages

** [[https://amzn.to/30CNY8N][The Theory that would not Die]] Subtitle: How Bayes’ Rule Cracked the Enigma Code, Hunted Down Russian Submarines and Emerged Triumphant from Two Centuries of Controversy

+BEGIN_HTML

+END_HTML

*** Sharon Bertsch McGrayne

*** 2012

360 pages

A talk based on this book is available here: https://www.youtube.com/watch?v=8oD6eBkjF9o

[[./img/the-theory-that-would-not-die-video.jpg]]

The book describes the contest between frequentist and Bayesian approaches. It has less mathematics and computation using the mathematical concepts and is rather narrative oriented about how the different ideas panned out.

** [[https://amzn.to/31od7UW][The Art Of Statistics]] Subtitle: How to Learn from Data

David Spiegelhalter

2020

+BEGIN_HTML

+END_HTML

An easy introduction into the world of statistics

** [[https://amzn.to/3rrdZmU][Naked Statistics]]

Charles Wheelan

2014

+BEGIN_HTML

+END_HTML

A narrative driven account of the field of statistics.

Logical / Visual

** [[https://betanalpha.github.io/assets/case_studies/probability_theory.html][Probability Theory (for Scientists and Engineers)]]

[[./img/probability-theory-for-scientists-and-engineers.png]]

Michael Betancourt

October 2018

An online book that provides visual intuition into the ideas of probability along with a good ground work for the mathematical symbolic language that undergirds modern probability theory. The topics are touched upon in a rather cursory manner and might need the support of some other books to thoroughly unravel the underpinnings.

There is also a follow up book from here under [[https://betanalpha.github.io/assets/case_studies/modeling_and_inference.html][Probabilistic Modeling and Statistical Inference]]

** [[https://archive.org/details/ProbabilityTheoryfirstSteps/mode/2up][Probability Theory: First Steps]]

+BEGIN_HTML

+END_HTML

An introduction to probability theory in popular language

** [[http://setosa.io/conditional/][Conditional Probability Explained Visually]]

[[./img/conditional-probability-explained-visually.png]]

Victor Powell

2014

Blog post

A neat visualization of conditional probability by Victor Powell

Logical / Symbolic

+BEGIN_HTML

+END_HTML

Judea Pearl, Madelyn Glymour, Nicholas P. Jewell

160 pages

2016

Might be a nice book to start reading after The Book of Why to get into some of the nitty gritty on inference from data. There seems also to be a more rigorous work on Causality by Pearl in [[https://amzn.to/3CTFLux][Causality: Models, Reasoning, and Inference]]

** [[https://amzn.to/3dbBGar][Discrete Probability]]

Hugh Gordon

2012

+BEGIN_HTML

+END_HTML

A book explaining discrete probability in an accessible language.

Empirical / Symbolic

Frequentist introduction to statistical inference giving a comparison of various approaches.

** [[https://amzn.to/3xM0Qpq][All of Statistics: A Course in Statistical Inference]]

+BEGIN_HTML

+END_HTML

Larry Wasserman

A thorough book giving sound overview of the fundamentals of statistics.

** [[https://amzn.to/3dcA8ge][Statistics]]

+BEGIN_HTML

+END_HTML

David Freedman, Robert Pisani, Roger Purves

4th Edition (2007)

A book directed towards people with minimal mathematics exposure. The organization of the book in helping build the intuition gradually is remarked by people have read it.

** [[https://amzn.to/3xMEsw9][Introduction to Probability and Statistics for Engineers and Scientists]]

Sheldon Ross

First Edition: 1987, 6th Edition: 2020

+BEGIN_HTML

+END_HTML

Used as a common textbook in many universities.

** [[https://chance.dartmouth.edu/teaching_aids/books_articles/probability_book/book.html][Grinstead and Snell’s Introduction to Probability]]

Charles M. Grinstead, J. Laurie Snell

1997

528 pages

+BEGIN_HTML

+END_HTML

An open source text book to learn probability that takes a calculus based approach instead of combinatorial one.

** [[https://amzn.to/3odDsOI][Probability and Statistics]]

Morris DeGroot, Mark Schervish

First Edition: 1975, Fourth Edition: 2013

+BEGIN_HTML

+END_HTML

A highly recommended book for learning both probability and statistics.

** [[https://greenteapress.com/wp/think-bayes/][Think Bayes]]

Allen Downey

+BEGIN_HTML

+END_HTML

2012

An introduction to Bayesian probability in the style of [[#think-stats][Think Stats]]

** [[https://amzn.to/31oVpAI][Understanding Probability]]

Henk Tijms

198 pages

An introduction to the subject of probability motivated by examples from decision making, control theory, biology, and quantum mechanics. Was originally published in Russian and translated to English.

** [[https://github.com/norvig/pytudes/][Peter Norvig Python Notebooks]]

There are some really well written computational notebooks by Peter Norvig elucidating the probabilty concepts.

TODO: Add images for each of the Python notebooks

*** [[https://github.com/norvig/pytudes/blob/main/ipynb/Probability.ipynb][A Concrete Introduction to Probability]]

[[./img/a-concrete-introduction-to-probability.png]]

*** [[https://github.com/norvig/pytudes/blob/main/ipynb/ProbabilityParadox.ipynb][Probability, Paradox, and the Reasonable Person Principle]]

[[./img/probability-paradox-and-the-reasonable-person-principle.png]]

*** [[https://github.com/norvig/pytudes/blob/main/ipynb/ProbabilitySimulation.ipynb][Estimating Probabilities with Simulations]]

[[./img/estimating-probabilities-with-simulations.png]]

** [[https://amzn.to/3Edl51Q][Bayes Theorem: A Visual Introduction for Beginners]]

Dan Morris

114 pages

+BEGIN_HTML

+END_HTML

A short and quick introduction to Bayes Theorem.

** [[https://seeing-theory.brown.edu/index.html][Seeing Theory]]

+BEGIN_HTML

+END_HTML

Daniel Kunin, Jingru Guo, Tyler Dae Devlin, Daniel Xiang

An interactive website introducing the concepts of probability and statistics.

History

An overview of the history would benefit by providing the motivation and original scenarios in which the concepts originated. They are also a good way for people looking to research into this area to understand some of the original strands and possible find a wealth of problems that are linked with the genesis of the ideas.

** [[https://amzn.to/3FZZaf8][The Taming of Chance]]

Ian Hacking

1990

+BEGIN_HTML

+END_HTML

A standard recommendation to learn about the origins of probability ad statistics. This book, along with [[https://amzn.to/3dalnuF][The Emergence of Probability]] by Hacking, which has a more philosophical bent, might serve as a decent bundle for exposure to the historical details on how probability took shape as a science.

** [[https://amzn.to/3FARmQM][The Empire of Chance: How Probability Changed Science and Everyday Life]]

+BEGIN_HTML

+END_HTML

Gerd Gigerenzer, Zeno Swijtink, Theodore Porter, Lorraine Daston, John Beatty, Lorenz Krüger

October 26, 1990

360 pages

History of modern statistics and its connections with other domains of knowledge.

** [[https://amzn.to/30TNewk][The Rise of Statistical Thinking - 1820 – 1900]]

Theodore M. Porter

August 18, 2020

360 pages

+BEGIN_HTML

+END_HTML

History of the subject with more of an academic bent. A general reader might find Ian Hacking’s work more approachable.

** [[https://archive.org/details/ofmathemahistory00todhrich][A history of mathematical theory of probability: from the time of Pascal to Laplace]] I. Todhunter

1865

** [[][Chance, Logic, And Intution: An Introduction to the Counter-Intuitive Logic of Chance]] Steven Tjims

18 February 2021 256 pages

TODO: Add link and brief

Videos

https://www.youtube.com/playlist?list=PL17567A1A3F5DB5E4

Rigorous works

** [[https://amzn.to/3Ekhsay][Introduction to Probability Theory]]

Paul G. Hoel, Sidney C. Port, Charles J. Stone

1972

They also have a similar book on [[https://amzn.to/3ohkRBp][Statistical Theory]].

A rigorous introduction to probability theory. It has been likened to Rudin’s book on mathematical analysis.

** [[https://amzn.to/3xMzJuB][Probability and Measure]] Patrick Billingsley

2012

A self-contained book on probability and commonly recommended as a rigorous introduction to the subject.

** [[https://amzn.to/3dpIukI][Probability]] Jim Pitman

** [[https://amzn.to/31qCpSM][A Course in Probability Theory]] Kai Lai Chung

** [[https://amzn.to/3xUJYNw][A Probability Path]] Sidney Resnick

** [[https://amzn.to/3lDbFFB][Measure Theory and Probability Theory]] Krishna B. Athreya, Soumendra N. Lahiri

** [[http://www.stat.columbia.edu/~gelman/book/][Bayesian Data Analysis]]

Andrew Gelman, John B. Carlin, Hal S. Stern, David B. Dunson, Aki Vehtari, Donald B. Rubin

First Edition: 1995, Second Edition: 2003, Third Edition: 2013

+BEGIN_HTML

+END_HTML

** [[https://amzn.to/3oq7A9M][Probability with Martingales]] David Williams

** [[https://amzn.to/3ooj5yr][Probability: Theory and Examples]] Rick Durrett

** [[https://amzn.to/3okKzF4][The Foundations of Statistics]] Leonard J. Savage

** [[https://amzn.to/3EpJ7qF][Probability Theory and Stochastic Processes With Applications]] Oliver Knill

** [[https://web.stanford.edu/~hastie/CASI/][Computer Age Statistical Inference]] Bradley Efron, Trevor Hastie

** [[https://amzn.to/3rykG6R][Statistical Inference]]

George Casella and Roger Berger

Typically used in many universities as the starting text. [[https://stats.stackexchange.com/questions/353138/casella-and-berger-vs-wasserman-to-acquire-a-good-statistics-foundation][Apparently]] more rigorous and more focused on technical details than [[#all-of-statistics][All of Statistics]].

** [[https://amzn.to/31lXjlA][An Introduction to Probability Theory and Its Applications]] William Feller

** [[https://amzn.to/3ddOXPL][Probability Theory: A Comprehensive Course]] Achim Klenke

Recommended as a reference book on probability

** [[https://amzn.to/3DoL4Cs][Probability and Random Processes]] Geoffrey R. Grimmett, David R. Stirzaker

Considered a standard reference to the subject

** [[https://amzn.to/3IdxLIs][Data Analysis Using Regression and Multilevel/Hierarchical Analysis]]

Andrew Gelman, Jennifer Hill

2006

Additional Resources

** [[https://amzn.to/3xTAWAl][Against the Gods]] Peter L. Bernstein 1998

** [[https://amzn.to/3IoaOlO][The (Mis)Behaviour of Markets]] Benoit B. Mandelbrot, Richard L Hudson

** [[http://www.inference.org.uk/itila/book.html][Information Theory, Inference, and Learning Algorithms]] David MacKay If you want to have an Information Theory bend

** [[http://www.stat.cmu.edu/~cshalizi/ADAfaEPoV/][Advanced Data Analysis from an Elementary Point of View]] Cosma Rohilla Chalizi

** [[https://www.softcover.io/downloads/bf34ea25/math_for_finance][Mathematical Preparation for Finance]] Kaisa Taipale

An idiosyncratic tour through ideas in probability, calculus, linear algebra, and statistics

Sampled but not included / need another pass

** [[https://web.math.princeton.edu/~nelson/books/rept.pdf][Radically Elementary Probability Theory]] Edward Nelson

** [[https://amzn.to/3luibyC][Mastering Metrics]]

The book has an econometric viewpoint towards how to infer cause and effect using statistics.

** [[https://amzn.to/3Dkbkxv][Probability and Statistics for Engineers and Scientists]] Anthony Hayter

** [[https://amzn.to/3EoYXBG][The Probabilistic Method]] Noga Alon and Joel H. Spencer

** [[https://amzn.to/3rDq6xg][Statistics: Learning in the presence of variation]] Robert L. Waldrop

** [[https://amzn.to/3GdZx5L][A Natural Introduction to Probability]]

** [[https://amzn.to/3dlEKAW][Introduction to Statistics]] Jim Frost

** [[https://michael-franke.github.io/intro-data-analysis/][An Introduction to Data Analysis]] Michael Franke

prathyvsh / probability-statistics-reading-list

readme

+BEGIN_HTML

+END_HTML

+BEGIN_HTML

+END_HTML

+BEGIN_HTML

+END_HTML

+BEGIN_HTML

+END_HTML

+BEGIN_HTML

+END_HTML

+BEGIN_HTML

+END_HTML

+BEGIN_HTML

+END_HTML

+BEGIN_HTML

+END_HTML

+BEGIN_HTML

+END_HTML

+BEGIN_HTML

+END_HTML

+BEGIN_HTML

+END_HTML

+BEGIN_HTML

+END_HTML

+BEGIN_HTML

+END_HTML

+BEGIN_HTML

+END_HTML

+BEGIN_HTML

+END_HTML

+BEGIN_HTML

+END_HTML

+BEGIN_HTML

+END_HTML

+BEGIN_HTML

+END_HTML

+BEGIN_HTML

+END_HTML

+BEGIN_HTML

+END_HTML

+BEGIN_HTML

+END_HTML

+BEGIN_HTML

+END_HTML

+BEGIN_HTML

+END_HTML

+BEGIN_HTML

+END_HTML

+BEGIN_HTML

+END_HTML

+BEGIN_HTML

+END_HTML

+BEGIN_HTML

+END_HTML

+BEGIN_HTML

+END_HTML

+BEGIN_HTML

+END_HTML

+BEGIN_HTML

+END_HTML

+BEGIN_HTML

+END_HTML

+BEGIN_HTML

+END_HTML

+BEGIN_HTML

+END_HTML

+BEGIN_HTML

+END_HTML

+BEGIN_HTML

+END_HTML

+BEGIN_HTML

+END_HTML