Motivating example in chapter 1

[sivark]

Chapter 1 starts with an interesting example: a programmer who updates his/her estimate of the probability of some code being buggy, conditioned on the knowledge that it passes certain test cases. While this is a great example of inferential reasoning, I wonder whether this situation lends itself better to a Frequentist description rather than a Bayesian one, for the following reasons:

There is a "ground truth" to whether the program is buggy or not, which is more naturally modelled by an unknown parameter rather than a random variable. The inference we make is usually akin to a point estimator (rather than a posterior on the bugginess random variable). Further, there is a natural Frequentist interpretation in terms of p-values and confidence intervals. "Given that the first few test cases have passed, I expect $X%$ of my programs to be bug free."

Of course, one could definite study this problem using a Bayesian formalism, and get correct answers.

I'd like to know the author's thoughts on having this as a motivating example, and whether it might be worth considering another example. Needless to say, this book is a great resource -- I don't mean to detract from it -- only to make it more awesome! :-)

[CamDavidsonPilon]

Hey @sivark, this is a great insight, I'll leave some comments inline:

There is a "ground truth" to whether the program is buggy or not

This is perfectly inline with Bayesian inference: in most problems there is a ground truth - it's simply our belief in what the ground truth that is modelled as a random variable. The next example in the book, where I don't know what the bias of a coin is, is another example of this: the coin really has a ground-truth bias of 0.5, but my belief in what the bias is is a random variable.

It is, in fact, very unlikely that the ground-truth is indeed a random variable. True stochasticism occurs somewhat infrequently in real life (I can think of quantum physics as one example) - most often randomness is the result of lack of information (ex: I don't know what the next card in the deck will be, but there is a deterministic ground-truth). This uncertainty is what we model as a random variable.

Further, there is a natural Frequentist interpretation in terms of p-values and confidence intervals. "Given that the first few test cases have passed, I expect $X%$ of my programs to be bug free."

In most statistics problems, there is an appropriate Frequentist solution as well. Most examples in this book also have a frequentist solution, but I chose to keep them out to show off that alternative side of the inference coin.

Does that help?

[sivark]

Hey @CamDavidsonPilon, I agree with everything you've said. I'm trying to emphasize that there are other situations, where a Bayesian approach is pretty natural, but a Frequentist interpretation is quite weird. (Such examples would help show off the power of the Bayesian approach by contrasting it with regular experience)

Consider the following example problems:

1. Which team will win the next soccer world cup?
2. Will the Github platform (suitably defined) survive till the year 2150 A.D.?

At some time in the future, these questions will be settled, and there will be no uncertainty in the answer -- but it doesn't make sense to say that a ground truth already exists. The future could play out in a multitude of ways. Also, in these cases, the random variable in question will only have a single realization (there are no realizable ensembles of independent, identically distributed world cup victories or Github futures). So, Frequentist performance notions such as "this estimator will work in 95% of attempts" don't make any sense. IMHO, such examples illustrate the unique potency of Bayesian inference.

very unlikely that the ground-truth is indeed a random variable. True stochasticism occurs somewhat infrequently in real life [...] most often randomness is the result of lack of information

^Exactly! Frequentist notions can be applied only when there are many realizations of a random variable characterized by a "ground truth" parameter value (interpreting the probabilistic model of the random variable as corresponding to occurrence frequencies in its realizations).

One could get away with attaching Frequentist notions to bugginess of programs or biases of coins because we encounter many examples of both. This leads to confusion regarding the interpretation of probabilities generated by Bayesian inference: anyone who's studied a little bit of probability in school is unfortunately led to think about frequencies! (whereas one should instead be thinking of betting odds that one might be willing to accept)

Bayesian approaches are particularly potent when there is a single realization of the data, and our knowledge of something is intrinsically uncertain. My rumination was regarding whether that would be worth elaborating on, and emphasizing, in the introduction.

[CamDavidsonPilon]

Actually, I really like this. Let me think more about this.