Add small details on expectation, variance, independence and Mutual Information

[brunomaga]

Hi

These are 4 topics that usually are mentioned in talks, examinations and job interviews. Can you add them?

• variance of a random variable $V[x]$: how to get from $E[ (x-E[x])^2 ]$ to $E[x^2] - E[x]^2$
• variance of the sum of random vars: how to get from $V[X+Y]$ to $V[X]+V[Y]+ Cov[X,Y] + Cov[Y,X]$
• Law of Large Numbers, and its relevance
• Central Limit Theorem, and its relevance

---

In the independence of random variables (section 6.4.5) you have the following properties

• $p(y \mid x) = p(y)$
• $p(x \mid y) = p(x)$
• $V_{X,Y} [x + y] = V_X[x] + V_Y [y]$
• $Cov{X,Y} [x, y] = 0$, missing mention that this one follows from $Cov{X,Y} [x,y]= E [xy] - E[x] E[y]$.

However, an important property is missing:

• $E[xy] = E[x] E[y]$

---

For completion, add the formulation of cross-entropy (and mention its relevance to classification problems):

• multi-class: $H(p,q) = -\sum_x p(x) \log(q(x))$
• binary: $H(p) = -(y \log p(x) + (1-y) \log (1-p(x)))$

---

And finally, some information on Mutual Information:

• its relationship with entropy;
• how it can be interpreted as an error metric in machine learning (e.g. "Mutual information is calculated between two variables and measures the reduction in uncertainty for one variable given a known value of the other variable.").
• And its formulation and the equalities:
  • $I(X;Y)= ∑{x,y}p{XY}(x,y) \log \frac{p_{XY}(x,y)}{P_X(x) PY(y)} = \mathbb{E} \left[ D{KL} \left( p_{X \mid Y} \mid \mid p_X \right) \right] = H(Y) - H(Y \mid X)$
  • and an important remark that most people do not know, the base of that $\log$ operator is typically 2 in that formulation, as this is usually utilised on binary information channels.

And thank you, great book!