And finally, some information on Mutual Information:
its relationship with entropy;
how it can be interpreted as an error metric in machine learning (e.g. "Mutual information is calculated between two variables and measures the reduction in uncertainty for one variable given a known value of the other variable.").
and an important remark that most people do not know, the base of that $\log$ operator is typically 2 in that formulation, as this is usually utilised on binary information channels.
Hi
These are 4 topics that usually are mentioned in talks, examinations and job interviews. Can you add them?
In the independence of random variables (section 6.4.5) you have the following properties
However, an important property is missing:
For completion, add the formulation of cross-entropy (and mention its relevance to classification problems):
And finally, some information on Mutual Information:
And thank you, great book!