Minor questions on Eq.(2)

bellymonster / Weighted-Soft-Label-Distillation

55 stars 8 forks source link

Minor questions on Eq.(2) #3

Closed HolmesShuan closed 3 years ago

HolmesShuan commented 3 years ago

Hi, I have read your excellent work several times. The bias-variance idea is very interesting!

However, due to my poor knowledge of "bias/variance" theorem, I found the variance term in Equation (2) hard to understand. E.g., How to prove that $\mathbb{E}_D[\mathbb{E}_\mathrm{x}[\mathrm{y}\log\frac{\overline{\mathrm{y}}_{ce}}{\hat{y}_{ce}}]]]=\mathbb{E}_D[D_{KL}(\overline{\mathrm{y}}_{ce},\hat{\mathrm{y}}_{ce})]?$

I have referred to Heskes's paper and it seems that the derivation relies on the normalization constant Z in Equation (1). It is easy to prove that $\frac{\log \overline{\mathrm y}_{ce}}{\mathbb{E}_D[\log \hat{y}_{ce}]}=constant$ if Z in Equation (1) is a constant value. But I still did not find the relationship between this term and Equation (2).

Could you please kindly provide the detailed derivations of the variance term in Equation (2)? Thanks in advance.

lsongx commented 3 years ago

Hi @HolmesShuan , thanks for your question! The reason is that E(y) and E(\bar{y}) are all equals to 1. Please refer to the equation above Eqn 4 from Heskes's paper: an extra \bar{p}(y) in the second row. This follows the definition used in Eqn 2 from Heskes's paper. In their paper, it is defined \int dy a(y) = 1. (constraints in Eqn 2) In our notations, y becomes x and a() becomes y=t(). We originally kept all notations the same as Heskes's paper, but a reviewer thinks the notations are not appropriate thus we changed all notations. Sorry for the confusion! Please let me know if there are any further questions.

HolmesShuan commented 3 years ago

Hi @LcDog , thanks for your quick reply!

I have double-checked the derivation of Heskes's paper and gradually understood the decomposition for the expected error defined on the KL Div. Unfortunately, I still did not get the idea of how E(y) and E(\bar{y}) are all equals to 1 leads to $\mathbb{E}_D[\mathbb{E}_\mathrm{x}[\mathrm{y}\log\frac{\overline{\mathrm{y}}_{ce}}{\hat{y}_{ce}}]]]=\mathbb{E}_D[D_{KL}(\overline{\mathrm{y}}_{ce},\hat{\mathrm{y}}_{ce})]?$

I am really sorry for bothering you again :(

As far as I can tell, $\mathbb{E}_D[\mathbb{E}_x[\mathrm y\log\frac{\overline{\mathrm y}}{\hat{\mathrm y}}]]=\mathbb{E}_D[\mathbb{E}_x[\mathrm y\log\overline{\mathrm y}-\mathrm y\log\hat{\mathrm y}]]=\mathbb{E}_x[\mathrm y\log\overline{\mathrm y}]-\mathbb{E}_D[\mathbb{E}_x[\mathrm y\log\hat{\mathrm y}]]$ I am not sure how to use E(y) and E(\bar{y}) are all equals to 1 in the above equation. It seems that $\mathbb{E}_x[f(x)g(x)]\neq\mathbb{E}_x[f(x)]\mathbb{E}_x[g(x)]$

Many thanks!

lsongx commented 3 years ago

No problem! You are very welcome to ask anything~

No need to decompose the two terms. We first have $\mathbb{E}_D[\log\frac{\overline{\mathrm y}}{\hat{\mathrm y}}]=\log Z$ then $\int y\mathbb{E}_D[\log\frac{\overline{\mathrm y}}{\hat{\mathrm y}}] \mathrm{d}x=\int y\log Z \mathrm{d}x$ so $\mathbb{E}_D[\int y\log\frac{\overline{\mathrm y}}{\hat{\mathrm y}} \mathrm{d}x]=\int y\log Z \mathrm{d}x$ similar we have $\mathbb{E}_D[\int \bar{y}\log\frac{\overline{\mathrm y}}{\hat{\mathrm y}} \mathrm{d}x]=\int \bar{y}\log Z \mathrm{d}x$

By the way, could you further explain how to get the equation in your first comment? $\frac{\log \overline{\mathrm y}_{ce}}{\mathbb{E}_D[\log \hat{y}_{ce}]}=constant$ Seems like that our notations are somewhat misleading

HolmesShuan commented 3 years ago

Cool! The derivation is very clear!

It seems that $\mathbb{E}_D[\log\frac{\overline{\mathrm y}}{\hat{\mathrm y}}]]=-\log\mathrm Z?$ according to $\overline{\mathrm y}=\frac{1}{\mathrm Z}\exp(\mathbb{E}_D[\log \hat{\mathrm y}])$

The notation is ok but I found E[y]=1 and E[\overline y]=1 quite confusing, which makes me wonder is there any $\mathbb{E}[*]=\mathbb{E}[*]\cdot1=\mathbb{E}[*]\cdot\mathbb{E}[\mathrm y]=\mathbb{E}[*]\cdot\mathbb{E}[\overline{\mathrm y}]...$ tricks in Equation(2). :sweat_smile:

By the way, could you further explain how to get the equation in your first comment?

As for $\frac{\log \overline{\mathrm y}_{ce}}{\mathbb{E}_D[\log \hat{y}_{ce}]}=constant$ I just quoted the conclusion in the ICLR paper "The derivation of the variance term is based on the facts that $\frac{\log \overline{y}{ce}}{\mathbb{E}\mathcal{D}[\log \hat{y}_{ce}]}$ is a constant and ...".

Thanks for your responses!

lsongx commented 3 years ago

Glad that I have addressed your concerns.

The equation in the paper seems wrong. It should be $\mathbb{E}_D[\log\frac{\overline{\mathrm y}}{\hat{\mathrm y}}]=\log Z$

Thanks for your question!