search

burklight / nonlinear-IB-PyTorch

Pytorch Implementation of the Nonlinear Information Bottleneck
37 stars 8 forks source link

Math Formula Error #2

Open Mutual-Luo opened 8 months ago

Mutual-Luo commented 8 months ago

Thank you for your code! But I there exists errors in the derivation of mathematical formulas [Eqs.(10) in https://arxiv.org/pdf/1705.02436.pdf]

Authors cite paper "Estimating mixture entropy with pairwise distances". The authors of this paper has corrected their error in new version.

But "nonlinear information bottleneck" still use the still using the wrong math conclusions. [I don't think the last two items can be omitted] image Is there some principle that I don't understand or it's a mistake?

artemyk commented 8 months ago

Thank you for your careful reading. Eq (18) only applies in the more general case where the distribution over the input random variable U is itself a mixture that may loose some information about the component C (so I(X;C)>I(X;U)). If U is treated as a set of delta functions, as done in our approach, then I(X;C)=I(X;U) and that extra term cancels.

You can also derive our expression (10) from Eq (13) in the updated "Estimating mixture entropy with pairwise distances"