search

Linear95 / CLUB

Code for ICML2020 paper - CLUB: A Contrastive Log-ratio Upper Bound of Mutual Information
312 stars 39 forks source link

The symmetric problem about the CLUB MI estimator? #14

Open XiXiRuPan opened 2 years ago

XiXiRuPan commented 2 years ago

Thanks for your sharing. Your work is very interesting. The standard MI is symmetric but CLUB MI estimator is not. Have you tried building two variational estimators between x,y, which estimate x->y and y->x mi info? Thanks again.

Linear95 commented 2 years ago

Hi, sorry for replying late. Yes, our CLUB is not symmetric for x and y. But the asymmetry of CLUB is symmetric to x and y:) You can choose by yourself which variable to be x and which to be y. As you said, we can build two estimators x->y and y-x, to estimate MI(x;y). But here are some of my concerns:

  1. With the two variational estimators, we introduced more parameters to the model.
  2. The performance of the combination of x->y and y->x will be bounded by x->y or y->x (depends on which one is better). So why not only use the better one between x->y and y->x?

Empirically, I suggest using the variable with a lower dimension as y, because it is easier for the network q(y|x) to fit the lower dimension outputs. We also have a discussion about this problem (known as network expressiveness) in our paper Section 3.2. Thank you.