search

paulbrodersen / entropy_estimators

Estimators for the entropy and other information theoretic quantities of continuous distributions
GNU General Public License v3.0
130 stars 26 forks source link

getting negative values for mutual information #17

Closed keanepotato closed 11 months ago

keanepotato commented 1 year ago

Hi Paul,

Great work on the program. I'm utilising it primarily for getting entropy values for continous values. I realise, while utilising the get_mi function, however, I'm getting negative values. Based on what i've read, I believe this is not a possible value for MI? (i.e. the lower bound should be 0). Do I have my understanding of your program correct?

paulbrodersen commented 1 year ago

I believe this is not a possible value for MI? (i.e. the lower bound should be 0)

Correct.

However, get_mi implements the KSG estimator, which is not an unbiased estimator. The bias depends on your choice of k, the number of samples, and the distributions of your random variables. One approach to compute (and then correct for) this bias is to permute your random samples (thus removing any relationship there might be between them), compute the KSG estimate for the randomized samples that are known to have zero mutual information, and then subtract the average from the KSG estimate for your samples before randomization.