search

google-research / disentanglement_lib

disentanglement_lib is an open-source library for research on learning disentangled representations.
Apache License 2.0
1.38k stars 205 forks source link

Strange score distribution in the paper. #33

Open erow opened 3 years ago

erow commented 3 years ago

image This is Figure 7 MIG distribution on dSprites. The MIG score should be non-negative, and the scores are low. I analyze the pre-trained models and get the following results. image

I don't know why they are so different, and the other metrics are different too.

Near32 commented 3 years ago

Hello, The MIG computation function (and the Modularity one too) makes use of scikit-learn's mutual_info_score function (see here). I suspect scikit-learn's mutual_info_score function having changed throughout the different versions is the issue here, and this is likely to be the root cause of discrepancies between the results you computed in February 2021 (I assume you were using scikit-learn version 0.24.0) and the original results of the paper (published on ArXiv around Nov. 2018, when the latest scikit-learn version was 0.20.0).

I am experiencing a lot of discrepancies with the Modularity score on my end (with a code adapted to another framework though).