The spearmanr distance currently returns the actual Spearman rank-correlation coefficient, which behaves like a similarity measure rather than a dissimilarity measure. I believe this is not what NNDescent expects. Rather than using the numpy correlation matrix function, we can use the correlation distance after computing the rank transformation. Making this change also resolves a problem where spearmanr would return nan when both input vectors were identical.
lmcinnes
commented
2 years ago
The
spearmanrdistance currently returns the actual Spearman rank-correlation coefficient, which behaves like a similarity measure rather than a dissimilarity measure. I believe this is not whatNNDescentexpects. Rather than using the numpy correlation matrix function, we can use thecorrelationdistance after computing the rank transformation. Making this change also resolves a problem wherespearmanrwould returnnanwhen both input vectors were identical.