Recovering linear transformation matrix from saved Mahalanobis matrix (precision errors?)

sunshineinsandiego commented 4 years ago

Description

LMNN: Error recovering the linear transformation matrix (L.T) from the Mahalanobis matrix. I am trying to recover the linear transformation matrix (L.T) from a saved Mahalanobis matrix produced by the LMNN algorithm, and there seem to be quite a few differences between the manually transformed data (using X.dot(L.T)) and LMNN.transform(X). Is this a rounding or precision issue?

Steps/Code to Reproduce

Example:

import numpy as np
from metric_learn import LMNN
from sklearn.datasets import load_iris
iris_data = load_iris()
X = iris_data['data']
Y = iris_data['target']

lmnn = LMNN(k=5, learn_rate=1e-6)
X_transformed = lmnn.fit_transform(X, Y)
M_matrix = lmnn.get_mahalanobis_matrix()
array([[ 2.47937397,  0.36313715, -0.41243858, -0.78715282],
       [ 0.36313715,  1.69818843, -0.90042673, -0.0740197 ],
       [-0.41243858, -0.90042673,  2.37024271,  2.18292864],
       [-0.78715282, -0.0740197 ,  2.18292864,  2.9531315 ]])

# cholesky decomp of M_matrix
eigvalues, eigcolvectors = np.linalg.eig(M_matrix)
eigvalues_diag = np.diag(eigvalues)
eigvalues_diag_sqrt = np.sqrt(eigvalues_diag)
L = eigcolvectors.dot(eigvalues_diag_sqrt.dot(np.linalg.inv(eigcolvectors)))
L_transpose = np.transpose(L)
L_transpose.dot(L) # check to confirm that matches M_matrix
array([[ 2.47937397,  0.36313715, -0.41243858, -0.78715282],
       [ 0.36313715,  1.69818843, -0.90042673, -0.0740197 ],
       [-0.41243858, -0.90042673,  2.37024271,  2.18292864],
       [-0.78715282, -0.0740197 ,  2.18292864,  2.9531315 ]])

# test fit_transform() vs. transform() using LMNN functions
lmnn.transform(X[0:4, :])
array([[8.2487    , 4.41337015, 0.14988465, 0.52629361],
       [7.87314906, 3.77220291, 0.36015873, 0.525688  ],
       [7.59410008, 4.03369392, 0.17339877, 0.51350962],
       [7.41676205, 3.82012155, 0.47312948, 0.68515535]])

X_transformed[0:4, :]
array([[8.2487    , 4.41337015, 0.14988465, 0.52629361],
       [7.87314906, 3.77220291, 0.36015873, 0.525688  ],
       [7.59410008, 4.03369392, 0.17339877, 0.51350962],
       [7.41676205, 3.82012155, 0.47312948, 0.68515535]])

# test manual transform of X[0:4, :]
X[0:4, :].dot(L_transpose)
array([[8.22608756, 4.45271327, 0.24690081, 0.51206068],
       [7.85071271, 3.81054846, 0.45442718, 0.51144826],
       [7.57310259, 4.06981377, 0.26240745, 0.50067674],
       [7.39356544, 3.85511015, 0.55776916, 0.67615584]])

Expected Results

The manual transformation of X[0:4, :] should be equal to the lmnn.transform() = first 4 rows of lmnn.fit_transform(X,Y).

Actual Results

The results should be equal, but they are not. Is this a rounding / precision error?

Versions

Darwin-17.7.0-x86_64-i386-64bit Python 3.7.7 (default, Mar 10 2020, 15:43:27) [Clang 10.0.0 (clang-1000.11.45.5)] NumPy 1.18.2 SciPy 1.4.1 Scikit-Learn 0.22.2.post1 Metric-Learn 0.5.0

Thank you!

perimosocordiae commented 4 years ago

This is likely due to rounding errors. LMNN learns the L matrix directly, so get_mahalanobis_matrix is computing M = L.T @ L. Your example then decomposes M to recover L, which I would bet is the source of the floating point precision loss.

sunshineinsandiego commented 4 years ago

Thanks, is there any way to recover L directly without going the roundabout way of pulling M from the LMNN module and then recomputing L?

wdevazelhes commented 4 years ago

Hi @sunshineinsandiego, yes, you can get the learned transformation L with the attribute lmnn.components_

sunshineinsandiego commented 4 years ago

Perfect! Thank you

scikit-learn-contrib / metric-learn