Learn rate and length scale for MLKR

[Ribosome25]

Hi Dear Maintainers,

I found that MLKR does not support custom learning rate and length scale yet. Can I add these features to it? Thanks.

[wdevazelhes]

Hi @Ribosome25, thanks for your message,

• Regarding the learning rate: Our current implementation of MLKR uses lbfgs-b as an optimizer, where there is no learning rate (this is an advantage: it removes an additional hyperparameter, and lbfgs-b has a lot of advantages which why it is often used as a method of choice). See for instance this answer: https://stackoverflow.com/questions/60902315/mlpregressor-learning-rate-init-for-lbfgs-solver-in-sklearn, on a similar question about the MLP in sklearn that uses L-BFGS.
• Regarding the scaling: In theory, if such scaling is needed for achieving a good performance, the learned mahalanobis matrix should learn it already (to say it otherwise, pairwise_dist(x, x') /sigma = sqrt((x-x').T.dot(M).(x - x'))/sigma = sqrt((x-x').T.dot(M/sigma^2).(x - x'))), therefore, is there any particular reason to add it ?

[Ribosome25]

Thank you for the explanation. That is very helpful.

Thanks.

On Tue, Mar 2, 2021, 5:11 AM William de Vazelhes notifications@github.com wrote:

Hi @Ribosome25 https://github.com/Ribosome25, thanks for your message,

• Regarding the learning rate: Our current implementation of MLKR uses lbfgs-b as an optimizer, where there is no learning rate (this is an advantage: it removes an additional hyperparameter, and lbfgs-b has a lot of advantages which why it is often used as a method of choice). See for instance this answer: https://stackoverflow.com/questions/60902315/mlpregressor-learning-rate-init-for-lbfgs-solver-in-sklearn, on a similar question about the MLP in sklearn that uses L-BFGS.
• Regarding the scaling: In theory, if such scaling is needed for achieving a good performance, the learned mahalanobis matrix should learn it already (to say it otherwise, pairwise_dist(x, x') /sigma = sqrt((x-x').T.dot(M).(x - x'))/sigma = sqrt((x-x').T.dot(M/sigma^2).(x - x'))), therefore, is there any particular reason to add it ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/scikit-learn-contrib/metric-learn/pull/308#issuecomment-788828866, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKRLAAESVPLBWEGZNPKS6VDTBTBUVANCNFSM4YNL6SLA .

[wdevazelhes]

@Ribosome25 I'm glad it was helpful,

I'll close the PR, feel free to open another one it if you think of some other contribution !

Cheers,