search

YujiaBao / Distributional-Signatures

"Few-shot Text Classification with Distributional Signatures" ICLR 2020
https://arxiv.org/abs/1908.06039
MIT License
253 stars 57 forks source link

About learning lambda and alpha #14

Closed WeiyuCheng closed 4 years ago

WeiyuCheng commented 4 years ago

Hi Yujia, why did you learn lambda and alpha in the log space in r2d2.py? In the implementation of the paper "META-LEARNING WITH DIFFERENTIABLE CLOSED-FORM SOLVERS", the authors seem to just learn lambda and alpha in normal space. How does this affect your training procedure and final results?

YujiaBao commented 4 years ago

Hi Weiyu,

I think in the implementation (https://github.com/bertinetto/r2d2/blob/master/fewshots/models/adjust.py), the author also learn them in log space (when base != 1). Since they turned off the lambda & alpha learning (base_learner.learn_lambda: False in the fewshots.yaml), it is hard for me to tell whether it is actually used or not.

In our experiments, when we update lambda and alpha in the normal space, sometimes they become negative after the update. Learning them in the log space resolve this issue. If you prefer to learn them in the normal space, you can try to force them to be positive after the update.