Questions About Ranking Strategy

[Jiahao004]

Hi Yeon, may I ask the questions about the ranking strategy? From my understanding, the direct rank for contrastive learning actually has little help for the SimCSE model, however, it largely enhances the performance when we conduct the distillation. I am very glad if you could correct me if anything I misunderstood.

Does this mean that the ranking strategy is actually good for distilling information from teachers, what kind of information is unsupervised distilled?

[yeonsw]

Hi,

The performance gain is largely from Eq 2. Table 2 shows the performance improvement made by Eq 2. Also, we retrained our sentence encoder with the rank vectors as rank vectors show better performance than its base encoder.

About the second question (distilling information), We think the performance gain is mostly from leveraging a relation between an input sentence and sentences in a large corpus (Wiki), the additional feature that is captured by rank vectors.

Thanks!