question about baseline in table 1

timoschick / dino

This repository contains the code for "Generating Datasets with Pretrained Language Models".

https://arxiv.org/abs/2104.07540

Apache License 2.0

187 stars 24 forks source link

question about baseline in table 1 #11

Closed flipz357 closed 3 years ago

flipz357 commented 3 years ago

Hi!

Impressive results, many thanks for the code! I have a Q about Table 1, where it says "supervised SBERT baseline", but in the rest of the paper it only mentions SBERT trained on NLI and not tuning on STS train. I guess supervised means also it's fine-tuned on STS training, or do I err? Thanks, Juri

timoschick commented 3 years ago

Hi Juri, supervised training is performed using only NLI data. This is mentioned in the caption of Table 1 (highlight mine):

Table 1: Spearman’s rank correlation on STS12–16, STSb and SICK without finetuning on task-specific examples for models with NLI supervision (“sup.”) and fully unsupervised (“unsup.”) models [...]

flipz357 commented 3 years ago

Thanks! I also saw now that some other papers same terminology. I'd rather call that zero-shot or something, but I understand now.