Open castafra opened 1 year ago
Hi @castafra thanks for your interest in our work! I know @orenpereg ran some experiments on 2-sentence tasks like GLUE, but it wasn't entirely clear how one should create the text/label triples one needs for the contrastive learning step.
Perhaps he can share some more details on what worked / didn't work :)
Hi @orenpereg , could you share any feedbacks on your experiments ? Thanks
Hi @castafra, Thanks for your question. Currently Setfit does not support two sentences similarity classification tasks and it's not trivial to apply SetFit for that. It will probably require changes within the Sentence Transformer itself. We do have plans to extend SetFit and one of the options is to examine similarity classification task. Having said that, i did play a bit with it, as a workaround, I simply concatenated sentence1 and sentence2 and input pairs of 2 concatenated sentences to the contrastive training process. The results weren't so good. SetFit was on-par with the baseline model (a 'standard' BERT cross encoder) when using few-shot data (8,16,32 samples). The bottom line is that this workaround is not recommended.
I also work on the similarity classification task. I think it would be great to extend the framework for that task.
It seems setFit
uses the underlying sentence_transformer
library, right? FWIW, the example shows how to use sentence_transformer
for similarity learning - https://www.sbert.net/examples/training/sts/README.html . I'm wondering whether we just need to implement a sklearn-learn
classifier model to take two fine-tuned embeddings as inputs to predict the label.
Other references:
Hi @castafra, Thanks for your question. Currently Setfit does not support two sentences similarity classification tasks and it's not trivial to apply SetFit for that. It will probably require changes within the Sentence Transformer itself. We do have plans to extend SetFit and one of the options is to examine similarity classification task. Having said that, i did play a bit with it, as a workaround, I simply concatenated sentence1 and sentence2 and input pairs of 2 concatenated sentences to the contrastive training process. The results weren't so good. SetFit was on-par with the baseline model (a 'standard' BERT cross encoder) when using few-shot data (8,16,32 samples). The bottom line is that this workaround is not recommended.
This is a bummer :( I was so excited that I could few shot improve postal address comparisons... :D
Hello, I would like to test this promising framework on a similarity classification task. So basically, I have got a dataset with 3 columns: (sentence1,sentence2,label). From what I understand, currently it is only possible to train on a single sentence classification problem. Is there a get around to use Setfit for a pair sentence classification problem ? If not, would it be possible to add this feature in a future integration ?
Thank you in advance