search

UKPLab / sentence-transformers

State-of-the-Art Text Embeddings
https://www.sbert.net
Apache License 2.0
15.3k stars 2.48k forks source link

AugSBERT with MultipleNegativesRankingLoss? #1100

Open brianhance opened 3 years ago

brianhance commented 3 years ago

Hi again - is there a loss function in which you can train with multiple similar sentences for one query sentence? For example, the SBERT paper cites BERT, senteval, skip-though vectors, etc. Could all positive pairs (a1, b1), (a1,b2), (a1,b3) be used in training, without them being counted as negatives against each other?

Also, is it possible to implement AugSBERT with a MultipleNegativesRankingLoss or Triplet loss? I imagine that I would train a cross-encoder with all available labeled triplets, and then create new triplets using BERT, then combine the gold and silver data as usual for bi-encoder training. I am wondering however, is it possible to use semantic search/BM25 and BERT to create a silver dataset of triplets?

Thanks!

nreimers commented 3 years ago

Question 1: With MultipleNegativesRankingLoss this is not possible. Other loss function do not necessarily use other examples as negatives

Question 2: Yes, that is possible.

brianhance commented 3 years ago

Thanks - could you elaborate on the other loss functions that would be fit for this task? The BatchTripletLoss functions look like the best fit to me, but I'd love to hear your input. Thanks so much!

nreimers commented 3 years ago

Yes, the batch hard triplets losses are suitable for that