How to use two Huggingface/transformers models to produce joint contextualized word embeddings and add a pooling layer when creating a sentence-transformers

UKPLab / sentence-transformers

State-of-the-Art Text Embeddings

https://www.sbert.net

Apache License 2.0

15.21k stars 2.47k forks source link

How to use two Huggingface/transformers models to produce joint contextualized word embeddings and add a pooling layer when creating a sentence-transformers #811

Open GJHHHH opened 3 years ago

GJHHHH commented 3 years ago

I don't seem to find the relevant function in https://www.sbert.net/docs/training/overview.html#.Do I need to modify the code of the encode function in the SentenceTransformer.py file by myself?

nreimers commented 3 years ago

Have a look here at the Asym class: https://github.com/UKPLab/sentence-transformers/releases/tag/v0.4.1

If you use the version from Git (or wait until v 1.0.0 is released in the next days), then you can use the Asym class as your first module in SentenceTransformer with multiple different transformers models, e.g. one for query, one for document.