KeyError: 'sentence_embedding' when using Lonformer Model

[blablablubb123]

Hello, I'm trying to use the Longformer Model but I get the following error: KeyError: 'sentence_embedding'

My Implementation is like the following: embedding_model = models.Transformer(model_name, model_args={'gradient_checkpointing': True}) model = SentenceTransformer(modules=[embedding_model])

[nreimers]

Hi, you need to add a pooling layer to your network:

embedding_model = models.Transformer(model_name, model_args={'gradient_checkpointing': True})
pooling = models.Pooling(word_embedding_model.get_word_embedding_dimension())
model = SentenceTransformer(modules=[embedding_model, pooling_model])

[blablablubb123]

Thank you very much!

[thesby]

Hi, you need to add a pooling layer to your network:

embedding_model = models.Transformer(model_name, model_args={'gradient_checkpointing': True})
pooling = models.Pooling(word_embedding_model.get_word_embedding_dimension())
model = SentenceTransformer(modules=[embedding_model, pooling_model])

I use the model "allenai/longformer-base-4096", but every doc embedding is very similar. Have you met this problem?

[nreimers]

Have you fine-tuned the model on some data?

Out of the box, transformer model produce rather bad sentence embeddings

[thesby]

No, I didn't fine-tune it. Actually, I want to use model xlm-r-100langs-bert-base-nli-stsb-mean-tokens trained by sentence_transformers to build a longformer. But the model format seems very different