Closed boscoj2008 closed 2 years ago
Hi, what you would need to do is to create a new layer class that extracts the sentence lengths from the BERT tokenizer / the BERT attention mask.
However, adding a LSTM on top of BERT does not make really sense. Transformers is a much more powerful model than bi-lstm. Adding one on top is just a waste of compute.
@nreimers Oh, thanks. In that case, could you advise me how I can solve my problem;
I have fine-tuned sentence-bert on SLNI and Multi-SNLI (ALL-LNI) data, load the trained model, encode my records to get vectors of 768 dimensions. I PCA the embeddings, cluster them and evaluate, however, the recall values are always below 45%. Precision is above 80% which is satisfactory. InferSent gives me superior (recall=70% or more, precision=85%) results but I was thinking to use bert's transformer architecture to give better embeddings.
Any advice will be appreciated. Thanks.
Sorry, cannot help there.
But PCA is usually not needed when you run clustering.
I have been trying to figure out how to build sentence embeddings by leveraging the bi-lstm layer. In the folder "average_word_embeddings" i use the example in
training_stsbenchmark_bilstm.py
and try to adapt it totrain_nli.py
by adding an LSTM layer, but get the error; File "../lib/python3.6/site-packages/sentence_transformers/models/LSTM.py", line 30, in forward sentence_lengths = torch.clamp(features['sentence_lengths'], min=1) KeyError: 'sentence_lengths'Can anybody help? I'm stuck :(