Accessing sentence level embeddings and alter the same and pass that altered embedding to the Siamese network again

[heresavy]

I need to access the sentence level embeddings for each sentence individually so that I can alter or modify the embedding with some further with specific strategy and pass that altered embedding back to the Siamese Network. While I tried to access the SentenceTransformer.py to debug I found that when data variable is formed (in line 544) from data_iterator the embeddings appear with compounded manner (which further gets broken to features and labels in line 551), rather for each sentence and I couldn't dig any further. Any hints and help would be deeply appreciated.

[nreimers]

Have a look here: https://www.sbert.net/docs/training/overview.html#creating-networks-from-scratch

A SentenceTransformer consists of multiple modules, which are called in a row. Usually you start with a word embedding model (like Transformer), then a pooling model, then you could have a dense model.

You could create your own module and add it to the list of modules

[heresavy]

Thanks @nreimers , I looked at the reference and your suggestion quite thoroughly; but it doesn't help me where I am stuck. I have the following situation: With SentenceTransformer the siamese network is trained to detect if pair of sentences are matched or not (for its intention or understanding wise). Using BERT we get the embeddings which is employed at sentence level and matched. E.g. There is a Sentence, where there are 7 tokens T1, T2, ... T7 in order where we have a sentence level embedding gets computed using BERT of 768 Dimension and say few word tokens say T2, T5 is quite specific there and don't get much help with BERT and somehow with a rare domain specific word we prepare a new embeddings for them and for this sentence we want to concatenate those embeddings to this BERT embeddings at training phase for better accuracy. In this case I am stuck how to access the sentence level embedding individually and merge / concatenate the specific embedding I prepare.

Correct me if I am wrong, but I found that document says how to prepare a complex model stacking word embedding model (e.g. BERT), pooling model and dense model further; but the document hasn't provided any clue how I can access the embedding and alter and repack to pass through the deep learning network for further.

I would appreciate if you kindly help me with this specific as I have provided. Thanks !

[nreimers]

For this I recommend to have a look at the source code of the different modules: https://github.com/UKPLab/sentence-transformers/tree/master/sentence_transformers/models

See which models fits best what you need and adapt it to your use case

Creating your own module is simple and can be added to the list of modules.