carlesoctav
commented
11 months ago The best way to implement custom logic is to create a model abstraction that wraps the transformers model. This can be used with nn.modules or tf.keras.Model. I also believe that the author of the paper does the same.
Checking here in case other ppl might also be looking around without any success:
There are few papers out there that use modified version of SBERT, namely the component-focused or part-enhanced etcc. They are the same thing, basically just encoding parallelly also a version of the sentence made of only few "important" words. They then (weighted) sum this with the basic embedding and it all goes into the pooling layer. None of this paper shared a repo or a line of code. refs: https://www.mdpi.com/2076-3417/10/3/958 - https://www.sciencedirect.com/science/article/pii/S0957417422004900
If anyone has something please let me know. I supposed this would be built on sentence-transformer as it is still the core and dont feel like reinventing the wheel, but I am figuring out how to make this change in the architecture happen. Should I just build an extension of modules.Transformer and then put it in the list of modules and create a SentenceTransformer object? But then the tokenization is called outside so to the encoder forward things are already tokenized and I need to extract the important words and make a parallel forward pass. Any heads up are welcome. Thank you so much