Adding other features (categorical, numeric) to the SBERT architecture

[Nicolabo]

I am researching the example of using SBERT for Information Retrieval in Quora Duplicates Questions dataset. I am wondering how to proceed to apply other features to the SBERT architecture such as categorical, numeric or even other text features. Do you have any idea?

Using Quora example, I understand that one option is to just convert any feature into text and then concatenate with the question text data. However I worry that that such signals (i..e data points) might disappear, especially if we have very long questions.

[nreimers]

Yes, that is the most simple solution. Everything would require a change of the transformers architecture. Don't have to worry that it will disappear. Due to the full attention mechanism of the transformers architectures, all features are available for every token in every layer

[danielxu05]

I think one option could be to concatenate the categorical, numeric features with the pooling layer, so the pooling layer would contain both features from the language and also the structured feature. I guess it needs some work to deal with the architecture