Recherche für passendes BERT Model

[Shad00Z]

z.B. BERT Tiny

[maxagamer]

NBoost for Search(https://github.com/koursaros-ai/nboost)

• NBoost is a scalable, search-api-boosting platform for deploying transformer models to improve the relevance of search results or diffe rent platforms (i.e. Elasticsearch) Topics
• maybe more useful for the user pipeline
• is between user and the actual pipeline where it picks the best based on relevance and training

JMC and DISP (https://www.sciencedirect.com/science/article/pii/S0925231221008390)

• Joint Model Compression (JMC)
• Direct Importance-aware Structured Pruning
• Dense Knowledge Distillation (DKD)
• Paper at thulb

MiniLMv2 (https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2)

• found in https://arxiv.org/abs/2012.15828 (MiniLMv2)
• they state that their model outperform state of the art models
• uses deep distillated self-attention which are distributed and also give values to relations to copy teacher self-attention
• transfers knowledge of upper-middle- to last-layer(MiniLM only mimics the last layer)

Transit(https://arxiv.org/abs/2101.03289)