Do you support learning to distill knowledge from larger reranker models to smaller reranker models?

FlagOpen / FlagEmbedding

Retrieval and Retrieval-augmented LLMs

MIT License

6.81k stars 491 forks source link

Do you support learning to distill knowledge from larger reranker models to smaller reranker models? #1021

Open QuangTQV opened 1 month ago

545999961 commented 1 month ago

We currently do not support distillation for reranker; however, you can label the data and modify the fine-tuning code to include a KL loss, thereby implementing this function.