I would like to use Bert-base and Bert-large versions cross-encoder trained on ms-marco. I tried to fine-tune "cross-encoder/ms-marco-MiniLM-L-12-v2" on NQ and other standard datasets. But the fine-tuned performance drops. I suspect it's because of the actual objective of MiniLM is the distillation regression loss thus a cross-entropy based fine-tuning doesn't help with improved performance. Therefore, I am planning on fine-tuning the source bert-base/ bert-large/ albert-large that was initially used to distill MiniLM. Do you have those checkpoints ? And if so would you mind sharing them ? Thanks in advance!
Hi @nreimers
I would like to use Bert-base and Bert-large versions cross-encoder trained on ms-marco. I tried to fine-tune
"cross-encoder/ms-marco-MiniLM-L-12-v2"
on NQ and other standard datasets. But the fine-tuned performance drops. I suspect it's because of the actual objective of MiniLM is the distillation regression loss thus a cross-entropy based fine-tuning doesn't help with improved performance. Therefore, I am planning on fine-tuning the sourcebert-base/ bert-large/ albert-large
that was initially used to distill MiniLM. Do you have those checkpoints ? And if so would you mind sharing them ? Thanks in advance!