search

huawei-noah / Pretrained-Language-Model

Pretrained language model and its related optimization techniques developed by Huawei Noah's Ark Lab.
3.02k stars 628 forks source link

tinybert task pred_distil dev data loader is overwritten by mnli-mm #115

Open TobiasLee opened 3 years ago

TobiasLee commented 3 years ago

According to your code in task_distill.py, the evaluation on MNLI-mm dataset will overwrite the original dev loader of MNLI-m, thus the best checkpoints are selected based on the MNLI-mm instead of the MNLI-m.

itsucks commented 3 years ago

Yeah, this is a bug

iMountTai commented 1 year ago

@TobiasLee Have you ever encountered a situation where mnli task acc remains unchanged？

programmingLearner commented 11 months ago

@TobiasLee Have you ever encountered a situation where mnli task acc remains unchanged？

They may not encounter this issue if they use older versions of transformers instead of dataset lib to fine tune the Teacher. I’ve solved this here: https://github.com/huawei-noah/Pretrained-Language-Model/issues/229