search

princeton-nlp / CoFiPruning

[ACL 2022] Structured Pruning Learns Compact and Accurate Models https://arxiv.org/abs/2204.00408
MIT License
188 stars 32 forks source link

Discrepancy between my evaluation results and README for MNLI in evaluation.py #40

Open TinaChen95 opened 1 year ago

TinaChen95 commented 1 year ago

Hi, I'm running evaluation.py on MNLI as described in the README, but I'm getting different results compared to what's displayed there. I'm using Google Colab for this, and you can find my notebook here: https://colab.research.google.com/drive/1UahAOTIwALfEC_DXE11mVOp5iSgwHoYH?usp=sharing

When I run evaluation.py, it shows the following results: Task: mnli Model path: ../CoFi-MNLI-s95 Model size: 4330279 Sparsity: 0.949 Accuracy: 0.091 Seconds/example: 0.000561

However, in the README file, the results for the same evaluation are different: Task: MNLI Model path: princeton-nlp/CoFi-MNLI-s95 Model size: 4920106 Sparsity: 0.943 mnli/acc: 0.8055 Seconds/example: 0.010151

I need help figuring out why there's a discrepancy between my results and what's described in the README. I've tried to follow the instructions in the README as closely as possible, but I may have missed something. Thank you for any assistance you can provide.

xiamengzhou commented 1 year ago

this is interesting, what is the model path ../CoFi-MNLI-s95?

gaishun commented 1 year ago

Have you solved this problem? I encountered the same problem.

xiamengzhou commented 1 year ago

It seems to be an issue with transformers' versions. It should be compatible with transformers==4.17.0 and datasets==1.14.0 but might not work with versions beyond.

SHUSHENGQIGUI commented 2 months ago

It seems to be an issue with transformers' versions. It should be compatible with transformers==4.17.0 and datasets==1.14.0 but might not work with versions beyond. but that setting is always conficted with the huggingface version. can i check out your python env? I am crazy about this issue. please