Closed tma15 closed 1 year ago
Results by using the model fine-tune by me
[Infer][37] Accuracy: 0.9405844155844156 original acc: 0.9402597402597402 (3062) confident acc: 0.9448073154800783 (18) unconfident acc: 0.16666666666666666 new unconfident acc: 0.2222222222222222 final acc: 0.9405844155844156 TIME-9.28
Results by using the fine-tuned model:
[Infer][0] Accuracy: 0.9493506493506494 original acc: 0.9493506493506494 (3040) confident acc: 0.9549342105263158 (40) unconfident acc: 0.525 new unconfident acc: 0.4 final acc: 0.9477272727272728 TIME-9.40
please @HwwAncient follow it up
Hi, you can refer to the heperparameters in the two files: https://www.modelscope.cn/models/damo/nlp_space_pretrained-dialog-model/summary https://www.modelscope.cn/models/damo/nlp_space_pretrained-dialog-model/file/view/master/intent_train_config.json
Thanks for the great work. I'm interested in intent prediction tasks using SPACE-3 and want to reproduce the reported scores of BANKING77, HWU64, CLINC150.
I had confirmed that the fine-tuned model, which is distributed on this link, obtained reported accuracy on BANKING77. However, when I fine-tuned pre-trained models by myself, the accuracy did not match with the reported ones.
The following is
scripts/banking/train.sh
on my environment. I had only changedPROJECT_ROOT
andSAVE_ROOT
from the original script.Do you have any ideas for reproducing reported scores?