search

zihaohe123 / wiki-enhanced-stance-detection

21 stars 4 forks source link

There was a problem with reproduction results. the best f0 score was 0.74 #1

Open S1mplePike opened 2 years ago

S1mplePike commented 2 years ago

Hi, author. Recently I am follow this work. but，I got some problem. I cann't reproduction the results of VAST dataset f1 score=0.736 and f0 score=0.753, I wonder if the problem is caused by the version. my transformers package version == 4.18.0. I' d like to ask about the python packages version of experimental environment. and the hyperparameters of experiment.

zihaohe123 commented 2 years ago

Hi Pike,

Thanks for your interest! I just found that the number of epochs to train should be 50. In the script I made that 5 instead. Now I corrected it. All other hyperparameters are the same as what I used in my experiment. I don't think the version of transformers makes any difference.

One reason I can think of is the number of GPUs to use. Were you using 4 GPUs as I did? Single-GPU training results should be different from multi-GPU training.