search

zihaohe123 / wiki-enhanced-stance-detection

21 stars 4 forks source link

Problem reproducing results on Covid-19 dataset #3

Closed kgarg8 closed 1 year ago

kgarg8 commented 1 year ago

Hi,

I used the same parameters as this repo but I am having trouble reproducing results for Covid19 dataset, except the following changes:

  1. I used 1 GPU instead of 4 GPU training
  2. Reproduced numbers are average over 3 seeds. The authors don't mention the number of runs in the paper.

Covid19 | Fauci | Home | Mask | School | Avg. Original | 83.60 | 85.00 | 86.60 | 82.20 | 84.40 Reproduced | 78.78 | 84.07 | 84.26 | 80.75 | 81.96

BTW, I could pretty-much reproduce the results for PStance dataset without any issues.

zihaohe123 commented 1 year ago

Hi, thank you for your interest!

I think I used 2 V100 GPUs for training the COVID-19 dataset as the covid-twitter-bert model was large. If you wanna do single-gpu training, you may also wanna play with some of the hyperparameters, such as learning rate and batch size.

Thank you!