Open kgarg8 opened 11 months ago
Hi, thanks for your interest in the paper. I can't remember exactly what I chose to use "text_s". You can try both the processed text and original text and compare which one leads to better performance (and share your findings with me if possible). Thanks!
Regarding the use of the two columns, here are my findings:
The results vary slightly but there is no clear winner it seems.
Another followup question, My reproduced results are little lower than yours. Do you have any suggestions? FYI, I tuned the learning rate little bit as you see above, but it doesn't help much.
Reporting the results with the default lr for VAST:
VAST | Zero-shot | Few-shot | Overall Original | 75.3 | 73.6 | 74.5 Reproduced | 70.53 | 74.03 | 72.36
Any comments?
Thanks for sharing the results!
About your reproduced results on VAST, could you share with me your experimental setup?
Hey, sorry for not being to share my setup since I am working in a private repo and the work is unpublished still.
I did more hyperparameter tuning and got closer to the original results.
VAST | Zero-shot | Few-shot | Overall WS-BERT (original) | 75.3 | 73.6 | 74.5 WS-BERT (reproduced) | 72.16 | 75.02 | 73.62
One thing I observe in the experiments is that Few-Shot > Zero-Shot for this repository. I am curious if you could double-check whether you swapped zero-shot and few-shot by mistake.
Thanks.
Thanks for sharing the results. I'm quite sure that I didn't swap the results of zero-shot and few-shot.
Hi,
I am curious why you used lowercased processed text (with punctuations and stopwords removed) for VAST dataset. This could be a bit unnatural for transformer models.
https://github.com/zihaohe123/wiki-enhanced-stance-detection/blob/983decae7ae6165351d4168e05cdeca446c22a84/src/datasets.py#L194C22-L194C28
E.g.,
text_s
column looks like:whereas the original tweet (
Post
column) was: