Regarding VAST dataset

[kgarg8]

Hi,

I am curious why you used lowercased processed text (with punctuations and stopwords removed) for VAST dataset. This could be a bit unnatural for transformer models.

https://github.com/zihaohe123/wiki-enhanced-stance-detection/blob/983decae7ae6165351d4168e05cdeca446c22a84/src/datasets.py#L194C22-L194C28

E.g., text_s column looks like:

absolutely 's needs defined regulated use currently word natural used food products totally confusing meaningless. clearly trying imply item healthy possibly organic see food manufacturers like frito lay campbell 's products labelled natural alone set alarms seems. ;-)

whereas the original tweet (Post column) was:

Absolutely it's needs to be defined and regulated in its use, as currently the word 'natural' when used on food products is totally confusing and meaningless. Clearly they are trying to imply the item is 'healthy' or possibly 'organic', but when you see food 'manufacturers' like Frito-Lay or Campbell's with products labelled 'natural', that alone should set off alarms that all is not what it seems. ;-)

[zihaohe123]

Hi, thanks for your interest in the paper. I can't remember exactly what I chose to use "text_s". You can try both the processed text and original text and compare which one leads to better performance (and share your findings with me if possible). Thanks!

[kgarg8]

Regarding the use of the two columns, here are my findings:

The results vary slightly but there is no clear winner it seems.

Another followup question, My reproduced results are little lower than yours. Do you have any suggestions? FYI, I tuned the learning rate little bit as you see above, but it doesn't help much.

Reporting the results with the default lr for VAST:

VAST | Zero-shot | Few-shot | Overall Original | 75.3 | 73.6 | 74.5 Reproduced | 70.53 | 74.03 | 72.36

Any comments?

[zihaohe123]

Thanks for sharing the results!

About your reproduced results on VAST, could you share with me your experimental setup?

[kgarg8]

Hey, sorry for not being to share my setup since I am working in a private repo and the work is unpublished still.

I did more hyperparameter tuning and got closer to the original results.

VAST | Zero-shot | Few-shot | Overall WS-BERT (original) | 75.3 | 73.6 | 74.5 WS-BERT (reproduced) | 72.16 | 75.02 | 73.62

One thing I observe in the experiments is that Few-Shot > Zero-Shot for this repository. I am curious if you could double-check whether you swapped zero-shot and few-shot by mistake.

Thanks.

[zihaohe123]

Thanks for sharing the results. I'm quite sure that I didn't swap the results of zero-shot and few-shot.