search

fhamborg / NewsMTSC

Target-dependent sentiment classification in news articles reporting on political events. Includes a high-quality data set of over 11k sentences and a state-of-the-art classification model.
Other
140 stars 21 forks source link

NewsTSC #5

Closed Pager07 closed 3 years ago

Pager07 commented 3 years ago

Hi Felix,

I am writing to you about the NEWSTSC repo which is now archived. I ran the train script to train LCF_BERT with the following settings 

python train.py  --model_name lcf_bert --optimizer adam --initializer xavier_uniform_ --learning_rate 3e-5 --batch_size 16 --balancing lossweighting --num_epoch 4 --lsr False --use_tp_placeholders False --eval_only_after_last_epoch True --devmode False --local_context_focus cdm --SRD 3 --pretrained_model_name bert_news_ccnc_10mio_3ep --snem recall_avg --dataset_name newstsc --experiment_path ./experiments/newstsc_20191126-115759/0/ --crossval 0 --task_format newstsc --device cuda:0

I found that the recall_avg score is ~0.65 after 4 epochs. (this is approximately what the paper for NEWTSC describes )

I then downloaded the fine-tuned weights "lcf_bert_newstsc_val_recall_avg_0.5954_epoch3" and ran test on the given test set for NEWSTSC. I am getting recall_avg scores ~0.825.

Were the provided fine-tuned weights generated from the 3k sentence dataset provided here?

fhamborg commented 3 years ago

Hi there, I strongly recommend to use NewsMTSC. As for the archived repository, we cannot provide support for the old tool but that shouldn't be an issue since most functionality, including what it seems you're trying to use, is included in the new repository's tool, NewsMTSC (in a cleaner way and using more up-to-date Python libraries) :-)