sentiment predictions are not consistent

yya518 / FinBERT

A Pretrained BERT Model for Financial Communications. https://arxiv.org/abs/2006.08097

Apache License 2.0

560 stars 128 forks source link

sentiment predictions are not consistent #25

Open fanyuyu opened 2 years ago

fanyuyu commented 2 years ago

I am using your sentiment model to predict sentences from calls. There are two sentences:

The probability of neutral is .99 for the sentence 'Thanks, Martin.'
The probability of positive is .94 for the sentence 'Thank you.'

I am trying to understand why it gives quite different labels. Initially I thought it was the label confusion, but you have answered the question in Issue #17. Could you explain more about how you fine-tuning the model for analyst tones and what date you use for the classification model? Thank you!

yya518 commented 2 years ago

I cannot re-produce the issue that you described. I got Neutral for both sentences.

finbert = BertForSequenceClassification.from_pretrained('yiyanghkust/finbert-tone',num_labels=3)
tokenizer = BertTokenizer.from_pretrained('yiyanghkust/finbert-tone')
nlp = pipeline("text-classification", model=finbert, tokenizer=tokenizer)
results = nlp(['Thank you.', 
               'Thanks, Martin.'])
print(results)
#[{'label': 'Neutral', 'score': 0.8304300308227539},  {'label': 'Neutral', 'score': 0.9677242040634155}]