search

oliverguhr / german-sentiment-lib

An easy to use python package for deep learning-based german sentiment classification.
https://pypi.org/project/germansentiment/
MIT License
58 stars 7 forks source link

Er ist eine verdammmte Ziege! = neutral? #4

Closed agoeroeg closed 3 years ago

agoeroeg commented 3 years ago

Hi Oliver,

I just tested "Er ist eine verdammmte Ziege!" (He is a f... goat!) and I got "neutral" back. I was also wandering, why in the live demo on the huggingface site, I get a dict back (with 99% negative) and with python only a string. Any info would be great.

Best wishes Attila

(Windows 10, Conda pytorch 1.7, pip germansentiment).

agoeroeg commented 3 years ago

It could be the cleaning, because the core works, yields: "negative":

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

tokenizer = AutoTokenizer.from_pretrained("oliverguhr/german-sentiment-bert")
model = AutoModelForSequenceClassification.from_pretrained("oliverguhr/german-sentiment-bert")

texts = ['Er ist eien verdammte Ziege!']
input_ids = tokenizer(texts, padding=True, truncation=True, add_special_tokens=True)
input_ids = torch.tensor(input_ids["input_ids"])

with torch.no_grad():
    logits = model(input_ids)    

label_ids = torch.argmax(logits[0], axis=1)
labels = [model.config.id2label[label_id] for label_id in label_ids.tolist()]
oliverguhr commented 3 years ago

We trained the model on opinions rather then insults. If you need a model that can detect insults you need to retrain this model or train a new one. Since our source data also contained insults, some of them will work - but it's not specialized for this purpose.

I wanted to create a high level, easy to use lib for people with no ML background. This is why I am decided to not output the probabilities since they are somewhat misleading. Most of the time the model will output high probability values (98%+) even if the model is wrong. Instead, we tried to balance the data, so that the model will label uncertain samples as neutral.

If skipping the preprocessing works for you, this is great - but be aware that we trained and tested the model with the preprocessing steps. Omitting them can cause other issues. I suggest that you create a test dataset that contains examples specific for your task - and check which setting will yield the best results. If you do so, it would be great if you share your data with us.

oliverguhr commented 3 years ago

I close this issue for now. Feel free to reopen it in case you have any questions or issues.