search

marcotcr / checklist

Beyond Accuracy: Behavioral Testing of NLP models with CheckList
MIT License
2.01k stars 204 forks source link

Irony & Sarcasm as new capability #55

Closed MartaMarchiori closed 3 years ago

MartaMarchiori commented 3 years ago

Hi there, :) Just wondering if you have any suggestions, for the context of Sentiment Analysis, how to include-create a new capability concerning irony and sarcasm. The fact is that given the three types of tests, the most suitable to start would be MFT, but what about the labels? If "irony datasets" contain tweets only marked as ironic or not, they could be negative or positive with respect to the Sentiment, so what I thought is to use the Expect function is_not_1, expecting the label NOT to be neutral... But I am not convinced by this solution

marcotcr commented 3 years ago

I would suggest a bottom-up approach, where you examine a bunch of random examples of irony and sarcasm and try to generalize from them in order to create a few templates for an MFT.

I don't see how you could extract sentiment labels from a dataset only labeled for irony, since there are ironical tweets that are still neutral, e.g. this.

In terms of perturbation tests, if you can find some perturbation that indicates sarcasm, you could write a DIR that flips the label whenever it is applied.