search

DeveloperLiberationFront / AffectAnalysisToolEvaluation

SEmotion_18 paper on evaluating the reliability of sentiment and politeness analysis tools
3 stars 4 forks source link

Novel contribution regarding sentiment analysis tool evaluation #17

Closed nasifimtiazohi closed 6 years ago

nasifimtiazohi commented 6 years ago

R1

1 - The unreliability of sentiment tool in SE domain has already been studied in the papers cited and also by Lin et. al. in "Sentiment Analysis for Software Engineering for example:

How Far Can We Go?", with respect to other studies about sentiment analysis tools they only add a new tool Senti4SD.

where author manually rated 40k comments. Hence, regarding sentiment analysis tools the paper does not add relevant contribution.

nasifimtiazohi commented 6 years ago
  1. the paper does a training over 40k comments and still get poor results which tell us that even extensive training is not enough probably.

However, they evaluate the tools (sentistrength, nltk, stanford NLP) and two tools with training on SE data (sentistrength SE and stanford corenlp SP) over 2500 comments and get poor results,

nasifimtiazohi commented 6 years ago

The novel contribution for sentiment analysis tools is now:

  1. apart from Senti4SD, we also have sentiCR now. That's two new tool that are SE focused
  2. Performing it over GItHub discussion comments
  3. A manual investigation on why Senti4SD failed in the discussion