VADER Sentiment Analysis. VADER (Valence Aware Dictionary and sEntiment Reasoner) is a lexicon and rule-based sentiment analysis tool that is specifically attuned to sentiments expressed in social media, and works well on texts from other domains.
MIT License
4.38k
stars
1k
forks
source link
Special case huge performance regression in 3.3.1+ #110
I've been processing random comments from social media and noticed some strange spikes in processing time logs. Generally it takes less than a second to process a chunk of data. But here on the plot you can see a dot approaching 10 mins!
I traced the slow-down back to vader 3.3.1+ 100% cpu usage on texts with huge amount of emoticons.
Test script vader.py
#!/usr/bin/env python3
import sys
import json
from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer
si = SentimentIntensityAnalyzer()
for line in sys.stdin:
line = json.loads(line)
print(json.dumps(si.polarity_scores(line), sort_keys=True))
sys.stdout.flush()
I've been processing random comments from social media and noticed some strange spikes in processing time logs. Generally it takes less than a second to process a chunk of data. But here on the plot you can see a dot approaching 10 mins!
I traced the slow-down back to vader 3.3.1+ 100% cpu usage on texts with huge amount of emoticons.
Test script
vader.py
Sample input
slow.json
https://pastebin.com/nxjLSTMQvader 3.2.1:
vader 3.3.1+:
The input sample is not an artificial joke btw. Here are samples of what supposedly real people post on reddit and I feed vader with stuff like that: