search

uskay / TwitterStreams4MarkLogic

0 stars 0 forks source link

Java Batch(Twitter4J) throws OutOfMemoryError #2

Open uskay opened 9 years ago

uskay commented 9 years ago

Exception in thread "Twitter Stream consumer-1[Receiving stream]" java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOfRange(Arrays.java:2694) at java.lang.String.(String.java:203) at java.lang.StringBuffer.toString(StringBuffer.java:561) at java.io.BufferedReader.readLine(BufferedReader.java:352) at java.io.BufferedReader.readLine(BufferedReader.java:382) at twitter4j.StatusStreamBase.handleNextElement(StatusStreamBase.java:75) at twitter4j.StatusStreamImpl.next(StatusStreamImpl.java:56) at twitter4j.TwitterStreamImpl$TwitterStreamConsumer.run(TwitterStreamImpl.java:537)

uskay commented 9 years ago

The root problem could be this. Do not do heavy process in onStatus()... https://groups.google.com/forum/#!topic/twitter4j/Z0EagpSqID8

uskay commented 9 years ago

Lacking resource... PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4813 daemon 20 0 3775m 650m 34m S 61.3 32.4 1:35.56 MarkLogic
5244 ec2-user 20 0 1157m 100m 11m S 24.5 5.0 0:15.22 java

uskay commented 9 years ago

Created a watchDog

while true; do

size=$(wc -c tw.log | awk '{print $1}')
echo $size
if [ $size -eq 0 ]; then
    :
else 
    USER=ec2-user
    TARGET=TwitterStreamApp
    pids=(`ps -ef | grep ${USER} | grep ${TARGET} | grep -v grep | awk '{ print $2; }'`)
    for pid in ${pids[*]}
    do
    kill -9 ${pid}
    sleep 10
    bash start.sh
    done
fi

sleep 5

done