search

larvalabs / breaker

Chat rooms for subreddits.
http://www.breakerapp.com
20 stars 3 forks source link

Mark read job is crashing #20

Closed megamattron closed 8 years ago

megamattron commented 8 years ago

The job that periodically writes out the last read message time for users is crashing. Here's what I found in the logs:

115 <190>1 2016-06-08T10:53:25.178347+00:00 app web.1 - - 10:53:24,730 ERROR ~
94 <190>1 2016-06-08T10:53:25.178357+00:00 app web.1 - -
104 <190>1 2016-06-08T10:53:25.178358+00:00 app web.1 - - @70bc0gd81
160 <190>1 2016-06-08T10:53:25.178359+00:00 app web.1 - - Error during job execution (jobs.SaveLastReadTimeForAllPendingJob) Context
94 <190>1 2016-06-08T10:53:25.178359+00:00 app web.1 - -
116 <190>1 2016-06-08T10:53:25.178360+00:00 app web.1 - - Oops: OutOfMemoryError
186 <190>1 2016-06-08T10:53:25.178361+00:00 app web.1 - - An unexpected error occured caused by exception OutOfMemoryError: GC overhead limit exceeded
94 <190>1 2016-06-08T10:53:25.178361+00:00 app web.1 - -
147 <190>1 2016-06-08T10:53:25.178363+00:00 app web.1 - - play.exceptions.UnexpectedException: Unexpected Error
151 <190>1 2016-06-08T10:53:25.178364+00:00 app web.1 - - at play.Invoker$Invocation.onException(Invoker.java:245)
137 <190>1 2016-06-08T10:53:25.178364+00:00 app web.1 - - at play.jobs.Job.onException(Job.java:156)
130 <190>1 2016-06-08T10:53:25.178365+00:00 app web.1 - - at play.jobs.Job.call(Job.java:233)
119 <190>1 2016-06-08T10:53:25.178366+00:00 app web.1 - - at Invocation.Job(Play!)
159 <190>1 2016-06-08T10:53:25.178366+00:00 app web.1 - - Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded

Which is weird because there wasn't generally any other out of memory problems. Will increase heap and see.

megamattron commented 8 years ago

Back up to 600M heap, will monitor and reopen if it reoccurs.