search

stagemonitor / stagemonitor-mailinglist

GitHub issues abused as a mailing list
3 stars 0 forks source link

OutOfMemory due to messages stored in memory - when no connection to Elastic. #16

Open noban opened 8 years ago

noban commented 8 years ago

OutOfMemory due to messages stored in memory - when no connection to Elastic. Basically we have encountered this situation. Is there any way to configure Stagemonitor not to store these messages, or at least set some limit to count of messages / size of the queue?

felixbarny commented 8 years ago

What do you mean by messages? So your problem was that elasticsearch was unexpectedly down which lead to a OOM Error?

noban commented 8 years ago

We're talking about large amount of FutureTask objects in StageMonitor's ThreadPoolExecutor that appeared in the heapdump. We've had a network problem that caused delivery to fail and as a result workQueue of this executor has consumed all the available memory. histogram objects-1

felixbarny commented 8 years ago

I'm sorry for the issues. This should be easy to fix though. I'll probably limit the LinkedBlockingQueue of the Executor to hold max. 1000 Futures. Thx for the detailed analysis

noban commented 8 years ago

Great to hear, thank you in advance!

kgusarov commented 8 years ago

@felixbarny It would be nice idea to introduce this limit in the configuration.

felixbarny commented 8 years ago

I've just released a new version that contains the fix: https://github.com/stagemonitor/stagemonitor/releases/tag/0.18.0

noban commented 8 years ago

Thank you for a really quick fix!