shinken-solutions / shinken

Flexible and scalable monitoring framework
http://www.shinken-monitoring.org
GNU Affero General Public License v3.0
1.13k stars 335 forks source link

"Exception: Cannot allocate memory [12]" in shinken-arbiterd.log #1922

Open serge-marie opened 7 years ago

serge-marie commented 7 years ago

Hi,

I had a big broker with shinken-arbiter this morning.

Process shinken-arbiter didn't want to start after a "restart ou reload" action.

We get this message and log shinken-arbiterd.log

[1500629498] CRITICAL: [Shinken] I got an unrecoverable error. I have to exit. [1500629498] CRITICAL: [Shinken] You can get help at https://github.com/naparuba/shinken [1500629498] CRITICAL: [Shinken] If you think this is a bug, create a new ticket includingdetails mentioned in the README [1500629498] CRITICAL: [Shinken] Back trace of the error: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/shinken/daemons/arbiterdaemon.py", line 627, in main self.do_daemon_init_and_start() File "/usr/lib/python2.7/site-packages/shinken/daemon.py", line 670, in do_daemon_init_and_start self.daemonize(skip_close_fds=socket_fds) File "/usr/lib/python2.7/site-packages/shinken/daemon.py", line 572, in daemonize raise Exception("%s [%d]" % (e.strerror, e.errno)) Exception: Cannot allocate memory [12]

I had to do a reboot of the server because every start action failed ...

Memory consuption was a little hight :

image

Any idea ?

Related to "Memory leak" issues already opened ? https://github.com/naparuba/shinken/issues/1869

Upgrade to 3.00.03

Thank's in advance.

Serge

geektophe commented 6 years ago

Hello,

Sorry for my late answer.

Yes, it can be related related. The memory leak mentioned in this ticket was in the scheduler daemon, which is resident (it's not restarted when a new configuration is pushed).

If it happens again, could you check which daemon is consuming the most memory ? And tell us how many hosts you monitor ?