search

ossec / ossec-hids

OSSEC is an Open Source Host-based Intrusion Detection System that performs log analysis, file integrity checking, policy monitoring, rootkit detection, real-time alerting and active response.
http://www.ossec.net
Other
4.42k stars 1.03k forks source link

[BUG] Remoted Error Socket #1592

Closed uilianmengue closed 5 years ago

uilianmengue commented 5 years ago

Hello,

I instaled OSSEC 3.1.0 anda the proccess remoted is stop anytime. Bellow log datails.

2018/12/10 20:09:33 ossec-remoted: socketerr (not available). 2018/12/10 20:09:33 ossec-remoted(1210): ERROR: Queue '/queue/ossec/queue' not accessible: 'Connection refused'. 2018/12/10 20:09:34 ossec-logcollector: socketerr (not available). 2018/12/10 20:09:34 ossec-logcollector(1224): ERROR: Error sending message to queue. 2018/12/10 20:09:36 ossec-remoted(1210): ERROR: Queue '/queue/ossec/queue' not accessible: 'Connection refused'. 2018/12/10 20:09:36 ossec-remoted(1211): ERROR: Unable to access queue: '/queue/ossec/queue'. Giving up.. 2018/12/10 20:09:37 ossec-logcollector(1210): ERROR: Queue '/var/ossec/queue/ossec/queue' not accessible: 'Connection refused'. 2018/12/10 20:09:37 ossec-logcollector(1211): ERROR: Unable to access queue: '/var/ossec/queue/ossec/queue'. Giving up.. 2018/12/10 20:10:01 ossec-agentlessd: ERROR: Too many failures for 'ssh_integrity_check_linux'. Ignoring it. 2018/12/10 20:23:17 ossec-monitord: socketerr (not available). 2018/12/10 20:23:17 ossec-monitord(1224): ERROR: Error sending message to queue. 2018/12/10 20:25:17 ossec-monitord: socketerr (not available). 2018/12/10 20:25:17 ossec-monitord(1224): ERROR: Error sending message to queue. 2018/12/10 20:27:17 ossec-monitord: socketerr (not available). 2018/12/10 20:27:17 ossec-monitord(1224): ERROR: Error sending message to queue. 2018/12/10 20:27:17 ossec-monitord: socketerr (not available). 2018/12/10 20:27:17 ossec-monitord(1224): ERROR: Error sending message to queue.

ddpbsd commented 5 years ago

Does this happen immediately? Is ossec-analysisd running?

uilianmengue commented 5 years ago

Does it happen immediately? Yes, once a day at different times Is ossec-analysis running? No, it also stops.

uilianmengue commented 5 years ago

Dec 10 20:09:32 kernel: Out of memory: Kill process 28057 (ossec-analysisd) score 907 or sacrifice child Dec 10 20:09:32 kernel: Killed process 28057 (ossec-analysisd) total-vm:10894700kB, anon-rss:7495996kB, file-rss:0kB, shmem-rss:0kB Dec 10 20:10:02 kernel: gsch_redirfs_add_mnt(/run/user/0 @ Unknown[1021994(tmpfs)]) done: 0 Dec 10 20:10:02 kernel: [650(systemd-logind)]: gsch_mount_hook_fn(tmpfs,/run/user/0,tmpfs,6,000055baa09267f0) done

ddpbsd commented 5 years ago

Ok, so when ossec-analysisd is killed, the other daemons start to fail.

uilianmengue commented 5 years ago

Bellow ossec-monitord ossec-logcollector ossec-analysisd

It was killed.

uilianmengue commented 5 years ago

Bellow errors at the restart teh service.

[root@host bin]# ./ossec-control stop Deleting PID file '/var/ossec/var/run/ossec-logcollector-26235.pid' not used... Deleting PID file '/var/ossec/var/run/ossec-remoted-26242.pid' not used... Deleting PID file '/var/ossec/var/run/ossec-syscheckd-26247.pid' not used... Deleting PID file '/var/ossec/var/run/ossec-analysisd-26231.pid' not used... Killing ossec-monitord .. ossec-logcollector not running .. Killing ossec-remoted .. ossec-syscheckd not running .. ossec-analysisd not running .. ossec-maild not running .. ossec-execd not running .. Killing ossec-csyslogd .. Killing ossec-agentlessd .. OSSEC HIDS v3.0.0 Stopped [root@host bin]#

ddpbsd commented 5 years ago

So the problem is ossec-analysisd is taking up too much memory for the system. Either the system doesn't have enough ram, ossec-analysisd has been configured in a way that eats too much memory, or you've triggered some memory leak. The first is possibly the easiest to determine. If there are a lot of agents or very busy agents analysisd will use more ram. The second might be easy to check for, and it might not. Do you have very broad rules? Maybe something watching for just a specific srcip? The third is a lot tougher. Not sure how to track that down specifically.

uilianmengue commented 5 years ago

I'm using default configuration for rules. I have 700 agents registered, and 550 connected now.

Memory: total used free
7822 1468 5234

uilianmengue commented 5 years ago

Thanks for your help. I found the problem, ossec-analysisd overloaded the ram memory. The OOM_Killer of the linux killer the process because system dont have more resources.

image