search

ossec / ossec-hids

OSSEC is an Open Source Host-based Intrusion Detection System that performs log analysis, file integrity checking, policy monitoring, rootkit detection, real-time alerting and active response.
http://www.ossec.net
Other
4.42k stars 1.03k forks source link

Many "Duplicate error" in logs after upgrade to 3.2.0 #1679

Closed umgeorgem closed 1 year ago

umgeorgem commented 5 years ago

Hello,

We upgraded our ossec servers from Ubuntu 14.04 to Ubuntu 16.04 and in the same time from Ossec 2.8.3 to the latest version, 3.2.0. We made an upgrade and kept the existing clients (hundreds of them, most of then Ubuntu 14.04, Ubuntu 16.04 and Centos 7).

Everything seemed fine, except that we noticed a lot of warnings/errors in the /var/ossec/logs/ossec.log file on the ossec servers. The errors are like this:

# grep Duplicate /var/ossec/logs/ossec.log  | tail -n 25
2019/03/01 12:15:34 ossec-remoted(1407): ERROR: Duplicated counter for '<server1>'.
2019/03/01 12:15:34 ossec-remoted: WARN: Duplicate error:  global: 740, local: 8875, saved global: 740, saved local:8876
2019/03/01 12:15:34 ossec-remoted(1407): ERROR: Duplicated counter for '<server1>'.
2019/03/01 12:16:36 ossec-remoted: WARN: Duplicate error:  global: 372, local: 5975, saved global: 372, saved local:5976
2019/03/01 12:16:36 ossec-remoted(1407): ERROR: Duplicated counter for '<server2>'.
2019/03/01 12:18:36 ossec-remoted: WARN: Duplicate error:  global: 372, local: 5984, saved global: 372, saved local:5985
2019/03/01 12:18:36 ossec-remoted(1407): ERROR: Duplicated counter for '<server2>'.
2019/03/01 12:20:02 ossec-remoted: WARN: Duplicate error:  global: 29, local: 3173, saved global: 29, saved local:3174
2019/03/01 12:20:02 ossec-remoted(1407): ERROR: Duplicated counter for '<server3>'.
2019/03/01 12:20:02 ossec-remoted: WARN: Duplicate error:  global: 29, local: 3172, saved global: 29, saved local:3174
2019/03/01 12:20:02 ossec-remoted(1407): ERROR: Duplicated counter for '<server3>'.
2019/03/01 12:20:09 ossec-remoted: WARN: Duplicate error:  global: 741, local: 2715, saved global: 741, saved local:2717
2019/03/01 12:20:09 ossec-remoted(1407): ERROR: Duplicated counter for '<server1>'.
2019/03/01 12:20:36 ossec-remoted: WARN: Duplicate error:  global: 372, local: 5999, saved global: 372, saved local:6000
2019/03/01 12:20:36 ossec-remoted(1407): ERROR: Duplicated counter for '<server2>'.
2019/03/01 12:24:55 ossec-remoted: WARN: Duplicate error:  global: 26, local: 5084, saved global: 26, saved local:5086
2019/03/01 12:24:55 ossec-remoted(1407): ERROR: Duplicated counter for '<server4>'.
2019/03/01 12:29:37 ossec-remoted: WARN: Duplicate error:  global: 372, local: 6049, saved global: 372, saved local:6050
2019/03/01 12:29:37 ossec-remoted(1407): ERROR: Duplicated counter for '<server2>'.
2019/03/01 12:30:01 ossec-remoted: WARN: Duplicate error:  global: 29, local: 3187, saved global: 29, saved local:3189
2019/03/01 12:30:01 ossec-remoted(1407): ERROR: Duplicated counter for '<server3>'.
2019/03/01 12:30:01 ossec-remoted: WARN: Duplicate error:  global: 29, local: 3188, saved global: 29, saved local:3189
2019/03/01 12:30:01 ossec-remoted(1407): ERROR: Duplicated counter for '<server3>'.
2019/03/01 12:30:19 ossec-remoted: WARN: Duplicate error:  global: 741, local: 8528, saved global: 741, saved local:8529
2019/03/01 12:30:19 ossec-remoted(1407): ERROR: Duplicated counter for '<server1>'.

(I replaced the real hostnames of the clients in the logs with )

and there are many errors.

# grep Duplicate /var/ossec/logs/ossec.log  | wc -l
7886

The ossec is working as expected, we triggered some manual alerts on the clients that appear in logs and we received the notifications, but probably some logs are skipped because of this issue.

We tried what the help says (https://ossec-docs.readthedocs.io/en/latest/faq/unexpected.html#fixing-duplicate-errors) with many of the clients, more than once, and it did not work. Before the upgrade we used this method a few times to fix similar problems and it worked.

We also tried with one of the clients to remove it from the ossec server and register it again, without any results, the errors appeared after that again. We also removed the older ossec agent (same client, from the logs above) and installed the last version (Centos 7 packet from the ossec repository), but nothing has changed. The client is an ldap server, it receives a lot of requests and has many entries in the logs.

All the clients are actually servers with many log entries.

I did not fund other reports with this issue after the upgrade to 3.2.0, that's why I opened this Github issue, I hope this is the right way to report the problem (and to hope that it will be fixed, if it really is a problem in the code).

Best regards, George Mihalcea

RAYs3T commented 5 years ago

Exactly the same issue after upgrading from 2.8.4 to 3.2.

Running on Suse with approx. 250 clients.

Also the normal ways of fixing this – including removing and new registering the clients – does not work. It seems to be related to LAN/WAN latency, because most of the affected clients are on remote locations.

Additionally active response is not working stable anymore. Sometimes this is OK sometimes events don’t trigger any response at all. Perhaps a result of the problems with the counter.

umgeorgem commented 5 years ago

In our case the problem is not the latency, the server and the clients (from the log file in my initial message) are in the same DC, and the network connections are very good.

trietphm commented 3 years ago

I'm having this problem with v3.6.0 as well. I tried the solution in the doc but it does not work https://www.ossec.net/docs/docs/faq/unexpected.html#fixing-duplicate-errors

titleistfour commented 2 years ago

Anyone find a solution for this? We have the same issue after upgrade to v3.6.0. Many clients showing these errors and completely removing the key and adding it again doesn't seem to fix. Clearing out queue/rids directory didn't fix the issue either.

libellux commented 1 year ago

Stop both the OSSEC server and the agent. In the agent server go to /var/ossec/queue/rids and remove all the files within the folder. At the OSSEC server go into /var/ossec/queue/rids and remove the file corresponding to the agents ID. Do not delete the sender_counter. Restart both.

Or disable the feature by editing /var/ossec/etc/internal_options.conf