realtime monitor is not working

[axot]

I added a line below in ossec.conf of agent. <directories report_changes="yes" realtime="yes" check_all="yes">/home/*/.ssh</directories>

But I can not found any changed(add file, alter file etc.) messages in alerts/alerts.log from server.

Does realtime monitor work now ? I'm using the latest ossec-hids-2.8.3-52.art.rpm

[axot]

When touch a new file with command touch add_file_test, the strace shows

)    = 1 (in [5], left {263, 672865})
read(5, "\7\0\0\0\0\1\0\0\0\0\0\0\20\0\0\0add_file_test\0\0\0"..., 65536) = 64
stat("/var/ossec/var/run/.syscheck_run", 0x7fff8f8ac6e0) = -1 ENOENT (No such file or directory)
select(6, [5], NULL, NULL, {300, 0})

[ddpbsd]

After adding the new directories option, did you run a full syscheck scan to add the existing files to the syscheck db? There has to be a baseline before there can be alerts.

[axot]

2016/03/22 20:52:35 ossec-syscheckd: DEBUG: Starting ...
2016/03/22 20:52:35 ossec-rootcheck: DEBUG: Starting ...
2016/03/22 20:52:35 ossec-rootcheck: Rootcheck disabled. Exiting.
2016/03/22 20:52:35 ossec-syscheckd: WARN: Rootcheck module disabled.
2016/03/22 20:52:41 ossec-syscheckd: INFO: (unix_domain) Maximum send buffer set to: '212992'.
2016/03/22 20:52:41 ossec-syscheckd: INFO: Started (pid: 6493).
2016/03/22 20:52:41 ossec-syscheckd: INFO: Monitoring directory: '/home/centos/.ssh'.
2016/03/22 20:52:41 ossec-syscheckd: INFO: Monitoring directory: '/home/admin/.ssh'.
2016/03/22 20:52:41 ossec-syscheckd: INFO: Monitoring directory: '/www'.
2016/03/22 20:52:41 ossec-syscheckd: INFO: Directory set for real time monitoring: '/home/centos/.ssh'.
2016/03/22 20:52:41 ossec-syscheckd: INFO: Directory set for real time monitoring: '/home/admin/.ssh'.
2016/03/22 20:52:41 ossec-syscheckd: INFO: Directory set for real time monitoring: '/www'.
2016/03/22 20:52:55 ossec-syscheckd: Setting SCHED_BATCH returned: 0
2016/03/22 20:54:35 ossec-syscheckd: INFO: Starting syscheck scan (forwarding database).
2016/03/22 20:54:35 ossec-syscheckd: INFO: Starting syscheck database (pre-scan).
2016/03/22 20:54:35 ossec-syscheckd: INFO: Initializing real time file monitoring (not started).
2016/03/22 20:54:35 ossec-syscheckd: DEBUG: Directory added for real time monitoring: '/home/centos/.ssh'.
2016/03/22 20:54:35 ossec-syscheckd: DEBUG: Directory added for real time monitoring: '/home/admin/.ssh'.
2016/03/22 20:54:35 ossec-syscheckd: DEBUG: Directory added for real time monitoring: '/www'.
2016/03/22 20:54:35 ossec-syscheckd: INFO: Real time file monitoring started.
2016/03/22 20:54:35 ossec-syscheckd: INFO: Finished creating syscheck database (pre-scan completed).
2016/03/22 20:54:49 ossec-syscheckd: INFO: Ending syscheck scan (forwarding database).

Here is the ossec.log of client. It shows syscheck database was created.

[ddpbsd]

Are the files in /home/*/.ssh present in the syscheck db for that system?

[axot]

touch add_file_test above was not present in syscheck db. Other files like authorized_keys did exists in syscheck db. In this case, even append content to authorized_keys, no alerts were received. strace of appending

read(5, "\6\0\0\0\2\0\0\0\0\0\0\0\20\0\0\0authorized_keys\0", 65536) = 32
lstat("/home/admin/.ssh/authorized_keys", {st_mode=S_IFREG|0644, st_size=1546, ...}) = 0
open("/home/admin/.ssh/authorized_keys", O_RDONLY) = 4
fstat(4, {st_mode=S_IFREG|0644, st_size=1546, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f461ba65000
read(4, "ssh-rsa AAAAB3NzaC1yc2EAAAADAQAB"..., 4096) = 1546
read(4, "", 4096)                       = 0
read(4, "", 4096)                       = 0
close(4)                                = 0
munmap(0x7f461ba65000, 4096)            = 0
stat("/var/ossec/queue/ossec/.wait", 0x7ffebf41ebe0) = -1 ENOENT (No such file or directory)
sendto(3, "8:syscheck:1546:33188:1001:1001:"..., 138, 0, NULL, 0) = 138
stat("/var/ossec/var/run/.syscheck_run", 0x7ffebf432f60) = -1 ENOENT (No such file or directory)
select(6, [5], NULL, NULL, {300, 0}

sendto method was called, but server did not receive this alert.

[ddpbsd]

New files are added to the database during a full system check. After you appended to the authorized_keys file, did a new entry with the new md5/sha1 appear in the syscheck db for that host?

[axot]

On agent

$ md5sum /home/admin/.ssh/authorized_keys
5982676ad2acb54d1a0611e501a743bc  /home/admin/.ssh/authorized_keys

On Server

$ ./syscheck_control -i 1035 -f /home/admin/.ssh/authorized_keys

Integrity changes for agent 'test (1035) - any/any':
Detailed information for entries matching: '/home/admin/.ssh/authorized_keys'

2016 Mar 18 21:45:30,0 - /home/admin/.ssh/authorized_keys
File added to the database. 
Integrity checking values:
   Size: 1514
   Perm: rw-r--r--
   Uid:  1001
   Gid:  1001
   Md5:  0a7780afb363e859dd85a7758a908db1
   Sha1: 47b4986fddc1f02e9ae38cabd83818ff9c5b8c6a

2016 Mar 22 12:24:50,0 - /home/admin/.ssh/authorized_keys
File changed. - 1st time modified.
Integrity checking values:
   Size: >1501
   Perm: rw-r--r--
   Uid:  1001
   Gid:  1001
   Md5:  >09b0637b43267b2cc23f52201ffeefec
   Sha1: >fbcd6ef2d0aea21d89aa3ab7501a30e6e1f81bcf

2016 Mar 22 12:50:39,2 - /home/admin/.ssh/authorized_keys
File changed. - 2nd time modified.
Integrity checking values:
   Size: >1507
   Perm: rw-r--r--
   Uid:  1001
   Gid:  1001
   Md5:  >757153db20a83539c00406034a0c07e0
   Sha1: >ea36bc7891a064668cda14184e1c8b85885973ad

2016 Mar 22 12:53:23,3 - /home/admin/.ssh/authorized_keys
File changed. - 3rd time modified.
Integrity checking values:
   Size: >1501
   Perm: rw-r--r--
   Uid:  1001
   Gid:  1001
   Md5:  >09b0637b43267b2cc23f52201ffeefec
   Sha1: >fbcd6ef2d0aea21d89aa3ab7501a30e6e1f81bcf

[axot]

After run syscheck_update command on server side, the problem seems be solved?

[server bin]# tail -f /var/ossec/logs/alerts/alerts.log
2016 Mar 22 23:57:34 (test) any->syscheck
Rule: 552 (level 7) -> 'Integrity checksum changed again (3rd time).'
Integrity checksum changed for: '/home/admin/.ssh/authorized_keys'
Size changed from '1572' to '1585'
Old md5sum was: 'a0091decf2335b7f20abdf73b2b47a50'
New md5sum is : 'df3b083dac3d6da2005eba34330520d0'
Old sha1sum was: '4e67a7f05be43d9cff28cf061879787cd3031e87'
New sha1sum is : '73583fb4a1cfa41ea2d9326e9a168e528dd31c70'

When report_changes will be alert?

Integrity checksum changed above not working again after few seconds when syscheck_update finished.

[axot]

I set a breakpoint at SendMSG. When I append hello to /home/admin/.ssh/authorized_keys SendMSG was called, but server did not show it in archives.log.

Breakpoint 1, SendMSG (queue=7,
    message=message@entry=0x7ffffffeb8d0 "1525:33188:1001:1001:791f82a1739ed26aac568d8a6fdc8207:1b866252196cedb6e7a072552cb1240e352b417f /home/admin/.ssh/authorized_keys\n5a6\n> hello\n",
    locmsg=locmsg@entry=0x5555555764aa "syscheck", loc=loc@entry=56 '8')
    at mq_op.c:83

If I call same SendMSG() with other MQs like ROOTCHECK_MQ in GDB, I can receive this archives.log message in server.

(gdb) call SendMSG(7,0x7ffffffeb8d0,0x5555555764aa,'9')

The archives.log message shows in server

2016 Mar 23 10:21:57 (test) any->syscheck 1525:33188:1001:1001:791f82a1739ed26aac568d8a6fdc8207:1b866252196cedb6e7a072552cb1240e352b417f /home/admin/.ssh/authorized_keys
5a6
> hello

[axot]

With source code debugging, I found this problem was occurred by

DB_Search function in syscheck.c

int DB_Search(char *f_name, char *c_sum, Eventinfo *lf)
...
/* Checking the number of changes */
if(!Config.syscheck_auto_ignore)                         // <- this cause issue
{
    sdb.syscheck_dec->id = sdb.id1;
}
else
{
    switch(p)
    {
        case 0:
        sdb.syscheck_dec->id = sdb.id1;
        break;

        case 1:
        sdb.syscheck_dec->id = sdb.id2;
        break;

        case 2:
        sdb.syscheck_dec->id = sdb.id3;
        break;

        default:                                             
        lf->data = NULL;
        return(0);
        break;
    }
}
...

Add <auto_ignore>no</auto_ignore> to <syscheck> section solved this issue. Thank you.

[santiago-bassett]

Was following this with attention. Happy to learn this was the issue and not a bug. Auto_ignore is one of the first things I set to no. Sorry I didn't say that before (didn't thought that was it)

[axot]

@Because with the default auto_ignore option, realtime monitor won't work properly. It's better to add a description about auto_ignore in realtime manual.

[mael2]

@axot Thank you very much for your analysis, I had the same problem