Add Dynamic Eventlist fields with decoder support

[jrossi]

This is a break backwards compatibility feature

Kill all the chars in eventlist and add hash of key/value pairs as they are decoded.

Keys would be just about anything but some form of standard is needed to make all the current rules work. An example is syscheck decoder would create the following keys: values

• file.md5.before: aaaaaaaaaa
• file.md5.after: bbbbbbbbbb
• file.sha3.before: aaaaa....aaaaa
• file.sha3.after: bbbbbb....bbbbb
• file.owner.before: nobody
• file.owner.after: root
• file.perm.before: 0500
• file.perm.after: 1777
• file.name: /bin/ls
• hostname: host_dmz_hacked_1.owned.com

A second example would be fire wall decoding:

• src.port: 34240
• src.ipv4: 1.1.1.1
• proto: tcp
• dst.port: 443
• dst.ipv4: 2.2.2.2
• dst.uid: 100
• dst.username: apache
• dst.process: httpd
• hostname: www.example.com

Or something more complex like Linux squid iptables with NAT and transparent proxy:

• src.port: 34240
• src.ipv4: 1.1.1.1
• proto: tcp
• dst.port: 443
• dst.ipv4: 2.2.2.2
• dst.uid: 100
• dst.username: squid3
• dst.process: squid3
• dst.nat.ipv4: 127.0.0.1
• dst.nat.port: 3189
• hostname: proxy.example.com

The reason for the dotted names is that I want to be able to convert this info into json and still keep it readable amd usable. Like the following;

#!json

{
  "src":{
     "port": 34240,
     "ipv4": "1.1.1.1"
  },
  "proto": "tcp",
  "dst":{
     "port":443,
     "ipv4": "2.2.2.2"
   },
   "hostname": "proxy.example.com"
}

Why json? Well I want to remove the decoding of ossec generated messages and just receive well formatted json messages from remote agents. So much simpler then encoding things into a log message and decoding on the other end. This would allow so many other future features.

Now how to implement this. To be frank this would require breaking some things and a HUGE effort to port all the rules. The c code is easy in comparison.

Eventlist will just end up with a uthash.h lookup based on the key. Something like this

#!c

#define MAX_KEY_SIZE 64;
#define MAX_VALUE_SIZE 900;
struct os_key_value {
   UT_hash_handle hh;
   char key[MAX_KEY_SIZE];
   char value[MAX_VALUE_SIZE];
};

New with uthash we have a fast method of finding all keys in memory and allocation can be implemented in blocks as many as needed with a single malloc.

Why the single malloc, well i have done some testing of ossec and malloc is the second most used function and a huge amount of time is eaten up by it. I •think• we can make things faster by managing memory better. But alas I am optimizing before I code.

Rules would then look somethign like the following:

#!xml

<rule id="100000" level="7">
  <list lookup="match_key" field="src.ipv4">path/to/list/file</list>
  <description>Checking src.ipv4 against cdb list file</description>
</rule>

#!xml

<rule id="100000" level="15">
  <list lookup="match_key" field="file.name">path/to/list/file</list>
  <description>Checking file.name against cdb list file</description>
</rule>

The point of having the data be more structured is that rules have more exact fields to test for. And decoders have more space to extract data into.

Note: This issue has been automatically migrated from Bitbucket Created by jrossi on 2013-09-13 00:54:47+00:00, last updated: 2013-09-13 01:15:44+00:00

[jrossi]

If possible please assign this to me.

Note: This comment has been automatically migrated from Bitbucket Created by jrossi on 2013-09-13 00:55:55+00:00