Parsing a Custom Attribute in All Message and/or Modifying Parsers

[GoogleCodeExporter]

Hi,

All syslog messages ultimately received by our ELSA server pass through a 
syslog relay which injects an attribute into the message. The format of the 
injected attribute and its position in the message is up to us. Currently, the 
format is key=value. We're interested in parsing out this attribute and making 
'key' a field that can be used for ELSA queries. Is this possible? Seeing that 
the t-ops template uses STRUCTURED-DATA, I tried to modify the t_db_parsed 
template to also have a STRUCTURED-DATA field and send the attribute as 
STRUCTURED-DATA, but I couldn't get that to work, and I'm not sure if I should 
have expected it to.

I then decided to make do with only parsing out the attribute in SNORT messages 
and tried to modify the parser to accept them as suggested in the 
documentation. I changed the first pattern in patterndb.xml to read:

key=@STRING:s0@ @QSTRING:s1:[]@ @ESTRING:s2:[@Classification:@QSTRING:s3: ]@ 
[Priority: @NUMBER:i0:@] @QSTRING:i1:{}@ @IPv4:i2:@:@NUMBER:i3:@ -> 
@IPv4:i4:@:@NUMBER:i5:@

I then tried to use 'pdbtool match' to verify this part was working correctly, 
and it did:

$ /usr/local/syslog-ng/bin/pdbtool match -p 
/usr/local/elsa/node/conf/patterndb.xml -P snort -M "key=foo [1:2013028:3] ET 
POLICY curl User-Agent Outbound [Classification: Attempted Information Leak] 
[Priority: 2] {TCP} 10.0.0.218 -> 23.63.227.176"
MESSAGE=key=foo [1:2013028:3] ET POLICY curl User-Agent Outbound 
[Classification: Attempted Information Leak] [Priority: 2] {TCP} 10.0.0.218 -> 
23.63.227.176
PROGRAM=snort
.classifier.class=8
.classifier.rule_id=8
i0=2
s0=foo
i1=TCP
i2=10.8.7.218
s1=1:2013028:3
i4=23.63.227.176
s2=ET POLICY curl User-Agent Outbound
s3=Attempted Information Leak

I then modified the syslog database as follows:

mysql> select * from fields;
+-----+--------------------+------------+--------------+------------------+
| id  | field              | field_type | pattern_type | input_validation |
+-----+--------------------+------------+--------------+------------------+
|   1 | timestamp          | int        | NONE         | NULL             |
|   2 | minute             | int        | NONE         | NULL             |
|   3 | hour               | int        | NONE         | NULL             |
|   4 | day                | int        | NONE         | NULL             |
|   5 | host_id            | int        | NONE         | NULL             |
|   6 | program_id         | int        | NONE         | NULL             |
<snip>
| 131 | pkts_out           | int        | NUMBER       | NULL             |
| 132 | bytes_in           | int        | NUMBER       | NULL             |
| 133 | bytes_out          | int        | NUMBER       | NULL             |
| 142 | key      | string     | QSTRING      | NULL             |
+-----+--------------------+------------+--------------+------------------+
110 rows in set (0.00 sec)

mysql> select * from fields_classes_map where class_id = 8;
+----------+----------+-------------+
| field_id | class_id | field_order |
+----------+----------+-------------+
|      142 |        8 |           5 |
|       43 |        8 |           6 |
|       13 |        8 |           7 |
|       15 |        8 |           8 |
|       16 |        8 |           9 |
|       18 |        8 |          10 |
|       19 |        8 |          11 |
|       40 |        8 |          12 |
|       41 |        8 |          13 |
|       42 |        8 |          14 |
|       67 |        8 |          15 |
+----------+----------+-------------+
11 rows in set (0.00 sec)

syslog-ng was restarted. However, messages like "key=foo [1:2013028:3] ET 
POLICY curl User-Agent Outbound [Classification: Attempted Information Leak] 
[Priority: 2] {TCP} 10.0.0.218 -> 23.63.227.176" are not identified as SNORT 
class messages, and I can't seem to search using 'key' as a field.

What am I missing?

Thanks

Original issue reported on code.google.com by ruvi...@gmail.com on 15 Sep 2013 at 10:59

[GoogleCodeExporter]

That's an interesting challenge.  There are a number of ways to do what you're 
trying to do, but none of them are particularly easy.  Let's try to make the 
method you've started to do work.  The best thing to do when trying to 
troubleshoot a parser is to use contrib/livetail.pl to monitor the incoming 
messages as they are parsed.  That utility will print the messages as they 
appear to the elsa.pl script that normalizes and eventually sends logs to the 
indexer.   You can change merged.xml then without restarting syslog-ng to watch 
the changes.

Note that it is merged.xml, not patterndb.xml that syslog-ng reads.  Custom 
patterns are meant to be put in /etc/elsa/patterns.d, and then the install.sh 
script will merge those patterns with stock ELSA patterns into the 
/usr/local/elsa/node/conf/merged.xml file, which is what syslog-ng actually 
uses.

I believe your field_order is wrong for the newly added "key" field.  It should 
be field_order 11 (s0) and not 5 (i0).

Original comment by mchol...@gmail.com on 17 Sep 2013 at 2:37

[GoogleCodeExporter]

Thanks for your reply. I will give it a try. I have a question before I do, 
though: are the field_order values documented anywhere? The documentation 
points to $Field_order in web/lib/API.pm, but I don't see that in that file.

Original comment by ruvi...@gmail.com on 17 Sep 2013 at 1:37

[GoogleCodeExporter]

Check out the top of the elsa/web/lib/Fields.pm file which lays those out.

Original comment by mchol...@gmail.com on 19 Sep 2013 at 2:10

[GoogleCodeExporter]

Thanks! I was able to get my custom snort parsers to work. The problems were as 
you suggested: the merged.xml file wasn't being modified, and my field_order 
was wrong.

If I could turn back to my more general question, is there a way to parse out 
my field in all messages (like the generic hostname and program fields)?

Original comment by ruvi...@gmail.com on 24 Sep 2013 at 12:00

[GoogleCodeExporter]

Glad to hear it's working.  No, there's no good way to parse that field out 
from all messages without creating a parser for every kind of message you would 
receive.

Original comment by mchol...@gmail.com on 27 Sep 2013 at 3:00

• Changed state: Done