New Parser for SFIMS - trouble reading double quotes within message?

[GoogleCodeExporter]

We're running on 64-bit Ubuntu with the following versions installed (as part 
of the script):
eventlog_0.2.12.tar.gz
syslog-ng_3.2.2.tar.gz
elsa-0.1.1.tar.gz

We're trying to write a new parser for SFIMS messages; there is already an 
sfims program defined but these messages currently fall into the "NONE" class. 
I have modified the MySQL DB so that the tables are as follows (no new fields 
created - only the class and the entries in field_classes_map).....

mysql> select * from classes where id = 10000;
+-------+-------+-----------+
| id    | class | parent_id |
+-------+-------+-----------+
| 10000 | SFIMS |         0 |
+-------+-------+-----------+
1 row in set (0.00 sec)

mysql> select * from fields_classes_map where class_id=10000;
+----------+----------+-------------+
| field_id | class_id | field_order |
+----------+----------+-------------+
|       43 |    10000 |           5 |
|       15 |    10000 |           6 |
|       16 |    10000 |           7 |
|       18 |    10000 |           8 |
|       19 |    10000 |           9 |
|       13 |    10000 |          11 |
+----------+----------+-------------+
6 rows in set (0.00 sec)

mysql> select * from fields where id in (select field_id from 
fields_classes_map where class_id=10000);
+----+--------------+------------+--------------+------------------+
| id | field        | field_type | pattern_type | input_validation |
+----+--------------+------------+--------------+------------------+
| 13 | proto        | int        | QSTRING      | NULL             |
| 15 | srcip        | int        | IPv4         | IPv4             |
| 16 | srcport      | int        | NUMBER       | NULL             |
| 18 | dstip        | int        | IPv4         | IPv4             |
| 19 | dstport      | int        | NUMBER       | NULL             |
| 43 | sig_priority | int        | NUMBER       | NULL             |
+----+--------------+------------+--------------+------------------+
6 rows in set (0.00 sec)

I have then entered the following at the bottom of the patterndb.xml (under 
/opt/elsa/node/conf/ in our case).....

<ruleset name='sfims' id='10000'>
                <pattern>sfims</pattern>
                <rules>
                        <rule provider='ELSA' class='10000' id='10000'>
                                <patterns>
                                        <pattern>[@ESTRING::]@[@ESTRING::]@[@ESTRING::]@ @ESTRING::]@ User@ESTRING::[@Priority: @NUMBER:i0:@] @QSTRING:s0:{}@ @IPv4:i1:@:@NUMBER:i2:@ -&gt; @IPv4:i3:@:@NUMBER:i4:@</pattern>
                                        <pattern>[@ESTRING::]@[@ESTRING::]@[@ESTRING::]@ @ESTRING::]@ User@ESTRING::[@Priority: @NUMBER:i0:@] @QSTRING:s0:{}@ @IPv4:i1:@ -&gt; @IPv4:i3:@</pattern>
                                </patterns>
                                <examples>
                                        <example>
                                                <test_message program='sfims'>[Primary Detection Engine (f3fe03e8-8a2f-11e3-97cb-abab35940b61)][PASSIVE_CATCH_ALL][1:13249:8] "PROTOCOL-DNS dns response for rfc1918 10/8 address detected" [Classification: Potential Corporate Policy Violation] User: Unknown, Application: Unknown, Client: Unknown, App Protocol: UnknownInterface Ingress: s1p2, Interface Egress: s1p1, Security Zone Ingress: Passive, Security Zone Egress: Passive, [Priority: 1] {UDP} 10.0.1.2:53 -&gt; 10.5.1.13:57298</test_message>
                                                <test_values>
                                                        <test_value name='i0'>1</test_value>
                                                        <test_value name='s0'>UDP</test_value>
                                                        <test_value name='i1'>10.0.1.2</test_value>
                                                        <test_value name='i2'>53</test_value>
                                                        <test_value name='i3'>10.5.1.13</test_value>
                                                        <test_value name='i4'>57298</test_value>
                                                </test_values>
                                        </example>
                                </examples>
                        </rule>
                </rules>
</ruleset>

Following a "service syslog-ng restart" I have then attempted to test the 
parser against a sample SFIMS message which seems to be successful.....

root@ELSA-SRV:/opt/elsa/node/conf# /usr/bin/pdbtool match -p 
/opt/elsa/node/conf/patterndb.xml -P sfims -M "[Primary Detection Engine 
(f3fe03e8-8a2f-11e3-97cb-abab35940b61)][PASSIVE_CATCH_ALL][138:5:1] Snort Alert 
[Classification: Sensitive Data] User: Unknown, Application: Unknown, Client: 
Unknown, App Protocol: UnknownInterface Ingress: s1p2, Interface Egress: s1p1, 
Security Zone Ingress: Passive, Security Zone Egress: Passive, [Priority: 2] 
{TCP} 172.17.1.13:54493 -> 172.17.5.6:8080"
MESSAGE=[Primary Detection Engine 
(f3fe03e8-8a2f-11e3-97cb-abab35940b61)][PASSIVE_CATCH_ALL][138:5:1] Snort Alert 
[Classification: Sensitive Data] User: Unknown, Application: Unknown, Client: 
Unknown, App Protocol: UnknownInterface Ingress: s1p2, Interface Egress: s1p1, 
Security Zone Ingress: Passive, Security Zone Egress: Passive, [Priority: 2] 
{TCP} 172.17.1.13:54493 -> 172.17.5.6:8080
PROGRAM=sfims
.classifier.class=10000
.classifier.rule_id=10000
i0=2
s0=TCP
i1=172.17.1.13
i2=54493
i3=172.17.5.6
i4=8080
TAGS=.classifier.10000

However, if I try this against another sample message (which happens to contain 
"double quotes"), the message is only understood up to 'SERVER-IIS' so the 
parser subsequently returns no results. If I modify the message by removing the 
double quotes and re-run it, the parsing is successful.....

root@ELSA-SRV:/opt/elsa/node/conf# /usr/bin/pdbtool match -p 
/opt/elsa/node/conf/patterndb.xml -P sfims -M "[Primary Detection Engine 
(f3fe03e8-8a2f-11e3-97cb-abab35940b61)][PASSIVE_CATCH_ALL][1:2667:14] 
"SERVER-IIS ping.asp access" [Classification: Access to a Potentially 
Vulnerable Web Application] User: Unknown, Application: Unknown, Client: 
Unknown, App Protocol: UnknownInterface Ingress: s1p2, Interface Egress: s1p1, 
Security Zone Ingress: Passive, Security Zone Egress: Passive, [Priority: 2] 
{TCP} 172.17.1.13:54486 -> 172.17.5.6:8080"
MESSAGE=[Primary Detection Engine 
(f3fe03e8-8a2f-11e3-97cb-abab35940b61)][PASSIVE_CATCH_ALL][1:2667:14] SERVER-IIS
PROGRAM=sfims
.classifier.class=unknown
TAGS=.classifier.unknown

root@ELSA-SRV:/opt/elsa/node/conf# /usr/bin/pdbtool match -p 
/opt/elsa/node/conf/patterndb.xml -P sfims -M "[Primary Detection Engine 
(f3fe03e8-8a2f-11e3-97cb-abab35940b61)][PASSIVE_CATCH_ALL][1:2667:14] 
SERVER-IIS ping.asp access [Classification: Access to a Potentially Vulnerable 
Web Application] User: Unknown, Application: Unknown, Client: Unknown, App 
Protocol: UnknownInterface Ingress: s1p2, Interface Egress: s1p1, Security Zone 
Ingress: Passive, Security Zone Egress: Passive, [Priority: 2] {TCP} 
172.17.1.13:54486 -> 172.17.5.6:8080"
MESSAGE=[Primary Detection Engine 
(f3fe03e8-8a2f-11e3-97cb-abab35940b61)][PASSIVE_CATCH_ALL][1:2667:14] 
SERVER-IIS ping.asp access [Classification: Access to a Potentially Vulnerable 
Web Application] User: Unknown, Application: Unknown, Client: Unknown, App 
Protocol: UnknownInterface Ingress: s1p2, Interface Egress: s1p1, Security Zone 
Ingress: Passive, Security Zone Egress: Passive, [Priority: 2] {TCP} 
172.17.1.13:54486 -> 172.17.5.6:8080
PROGRAM=sfims
.classifier.class=10000
.classifier.rule_id=10000
i0=2
s0=TCP
i1=172.17.1.13
i2=54486
i3=172.17.5.6
i4=8080
TAGS=.classifier.10000

So my question is two part:-
1) Is ELSA capable of parsing messages containing double quotes and is this 
issue therefore just an issue with the pdbtool?
2) If the answer to 1) is YES then is there anything potentially wrong with the 
database entries or the pattern(s) defined - because although the SFIMS class 
is now available through ELSA search, queries do not yield any results?

Thanks,
Matt

Original issue reported on code.google.com by matt.jo...@e2e-agile.com on 28 Mar 2014 at 1:03

[GoogleCodeExporter]

You are surrounding double quotes with double quotes.  Not gonna work.  Try 
replacing the outer doubles surrounding the whole message with single quotes:

$ /usr/local/syslog-ng/bin/pdbtool match -p patterndb.xml -P sfims -M '[Primary 
Detection Engine 
(f3fe03e8-8a2f-11e3-97cb-abab35940b61)][PASSIVE_CATCH_ALL][1:2667:14] 
"SERVER-IIS ping.asp access" [Classification: Access to a Potentially 
Vulnerable Web Application] User: Unknown, Application: Unknown, Client: 
Unknown, App Protocol: UnknownInterface Ingress: s1p2, Interface Egress: s1p1, 
Security Zone Ingress: Passive, Security Zone Egress: Passive, [Priority: 2] 
{TCP} 172.17.1.13:54486 -> 172.17.5.6:8080'
MESSAGE=[Primary Detection Engine 
(f3fe03e8-8a2f-11e3-97cb-abab35940b61)][PASSIVE_CATCH_ALL][1:2667:14] 
"SERVER-IIS ping.asp access" [Classification: Access to a Potentially 
Vulnerable Web Application] User: Unknown, Application: Unknown, Client: 
Unknown, App Protocol: UnknownInterface Ingress: s1p2, Interface Egress: s1p1, 
Security Zone Ingress: Passive, Security Zone Egress: Passive, [Priority: 2] 
{TCP} 172.17.1.13:54486 -> 172.17.5.6:8080
PROGRAM=sfims
.classifier.class=10000
.classifier.rule_id=10000
i0=2
s0=TCP
i1=172.17.1.13
i2=54486
i3=172.17.5.6
i4=8080
TAGS=.classifier.10000

Original comment by kebut...@gmail.com on 29 Mar 2014 at 3:46

[GoogleCodeExporter]

Thank you - encapsulating the message in single quotes makes the match work 
when using pdbtool but ELSA still doesn't appear to be actually parsing any 
SFIMS messages at all (GUI search for CLASS=SFIMS returns nothing and there are 
no entries in the DB).

Have I got something wrong elsewhere? I've checked the field/classes mappings 
several times and we know that the pattern match is correct since it works with 
the pbdtool match so I'm still pretty stumped!

Thanks,
Matt

Original comment by matt.jo...@e2e-agile.com on 31 Mar 2014 at 7:29

[GoogleCodeExporter]

Given your installation path, It would need to be in 
/opt/elsa/node/conf/merged.xml
During an install or upgrade, /opt/elsa/node/conf/patterndb.xml is copied to 
/etc/elsa/patterns.d/patterndb.xml and then merged with 
/etc/elsa/patterns.d/local_patterndb.xml via the command:
$BASE_DIR/syslog-ng/bin/pdbtool merge -p $BASE_DIR/elsa/node/conf/merged.xml -r 
-D /etc/elsa/patterns.d

You can test your pattern by putting it in merged.xml directly, but to maintain 
you customization across upgrades, place it in local_patterndb.xml before 
performing an install.sh based upgrade.

Original comment by kebut...@gmail.com on 31 Mar 2014 at 11:54

[GoogleCodeExporter]

Thanks again. We don't have a merged.xml in /opt/elsa/node/conf, nor do we have 
a patterndb.xml or local_patterndb.xml under /etc/elsa/patterns.d/

Are you saying that the patterns defined in /opt/elsa/node/conf/patterndb.xml 
don't take effect until either an upgrade takes place or the merge command is 
run with pdbtool and they are subsequently copied into merged.xml?

If so, I could copy my patterndb.xml file into /etc/elsa/patterns.d then run 
$BASE_DIR/syslog-ng/bin/pdbtool merge -p $BASE_DIR/elsa/node/conf/merged.xml -r 
-D /etc/elsa/patterns.d should I? Or do I also need to create a 
local_patterndb.xml with which to merge it first? This server is currently 
being used within an operational environment - could service be unaffected by 
doing any of this?

Thanks,
Matt

Original comment by matt.jo...@e2e-agile.com on 31 Mar 2014 at 12:59

[GoogleCodeExporter]

Are you using Security Onion?  Your setup differs from a default install, so I 
wouldn't do any of that yet.

Original comment by kebut...@gmail.com on 31 Mar 2014 at 1:48

[GoogleCodeExporter]

We are, yes - apologies for that omission!

Original comment by matt.jo...@e2e-agile.com on 31 Mar 2014 at 1:54

[GoogleCodeExporter]

it looks like you have the right file.  I fired up my onion and it looks like 
the patterndb file you are modifying is the correct file.  Perhaps you need to 
give a hint as to the program where the logs are originating from.  For 
example, my bro_conn log is read via syslog-ng, and I specify a 
program_override so elsa knows to use the bro_conn pattern to match the log 
lines.
source s_bro_conn { file("/data/bro/logs/current/conn.log" flags(no-parse) 
program_override("bro_conn")); };

Original comment by kebut...@gmail.com on 31 Mar 2014 at 3:36

[GoogleCodeExporter]

Thank you, I haven't been able to do anything further on it over the last few 
weeks but I will give that a try when I can get back to it! Thanks for your help

Original comment by matt.jo...@e2e-agile.com on 13 May 2014 at 10:47