search

djebbibmw / enterprise-log-search-and-archive

Automatically exported from code.google.com/p/enterprise-log-search-and-archive
0 stars 0 forks source link

Need plugin for Fortinet FortiGate logs #5

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
I have a number of Fortinet FortiGate firewalls and would like to use this 
system with their syslog output.

I tried to look at the patterndb.xml and was confused as to what needed to be 
there to make this work.

Here is a couplf of the log entries that need to be parsed:

Feb 10 11:27:01 logsource kernel: date=2012-02-10 time=11:27:01 
devname=CUSTID01-SITEID-FW device_id=FG100C999999999 log_id=13312 
subtype=ftgd_allow type=webfilter pri=notice vd=VDOM policyid=44 identidx=1 
serial=369298248 user=USER group=AD/GROUP src=10.1.2.3 sport=2163 src_port=2163 
src_int=INT dst=4.3.2.1 dport=80 dst_port=80 dst_int=WAN service=http 
hostname=col.stb.s-msn.com profiletype=Webfilter_Profile profile=PROFILE 
status=passthrough req_type=referral 
url=/i/79/65F987C952BDA0E84AE52464ADD59.jpg method=domain class=0 cat=41 
cat_desc="Search Engines and Portals" carrier_ep=N/A msg="URL belongs to an 
allowed category in policy" class_desc=N/A profilegroup=N/A

Feb 10 11:27:01 logsource kernel: date=2012-02-10 time=11:27:01 
devname=CUSTID01-SITEID-FW device_id=FGT80C9999999999 log_id=2 subtype=allowed 
type=traffic pri=notice vd=VDOM dir_disp=org tran_disp=snat src=10.1.2.3 
srcname=10.1.2.3 src_port=53624 dst=4.3.2.2 dstname=4.3.2.2 dst_port=80 
tran_ip=5.4.3.2 tran_port=49648 service=80/tcp proto=6 app_type=N/A 
duration=120 rule=49 policyid=49 identidx=0 sent=1221 rcvd=2062 
shaper_drop_sent=0 shaper_drop_rcvd=0 perip_drop=0 sent_pkt=7 rcvd_pkt=6 
src_int=INT dst_int=WAN SN=16349534 app=N/A app_cat=N/A carrier_ep=N/A vpn=N/A 
status=accept user=N/A group=N/A shaper_sent_name=N/A shaper_rcvd_name=N/A 
perip_name=N/A

If I could get one or more pattern examples, I could work on others.  The 
fields and properties are not yet fully documented so am not sure of a starting 
point.

Original issue reported on code.google.com by edavi...@gmail.com on 10 Feb 2012 at 5:32

GoogleCodeExporter commented 9 years ago 
Here are the patterndb.xml entries which will parse fields from your log 
entries, assuming that the field order is always the same:

<ruleset name="fortinet_url" id='21'>
        <pattern>kernel</pattern>
        <rules>
            <rule provider="ELSA" class='21' id='21'>
                <patterns>
                    <pattern>date=@ESTRING:: @time=@ESTRING:: @devname=@ESTRING:: @device_id=@ESTRING:: @log_id=@ESTRING:: @subtype=@ESTRING:: @type=webfilter pri=@ESTRING:: @vd=@ESTRING:: @policyid=@ESTRING:: @identidx=@ESTRING:: @serial=@ESTRING:: @user=@ESTRING:s0: @group=@ESTRING:s1: @src=@IPv4:i0:@ sport=@ESTRING:i1: @src_port=@ESTRING:: @src_int=@ESTRING:: @dst=@IPv4:i2:@ dport=@ESTRING:i3: @dst_port=@ESTRING:: @dst_int=@ESTRING:: @service=@ESTRING:s2: @hostname=@ESTRING:s3: @profiletype=@ESTRING:: @profile=@ESTRING:: @status=@ESTRING:s4: @req_type=@ESTRING:: @url=@ESTRING:s5: @method=@ESTRING:: @class=@ESTRING:: @cat=@ESTRING:i4: @cat_desc=@QSTRING::""@ carrier_ep=@ESTRING:: @msg=@QSTRING::""@ class_desc=@ESTRING:: @profilegroup=</pattern>
                </patterns>
                <examples>
                    <example>
                        <test_message program="kernel">date=2012-02-10 time=11:27:01 devname=CUSTID01-SITEID-FW device_id=FG100C999999999 log_id=13312 subtype=ftgd_allow type=webfilter pri=notice vd=VDOM policyid=44 identidx=1 serial=369298248 user=USER group=AD/GROUP src=10.1.2.3 sport=2163 src_port=2163 src_int=INT dst=4.3.2.1 dport=80 dst_port=80 dst_int=WAN service=http hostname=col.stb.s-msn.com profiletype=Webfilter_Profile profile=PROFILE status=passthrough req_type=referral url=/i/79/65F987C952BDA0E84AE52464ADD59.jpg method=domain class=0 cat=41 cat_desc="Search Engines and Portals" carrier_ep=N/A msg="URL belongs to an allowed category in policy" class_desc=N/A profilegroup=N/A</test_message>
                        <test_values>
                            <test_value name="i0">10.1.2.3</test_value>
                            <test_value name="i1">2163</test_value>
                            <test_value name="i2">4.3.2.1</test_value>
                            <test_value name="i3">80</test_value>
                            <test_value name="s0">USER</test_value>
                            <test_value name="s1">AD/GROUP</test_value>
                            <test_value name="s2">http</test_value>
                            <test_value name="s3">col.stb.s-msn.com</test_value>
                            <test_value name="s4">passthrough</test_value>
                            <test_value name="s5">/i/79/65F987C952BDA0E84AE52464ADD59.jpg</test_value>
                            <test_value name="i4">41</test_value>
                        </test_values>
                    </example>
                </examples>
            </rule>
        </rules>
    </ruleset>
    <ruleset name="fortinet_traffic" id='22'>
        <pattern>kernel</pattern>
        <rules>
            <rule provider="ELSA" class='22' id='22'>
                <patterns>
                    <pattern>date=@ESTRING:: @time=@ESTRING:: @devname=@ESTRING:: @device_id=@ESTRING:: @log_id=@ESTRING:: @subtype=@ESTRING:: @type=traffic pri=@ESTRING:: @vd=@ESTRING:: @dir_disp=@ESTRING:: @tran_disp=@ESTRING:: @src=@IPv4:i0:@ srcname=@ESTRING:: @src_port=@NUMBER:i1:@ dst=@IPv4:i2:@ dstname=@ESTRING:: @dst_port=@NUMBER:i3:@ tran_ip=@ESTRING:: @tran_port=@ESTRING:: @service=@ESTRING:: @proto=@NUMBER:i4@ app_type=@ESTRING:: @duration=@NUMBER:i5@ rule=@ESTRING:: @policyid=@ESTRING:: @identidx=@ESTRING:: @sent=@ESTRING:: @rcvd=@ESTRING:: @shaper_drop_sent=@ESTRING:: @shaper_drop_rcvd=@ESTRING:: @perip_drop=@ESTRING:: @sent_pkt=@ESTRING:: @rcvd_pkt=@ESTRING:: @src_int=@ESTRING:: @dst_int=@ESTRING:: @SN=@ESTRING:: @app=@ESTRING:: @app_cat=@ESTRING:: @carrier_ep=@ESTRING:: @vpn=@ESTRING:: @status=@ESTRING:: @user=@ESTRING:: @group=@ESTRING:: @shaper_sent_name=@ESTRING:: @shaper_rcvd_name=@ESTRING:: @perip_name</pattern>
                </patterns>
                <examples>
                    <example>
                        <test_message program="kernel">date=2012-02-10 time=11:27:01 devname=CUSTID01-SITEID-FW device_id=FGT80C9999999999 log_id=2 subtype=allowed type=traffic pri=notice vd=VDOM dir_disp=org tran_disp=snat src=10.1.2.3 srcname=10.1.2.3 src_port=53624 dst=4.3.2.2 dstname=4.3.2.2 dst_port=80 tran_ip=5.4.3.2 tran_port=49648 service=80/tcp proto=6 app_type=N/A duration=120 rule=49 policyid=49 identidx=0 sent=1221 rcvd=2062 shaper_drop_sent=0 shaper_drop_rcvd=0 perip_drop=0 sent_pkt=7 rcvd_pkt=6 src_int=INT dst_int=WAN SN=16349534 app=N/A app_cat=N/A carrier_ep=N/A vpn=N/A status=accept user=N/A group=N/A shaper_sent_name=N/A shaper_rcvd_name=N/A perip_name=N/A</test_message>
                        <test_values>
                            <test_value name="i0">10.1.2.3</test_value>
                            <test_value name="i1">53624</test_value>
                            <test_value name="i2">4.3.2.2</test_value>
                            <test_value name="i3">80</test_value>
                            <test_value name="i4">6</test_value>
                            <test_value name="i5">120</test_value>
                        </test_values>
                    </example>
                </examples>
            </rule>
        </rules>
    </ruleset>

I will add it to the distributed patterndb.xml, so it should be ready to 
download from SVN.

Before updating the patterndb.xml, you will need to add classes to the database 
on each node.  I have included this code in the schema.sql, but it is commented 
out so as not to clutter the config for those who don't use Fortinet.  

First, we add the class:
INSERT INTO classes (id, class, parent_id) VALUES(21, "FORTINET_URL", 0);
INSERT INTO classes (id, class, parent_id) VALUES(22, "FORTINET_TRAFFIC", 0);

Then we add the fields not already present (this is also in the schema.sql now):
INSERT INTO fields (field, field_type, pattern_type) VALUES ("group", "string", 
"QSTRING");
INSERT INTO fields (field, field_type, pattern_type) VALUES ("status", 
"string", "QSTRING");

Then we map the fields to the class and the "i0/s0" column names by field_order 
(in the schema, but commented out);

INSERT INTO fields_classes_map (class_id, field_id, field_order) VALUES 
((SELECT id FROM classes WHERE class="FORTINET_URL"), (SELECT id FROM fields 
WHERE field="srcip"), 5);
INSERT INTO fields_classes_map (class_id, field_id, field_order) VALUES 
((SELECT id FROM classes WHERE class="FORTINET_URL"), (SELECT id FROM fields 
WHERE field="srcport"), 6);
INSERT INTO fields_classes_map (class_id, field_id, field_order) VALUES 
((SELECT id FROM classes WHERE class="FORTINET_URL"), (SELECT id FROM fields 
WHERE field="dstip"), 7);
INSERT INTO fields_classes_map (class_id, field_id, field_order) VALUES 
((SELECT id FROM classes WHERE class="FORTINET_URL"), (SELECT id FROM fields 
WHERE field="dstport"), 8);
INSERT INTO fields_classes_map (class_id, field_id, field_order) VALUES 
((SELECT id FROM classes WHERE class="FORTINET_URL"), (SELECT id FROM fields 
WHERE field="user"), 11);
INSERT INTO fields_classes_map (class_id, field_id, field_order) VALUES 
((SELECT id FROM classes WHERE class="FORTINET_URL"), (SELECT id FROM fields 
WHERE field="group"), 12);
INSERT INTO fields_classes_map (class_id, field_id, field_order) VALUES 
((SELECT id FROM classes WHERE class="FORTINET_URL"), (SELECT id FROM fields 
WHERE field="service"), 13);
INSERT INTO fields_classes_map (class_id, field_id, field_order) VALUES 
((SELECT id FROM classes WHERE class="FORTINET_URL"), (SELECT id FROM fields 
WHERE field="site"), 14);
INSERT INTO fields_classes_map (class_id, field_id, field_order) VALUES 
((SELECT id FROM classes WHERE class="FORTINET_URL"), (SELECT id FROM fields 
WHERE field="status"), 15);
INSERT INTO fields_classes_map (class_id, field_id, field_order) VALUES 
((SELECT id FROM classes WHERE class="FORTINET_URL"), (SELECT id FROM fields 
WHERE field="uri"), 16);

INSERT INTO fields_classes_map (class_id, field_id, field_order) VALUES 
((SELECT id FROM classes WHERE class="FORTINET_TRAFFIC"), (SELECT id FROM 
fields WHERE field="srcip"), 5);
INSERT INTO fields_classes_map (class_id, field_id, field_order) VALUES 
((SELECT id FROM classes WHERE class="FORTINET_TRAFFIC"), (SELECT id FROM 
fields WHERE field="srcport"), 6);
INSERT INTO fields_classes_map (class_id, field_id, field_order) VALUES 
((SELECT id FROM classes WHERE class="FORTINET_TRAFFIC"), (SELECT id FROM 
fields WHERE field="dstip"), 7);
INSERT INTO fields_classes_map (class_id, field_id, field_order) VALUES 
((SELECT id FROM classes WHERE class="FORTINET_TRAFFIC"), (SELECT id FROM 
fields WHERE field="dstport"), 8);
INSERT INTO fields_classes_map (class_id, field_id, field_order) VALUES 
((SELECT id FROM classes WHERE class="FORTINET_TRAFFIC"), (SELECT id FROM 
fields WHERE field="proto"), 9);
INSERT INTO fields_classes_map (class_id, field_id, field_order) VALUES 
((SELECT id FROM classes WHERE class="FORTINET_TRAFFIC"), (SELECT id FROM 
fields WHERE field="conn_duration"), 10);

Restart syslog-ng, update patterndb.xml, and your logs should now be parsed in 
the correct fields.  They should show up in the "Add Term" menu on the web 
interface.

Original comment by mchol...@gmail.com on 10 Feb 2012 at 9:05

GoogleCodeExporter commented 9 years ago 
Thanks!  As soon as I resolve my other issue, I will let you know how this 
works out.

Original comment by edavi...@gmail.com on 10 Feb 2012 at 11:05

GoogleCodeExporter commented 9 years ago 
I have this somewhat working and the FORTINET_URL filter produces results but I 
see nothing for the FORTINET_TRAFFIC class.

I do get an error when starting syslog-ng though:

Starting syslog-ng
Unknown parser type specified; type='subtype='

I checked and both FORTINET_URL and FORTINET_TRAFFIC have @subtype=@ESTRING:: 
so do not know which one it does not like.

Original comment by edavi...@gmail.com on 22 Feb 2012 at 11:40

GoogleCodeExporter commented 9 years ago 
Also, what if the field order changes slightly?  Does it completely break the 
pattern recognition?

Original comment by edavi...@gmail.com on 22 Feb 2012 at 11:41

GoogleCodeExporter commented 9 years ago 
Hm, I can't reproduce the issue you're seeing.  Are you sure there are no line 
breaks in the patterndb.xml file and that it is exactly as it is in the 
repository?

If the field order changes, the pattern will probably stop working.  At best, 
the wrong fields will be extracted.  If you don't care about fields, then there 
are ways to make the parser more resilient to changing patterns.

Original comment by mchol...@gmail.com on 23 Feb 2012 at 2:03

GoogleCodeExporter commented 9 years ago 
I just did a update-from-cvs.sh and now when I restart syslog-ng I do not get 
the error.

It seems that different versions of firmware have the field order slightly 
different so we may miss on a few log matching.  I will check the logs in a few 
minutes after the restart to see if both of these are now matching like they 
should.

Original comment by edavi...@gmail.com on 23 Feb 2012 at 4:10

GoogleCodeExporter commented 9 years ago 
Ok, let me know if it changes as there are lots of ways we can account for it, 
but I'll need specific examples.

Original comment by mchol...@gmail.com on 23 Feb 2012 at 4:28

GoogleCodeExporter commented 9 years ago 
I have FORTINET_TRAFFIC and FORTINET_URL now after the update from CVS.  I will 
watch for mismatched log entries and let you know.

Original comment by edavi...@gmail.com on 23 Feb 2012 at 4:54

GoogleCodeExporter commented 9 years ago 
Hi, i have the changes in the database and patterndb.xml but the logs of my 
fortinet is not parsed, when i try the test with pdbtool i receive this message:

PROGRAM=fortinet_traffic
.classifier.class=unknown
TAGS=.classifier.unknown

the pattern is:

<pattern>fortinet_traffic</pattern>
                <rules>
                        <rule class='22' id='22'>
                                <patterns>
                                        <pattern>time=@ESTRING:: @devname=@ESTRING:: @device_id=@ESTRING:: @log_id=@ESTRING:: @type=traffic subtype=@ESTRING:: @pri=@ESTRING:: @vd=@ESTRING:: @src=@IPv4:i0:@ src_port=@NUMBER:i1:@ src_int=@QSTRING::""@ dst=@IPv4:i2:@ dst_port=@NUMBER:i3:@ dst_int=@QSTRING:: @SN=@ESTRING:: @status=@ESTRING:: @policyid=@ESTRING:: @dst_country=@QSTRING::""@ src_country=@QSTRING::"" @service=@ESTRING:: @proto=@NUMBER:i4:@ duration=@NUMBER:i5: @sent=@ESTRING:: @rcvd=@ESTRING:: @msg</pattern>

This is the test:

pdbtool match -p /opt/elsa/node/conf/patterndb.xml -P fortinet_traffic -M 
"10:42:43 devname=FORTIVM02 device_id=FGVM01XXXXXXX log_id=0038000007 
type=traffic subtype=other pri=warning vd=X_VDOM src=1.1.1.1 src_port=1037 
src_int="X_MPLS" dst=2.2.2.2 dst_port=514 dst_int="X_LAN" SN=XXXXXX status=deny 
policyid=0 dst_country="Reserved" src_country="Reserved" service=SYSLOG 
proto=17 duration=0 sent=0 rcvd=0 msg="Denied by forward policy check""

This is the result:

PROGRAM=fortinet_traffic
.classifier.class=unknown
TAGS=.classifier.unknown

Any ideas?

Original comment by danielro...@gmail.com on 28 Aug 2013 at 7:26

GoogleCodeExporter commented 9 years ago 
Your messages are coming in without a program tag on them, so the first
part of the date is getting interpreted as the program name.  You need to
change your syslog-ng config to add a special source for your Fortinet
firewalls and set the flags(no-parse)
and program_override("fortinet_traffic") setting for them.

On Wed, Aug 28, 2013 at 2:27 PM, <
enterprise-log-search-and-archive@googlecode.com> wrote:

Original comment by mchol...@gmail.com on 2 Sep 2013 at 5:46