zentures / sequence

(Unmaintained) High performance sequential log analyzer and parser
http://sequencer.io
517 stars 72 forks source link

relaxing types #7

Closed cryptix closed 9 years ago

cryptix commented 9 years ago

Hi,

i'm trying to write rules for postfix, most of them contain a hexadecimal string, like a message ID.

Feb  4 02:01:15 mail postfix/oqmgr[86819]: BF468251C: removed
Feb  4 02:01:17 mail postfix/oqmgr[86819]: 746702526: removed
Feb  4 02:33:33 mail postfix/oqmgr[86819]: 24CB02536: removed
Feb  4 04:01:55 mail postfix/oqmgr[86819]: EC3DE2562: removed

The probleme here for me is that, in some cases these IDs are all digits and than sequence want's it to be an %integer% instead, forcing me to duplicate my rules.

In another instances of this, when it logs delays like this:

delay=0.4
delay=1

the numbers want to be %float% and %integer%.

With each such case the number of rules explodes. I'm asking if it wouldn't make more sense to let later processing deal with the validity of single fields. Maybe collapse float and int into a single number type? Or have maybe have another %literal% that ignores the type.

kind regards,

cryptix

zhenjl commented 9 years ago

That makes sense. Let me think about this and see what the right solution is.

Do you have any sample logs you can share, even if just a few lines on the delays?

thx

Jian

cryptix commented 9 years ago

No problem:

Feb  8 12:15:52 mail postfix/pipe[76139]: 499F62D65: to=<userA@company.office>, orig_to=<alias24@alias.com>, relay=dovecot, delay=0.24, delays=0.21/0/0/0.04, dsn=2.0.0, status=sent (delivered via dovecot service)
Feb  8 13:06:55 mail postfix/pipe[76850]: 7CD542D74: to=<userA@company.office>, orig_to=<userA@company.eu>, relay=dovecot, delay=0.3, delays=0.26/0/0/0.04, dsn=2.0.0, status=sent (delivered via dovecot service)
Feb  8 14:36:54 mail postfix/pipe[78111]: C21CC2D9B: to=<userB@company.office>, orig_to=<info@company.biz>, relay=dovecot, delay=1, delays=0.99/0/0/0.02, dsn=2.0.0, status=sent (delivered via dovecot service)
Feb  8 17:54:45 mail postfix/pipe[80790]: 3459A2DA6: to=<userA@company.office>, orig_to=<alias2@alias.com>, relay=dovecot, delay=0.19, delays=0.16/0/0/0.02, dsn=2.0.0, status=sent (delivered via dovecot service)
Feb  8 20:46:25 mail postfix/pipe[83144]: 6549C2DCE: to=<userA@company.office>, orig_to=<alias1@company.eu>, relay=dovecot, delay=0.15, delays=0.13/0/0/0.02, dsn=2.0.0, status=sent (delivered via dovecot service)
Feb  8 21:51:10 mail postfix/pipe[84059]: 440682230: to=<userB@company.office>, orig_to=<userB@company.biz>, relay=dovecot, delay=0.9, delays=0.87/0/0/0.03, dsn=2.0.0, status=sent (delivered via dovecot service)
Feb  8 21:51:37 mail postfix/pipe[84059]: 47FEE2DE5: to=<userB@company.office>, orig_to=<userB@company.eu>, relay=dovecot, delay=0.83, delays=0.81/0/0/0.02, dsn=2.0.0, status=sent (delivered via dovecot service)
Feb  8 23:49:58 mail postfix/pipe[85979]: B9E532E0B: to=<userB@company.office>, orig_to=<userB@company.eu>, relay=dovecot, delay=0.19, delays=0.16/0/0/0.03, dsn=2.0.0, status=sent (delivered via dovecot service)

i also noticed that the dns=... is falsely recognized as an ip address. should I open a speerate issue for that?

zhenjl commented 9 years ago

@cryptix now you can specify the type in the rule like this

"%msgtime% %apphost% %appname% [ %sessionid% ] : %msgid:integer% : to = < %srcemail% > , orig_to = < %string% > , relay = %string% , delay = %float% , delays = %string% , dsn = %string% , status = %status% ( %reason::+% )",

Notice %msgid:integer%, you can also specify %msgid:string%. So to match your examples, you can create two rules. One with integer, the other with string.

Also notice the last part of the rule, which is ( %reason::+% ). It means consume one or more reason tokens inside ( and ). You can actually just do %reason:+% as well.

The fields are now specified in cmd/sequence/sequence.toml. You can add your own fields of the format field:type where field is the field name, and type is the default type of the field. The default type is used when the field type is not specified in the rule.

I also fixed the DNS float issue.

Please let me know you would be able to test this out.

thx

Jian