split filter using a newline delimiter

[jwestbrook]

Tried to use a \n as a delimiter like this

filter://split://?delimiter=\n

the split filter didn't like that when getting indexOf(this.delimiter), if I change it to indexOf('\n') the newline split works correctly

[bpaquet]

Hi,

Can you encode \n to %0A and retry ?

Bertrand

On Wed, Jun 25, 2014 at 1:01 AM, Jason Westbrook notifications@github.com wrote:

Tried to use a \n as a delimiter like this

filter://split://?delimiter=\n

the split filter didn't like that when getting indexOf(this.delimiter), if I change it to indexOf('\n') the newline split works correctly

— Reply to this email directly or view it on GitHub https://github.com/bpaquet/node-logstash/issues/68.

[jwestbrook]

OK that seems to be working - but its catching random new lines within some log entries and breaking on those as well.

I did some digging and it looks like if you use the Regex multiline flag "m" then loop on exec() you can get all matches within a block of text.

so inside FilterRegex.prototype.process using the multiline flag you can get multiple log entries from one block of text. However the return of process()looks like an object and I dont think whatever is calling that method would expect an array of separate log entries

example text

<37>Jun 30 09:31:09 www0 su: (to root) jwestbrook on pts/0\n<86>Jun 30 09:31:09 www0 su: pam_unix(su-l:session): session opened for user root by jwestbrook(uid=0)

javascript

var reg = new Regexp("^[0-9]*\\s*<(\\S+)>\\S.([0-9]{4}-[0-9]{2}-[0-9]{2}T\\d+:\\d+:\\d+)\\S+\\s+(\\S+) (\\S+) (\\d+) - (-|\\[.+\\]) (.+)$","m");

var match;
while(match = req.exec(input))
{
   //populate an array with objects from the match
}

//return array of matches found in the block of text
return matches;

[bpaquet]

I do not understand the problem. The split filter does not use regex for spliting lines, but a simple indexOf.

Do you need a regex to split lines ?

[bpaquet]

So I think there is a confusion :

• if you want a split a single line of input into multiple, use the split filter (eg you line with \n and session opened)
• if you want to reassemble multiple lines into a single, use the multiline filter (eg a java stack trace).