apache2_error plugin not parsing all lines/files

What steps will reproduce the problem?
1. Not all lines within the error.log file are parsed.

a) Have a line within the Apache error logs that doesn't meet the standard 
format (which could be produced from a hacked web server being issued commands 
via a web shell etc.), i.e.:
"cat: /etc/passwrd: No such file or directory"
"find: `../etc/shadow': Permission denied"

2. Not all logs are parsed depending on their order and content & some 
scenarios don't produce output stating the logs weren't parsed successfully.

a) If the first error.log file parsed contains lines which don't adhere to the 
standard format, the program will exit & not continue to try any additional log 
files if multiple were included.  
- error message to STDOUT but doesn't say in what file and also doesn't say it 
didn't parse the rest of the files.

b) If the first error.log file parsed contains lines which first adhere to the 
standard format & then ones that don't, the good lines are parsed and then 
program exists once it hits the bad lines & won't continue to parse the other 
files if multiple were selected.
- error message to STDOUT says lines couldn't be parsed but doesn't say in what 
file and also doesn't say it didn't parse the rest of the files.

c) If the first error.log file parsed contains lines which all adhere to the 
standard format followed by other files which contain mixed lines/lines that 
don't adhere then the first (good) error.log file is parsed successfully but 
the other files aren't.  
- there is no error message to STDOUT stating there were any errors or files 
not not parsed.

What is the expected output? What do you see instead?
1) To parse all lines contained within an error.log file
2) If a file fails to be parsed, the rest of the files should still be tried.

- instead it will error out -or- just not parse at all depending on the scenario

What version of the product are you using? On what operating system?
- v0.63 on SIFT v2.13

Proposed Solution (but wanted community feedback)
1) Either the first line in the error log has to fit the standard format or one 
out of the first x lines (right now it's set to w/in the first 15);  If not, 
spit out an error stating that particular file couldn't be parsed & continue 
onto the next log file if there are multiple since the next one may have all 
valid entries.

2) As long as at least one line is found to meet the standard format, once a 
line isn't found to meet the standard format after that (i.e. doesn't start 
with [DOW month …]) then copy that information from the line before it (with 
the valid format/timestamp) and add it to the beginning so it meets the format 
and can be put into the timeline of events.

... sample logs are attached.  The three scenarios described in this ticket can 
be reproduced with these test samples, just change their names so on each test 
another one is parsed first.
Original issue reported on code.google.com by hiddenil...@gmail.com on 7 May 2012 at 11:31
Attachments:
gudcjfdldu / log2timeline

apache2_error plugin not parsing all lines/files #4
