search

Graylog2 / graylog2-server

Free and open log management
https://www.graylog.org
Other
7.33k stars 1.05k forks source link

Graylog GROK patterns buffer overflow #5155

Open james9182 opened 5 years ago

james9182 commented 5 years ago

Graylog GROK patterns buffer overflow

Expected Behavior

We have bad messages(corrupted) from nginx 1.14 error_log size 2048 byte with GROK pattern nginx:\s(?%{YEAR}[./-]%{MONTHNUM}[./-]%{MONTHDAY}[- ]%{TIME}) [%{LOGLEVEL:severity}] %{POSINT:pid222}#%{NUMBER}: (?(.|\r|\n)*)(?:, client: %{IPORHOST:remote_addr})(?:, server: %{IPORHOSTORUNDERSCORE:server})(?:, request: %{QS:request})?(?:, upstream: “%{URI:upstream}”)?(?:, host: %{QS:host})?(?:, referrer: “%{URI:http_referer}”)?$ 

<187>Sep 24 16:10:46 jamingo.xyz.org nginx: 2018/09/24 16:10:46 [error] 94210#94210: *768531 FastCGI sent in stderr: "-web/releases/jamingo-andy-bli-opacweb-20180922174622/cacadenol/modules/database/classes/database/mysqli.php on line 52
PHP message: PHP Warning mysqli_connect(): (HY000/2002): Connection refused in file /var/www/jamingo-andy-bli-opacweb/releases/jamingo-andy-bli-opacweb-20180922174622/cacadenol/modules/database/classes/database/mysqli.php on line 52
PHP message: PHP Warning mysqli_connect(): (HY000/2002): Connection refused in file /var/www/jamingo-andy-bli-opacweb/releases/jamingo-andy-bli-opacweb-20180922174622/cacadenol/modules/database/classes/database/mysqli.php on line 52
PHP message: PHP Warning mysqli_connect(): (HY000/2002): Connection refused in file /var/www/jamingo-andy-bli-opacweb/releases/jamingo-andy-bli-opacweb-20180922174622/cacadenol/modules/database/classes/database/mysqli.php on line 52
PHP message: PHP Warning mysqli_connect(): (HY000/2002): Connection refused in file /var/www/jamingo-andy-bli-opacweb/releases/jamingo-andy-bli-opacweb-20180922174622/cacadenol/modules/database/classes/database/mysqli.php on line 52
PHP message: PHP Warning mysqli_connect(): (HY000/2002): Connection refused in file /var/www/jamingo-andy-bli-opacweb/releases/jamingo-andy-bli-opacweb-20180922174622/cacadenol/modules/database/classes/database/mysqli.php on line 52
PHP message: PHP Warning mysqli_connect(): (HY000/2002): Connection refused in file /var/www/jamingo-andy-bli-opacweb/releases/jamingo-andy-bli-opacweb-20180922174622/cacadenol/modules/database/classes/database/mysqli.php on line 52
PHP message: PHP Warning mysqli_connect(): (HY000/2002): Connection refused in file /var/www/jamingo-andy-bli-opacweb/releases/jamingo-andy-bli-opacweb-20180922174622/cacadenol/modules/database/classes/database/mysqli.php on line 52
PHP message: PHP Warning mysqli_connect(): (HY000/2002): Connection refused in file /var/www/jamingo-andy-bli-opacweb/releases/jamingo-andy-bli-opacweb-20180922174622/caca

this message is parsed very well without errors. but if we send a message from nginx error log with a length of 4096 bytes. this message is not processed, and there are too many errors in the logs graylog

2018-09-25T16:42:20.260+03:00 WARN  [ProcessBuffer] Unable to process event MessageEvent{raw=null, message=null, messages=null}, sequence 285361587                                                        
java.lang.StackOverflowError: null                                                                                                                                                                         
        at java.util.regex.Pattern$GroupHead.match(Pattern.java:4660) ~[?:1.8.0_181]                                                                                                                       
        at java.util.regex.Pattern$Loop.match(Pattern.java:4787) ~[?:1.8.0_181]                                                                                                                            
        at java.util.regex.Pattern$GroupTail.match(Pattern.java:4719) ~[?:1.8.0_181]                                                                                                                       
        at java.util.regex.Pattern$BranchConn.match(Pattern.java:4570) ~[?:1.8.0_181]                                                                                                                      
        at java.util.regex.Pattern$CharProperty.match(Pattern.java:3779) ~[?:1.8.0_181]                                                                                                                    
        at java.util.regex.Pattern$Branch.match(Pattern.java:4606) ~[?:1.8.0_181]                                                                                                                          
        at java.util.regex.Pattern$GroupHead.match(Pattern.java:4660) ~[?:1.8.0_181]                                                                                                                       
        at java.util.regex.Pattern$Loop.match(Pattern.java:4787) ~[?:1.8.0_181]                                                                                                                            
        at java.util.regex.Pattern$GroupTail.match(Pattern.java:4719) ~[?:1.8.0_181]                                                                                                                       
        at java.util.regex.Pattern$BranchConn.match(Pattern.java:4570) ~[?:1.8.0_181]                                                                                                                      
        at java.util.regex.Pattern$CharProperty.match(Pattern.java:3779) ~[?:1.8.0_181]                                                                                                                    
        at java.util.regex.Pattern$Branch.match(Pattern.java:4606) ~[?:1.8.0_181]                                                                                                                          
        at java.util.regex.Pattern$GroupHead.match(Pattern.java:4660) ~[?:1.8.0_181]                                                                                                                       
        at java.util.regex.Pattern$Loop.match(Pattern.java:4787) ~[?:1.8.0_181]                                                                                                                            
        at java.util.regex.Pattern$GroupTail.match(Pattern.java:4719) ~[?:1.8.0_181]                                                                                                                       
        at java.util.regex.Pattern$BranchConn.match(Pattern.java:4570) ~[?:1.8.0_181]                                                                                                                      
        at java.util.regex.Pattern$CharProperty.match(Pattern.java:3779) ~[?:1.8.0_181]                                                                                                                    
        at java.util.regex.Pattern$Branch.match(Pattern.java:4606) ~[?:1.8.0_181]                                                                                                                          
        at java.util.regex.Pattern$GroupHead.match(Pattern.java:4660) ~[?:1.8.0_181]                                                                                                                       
        at java.util.regex.Pattern$Loop.match(Pattern.java:4787) ~[?:1.8.0_181]                                                                                                                            
        at java.util.regex.Pattern$GroupTail.match(Pattern.java:4719) ~[?:1.8.0_181]                                                                                                                       
        at java.util.regex.Pattern$BranchConn.match(Pattern.java:4570) ~[?:1.8.0_181]                                                                                                                      
        at java.util.regex.Pattern$CharProperty.match(Pattern.java:3779) ~[?:1.8.0_181]                                                                                                                    
        at java.util.regex.Pattern$Branch.match(Pattern.java:4606) ~[?:1.8.0_181]                                                                                                                          
        at java.util.regex.Pattern$GroupHead.match(Pattern.java:4660) ~[?:1.8.0_181]

this pattern multi line parsing (?(.|\r|\n)*) not work with 4096 bytes messages and buffer overflow occurs sequence 285361587 .

jalogisch commented 5 years ago

could you please include how we can recreate the issue on our end?

What transport did you use for the messages, what does this message contain? What kind of processing did you use? Extractors, Processing Pipelines? How does the message look like that cause the error?

Thank you for providing Information that helps us to reproduce the error to improve Graylog.

james9182 commented 5 years ago

Input from error log from nginx: 

Raw/Plaintext UDP 
recv_buffer_size: 262144

=========== Extractor Grok pattern: nginx\:\s(?<timestamp>%{YEAR}[./-]%{MONTHNUM}[./-]%{MONTHDAY}[- ]%{TIME}) \[%{LOGLEVEL:severity}\] %{POSINT:pid222}#%{NUMBER}: (?<message>(.|\r|\n)*)(?:, client: %{IPORHOST:remote_addr})(?:, server: %{IPORHOSTORUNDERSCORE:server})(?:, request: %{QS:request})?(?:, upstream: "%{URI:upstream}")?(?:, host: %{QS:host})?(?:, referrer: "%{URI:http_referer}")?$

Messages for input(2048 byte) is parsed very well without errors. :

<187>Sep 24 16:10:46 jamingo.xyz.org nginx: 2018/09/24 16:10:46 [error] 94210#94210: *768531 FastCGI sent in stderr: "-web/releases/jamingo-andy-bli-opacweb-20180922174622/cacadenol/modules/database/classes/database/mysqli.php on line 52
PHP message: PHP Warning mysqli_connect(): (HY000/2002): Connection refused in file /var/www/jamingo-andy-bli-opacweb/releases/jamingo-andy-bli-opacweb-20180922174622/cacadenol/modules/database/classes/database/mysqli.php on line 52
PHP message: PHP Warning mysqli_connect(): (HY000/2002): Connection refused in file /var/www/jamingo-andy-bli-opacweb/releases/jamingo-andy-bli-opacweb-20180922174622/cacadenol/modules/database/classes/database/mysqli.php on line 52
PHP message: PHP Warning mysqli_connect(): (HY000/2002): Connection refused in file /var/www/jamingo-andy-bli-opacweb/releases/jamingo-andy-bli-opacweb-20180922174622/cacadenol/modules/database/classes/database/mysqli.php on line 52
PHP message: PHP Warning mysqli_connect(): (HY000/2002): Connection refused in file /var/www/jamingo-andy-bli-opacweb/releases/jamingo-andy-bli-opacweb-20180922174622/cacadenol/modules/database/classes/database/mysqli.php on line 52
PHP message: PHP Warning mysqli_connect(): (HY000/2002): Connection refused in file /var/www/jamingo-andy-bli-opacweb/releases/jamingo-andy-bli-opacweb-20180922174622/cacadenol/modules/database/classes/database/mysqli.php on line 52
PHP message: PHP Warning mysqli_connect(): (HY000/2002): Connection refused in file /var/www/jamingo-andy-bli-opacweb/releases/jamingo-andy-bli-opacweb-20180922174622/cacadenol/modules/database/classes/database/mysqli.php on line 52
PHP message: PHP Warning mysqli_connect(): (HY000/2002): Connection refused in file /var/www/jamingo-andy-bli-opacweb/releases/jamingo-andy-bli-opacweb-20180922174622/cacadenol/modules/database/classes/database/mysqli.php on line 52
PHP message: PHP Warning mysqli_connect(): (HY000/2002): Connection refused in file /var/www/jamingo-andy-bli-opacweb/releases/jamingo-andy-bli-opacweb-20180922174622/caca

If nginx send full messages error: Messages for input(4182 byte) not pasrsing :

<187>Sep 24 16:10:46 jamingo.xyz.org nginx: 2018/09/24 16:10:46 [error] 94210#94210: *768531 FastCGI sent in stderr: "-web/releases/jamingo-andy-bli-opacweb-20180922174622/cacadenol/modules/database/classes/database/mysqli.php on line 52
PHP message: PHP Warning mysqli_connect(): (HY000/2002): Connection refused in file /var/www/jamingo-andy-bli-opacweb/releases/jamingo-andy-bli-opacweb-20180922174622/cacadenol/modules/database/classes/database/mysqli.php on line 52
PHP message: PHP Warning mysqli_connect(): (HY000/2002): Connection refused in file /var/www/jamingo-andy-bli-opacweb/releases/jamingo-andy-bli-opacweb-20180922174622/cacadenol/modules/database/classes/database/mysqli.php on line 52
PHP message: PHP Warning mysqli_connect(): (HY000/2002): Connection refused in file /var/www/jamingo-andy-bli-opacweb/releases/jamingo-andy-bli-opacweb-20180922174622/cacadenol/modules/database/classes/database/mysqli.php on line 52
PHP message: PHP Warning mysqli_connect(): (HY000/2002): Connection refused in file /var/www/jamingo-andy-bli-opacweb/releases/jamingo-andy-bli-opacweb-20180922174622/cacadenol/modules/database/classes/database/mysqli.php on line 52
PHP message: PHP Warning mysqli_connect(): (HY000/2002): Connection refused in file /var/www/jamingo-andy-bli-opacweb/releases/jamingo-andy-bli-opacweb-20180922174622/cacadenol/modules/database/classes/database/mysqli.php on line 52
PHP message: PHP Warning mysqli_connect(): (HY000/2002): Connection refused in file /var/www/jamingo-andy-bli-opacweb/releases/jamingo-andy-bli-opacweb-20180922174622/cacadenol/modules/database/classes/database/mysqli.php on line 52
PHP message: PHP Warning mysqli_connect(): (HY000/2002): Connection refused in file /var/www/jamingo-andy-bli-opacweb/releases/jamingo-andy-bli-opacweb-20180922174622/cacadenol/modules/database/classes/database/mysqli.php on line 52
PHP message: PHP Warning mysqli_connect(): (HY000/2002): Connection refused in file /var/www/jamingo-andy-bli-opacweb/releases/jamingo-andy-bli-opacweb-20180922174622/cacadenol/modules/database/classes/database/mysqli.php on line 52
PHP message: PHP Warning mysqli_connect(): (HY000/2002): Connection refused in file /var/www/jamingo-andy-bli-opacweb/releases/jamingo-andy-bli-opacweb-20180922174622/cacadenol/modules/database/classes/database/mysqli.php on line 52
PHP message: PHP Warning mysqli_connect(): (HY000/2002): Connection refused in file /var/www/jamingo-andy-bli-opacweb/releases/jamingo-andy-bli-opacweb-20180922174622/cacadenol/modules/database/classes/database/mysqli.php on line 52
PHP message: PHP Warning mysqli_connect(): (HY000/2002): Connection refused in file /var/www/jamingo-andy-bli-opacweb/releases/jamingo-andy-bli-opacweb-20180922174622/cacadenol/modules/database/classes/database/mysqli.php on line 52
PHP message: PHP Warning mysqli_connect(): (HY000/2002): Connection refused in file /var/www/jamingo-andy-bli-opacweb/releases/jamingo-andy-bli-opacweb-20180922174622/cacadenol/modules/database/classes/database/mysqli.php on line 52
PHP message: PHP Warning mysqli_connect(): (HY000/2002): Connection refused in file /var/www/jamingo-andy-bli-opacweb/releases/jamingo-andy-bli-opacweb-20180922174622/cacadenol/modules/database/classes/database/mysqli.php on line 52
PHP message: PHP Warning mysqli_connect(): (HY000/2002): Connection refused in file /var/www/jamingo-andy-bli-opacweb/releases/jamingo-andy-bli-opacweb-20180922174622/cacadenol/modules/database/classes/database/mysqli.php on line 52
PHP message: PHP Warning mysqli_connect(): (HY000/2002): Connection refused in file /var/www/jamingo-andy-bli-opacweb/releases/jamingo-andy-bli-opacweb-20180922174622/cacadenol/modules/database/classes/database/mysqli.php on line 52
PHP message: PHP Warning mysqli_connect(): (HY000/2002): Connection refused in file /var/www/jamingo-andy-bli-opacweb/releases/jamingo-andy-bli-opacweb-20180922174622/cacadenol/modules/database/classes/database/mysqli.php on line 52" while reading response header from upstream, client: 127.0.0.1, server: caskja.jamingo.com, request: "GET /cack/calaste_men HTTP/1.1", upstream: "fastcgi://127.0.0.1:9003", host: "caskja.jamingo.com"

this message is not processed, and there are too many errors in the logs graylog with buffer overflow sequence 285361587

zmf commented 4 years ago

We have seen the same error on some of our Grok patterns which also resulted on stuck message processing until the graylog instance was restarted. The evil part was a multiline matching pattern which was (.|\r|\n)* (same pattern that you use). Can't be sure of your use-case but as you're talking about nginx logs, you can probably safely change this pattern to something more specific.