Logparser Grok pattern - Haproxy

JulienChampseix commented 7 years ago

Directions

Trouble around haproxy grok pattern which looks not supported in telegraf

Bug report

Relevant telegraf.conf:

System info:

[Telegraf v1.1.2 (git: release-1.1.0 3da3a2279108d02c2158782794532de7cd7deacd)]

Steps to reproduce:

Edit telegraf.conf + uncomment inputs.logparser and inputs.logparser.grok
Add patterns = ["%{HAPROXYHTTP}"]
Add measurement = "metrics_haproxy_logs"

Add those custom_patterns

custom_patterns = '''
    HAPROXYTIME (?!<[0-9])%{HOUR:haproxy_hour}:%{MINUTE:haproxy_minute}(?::%{SECOND:haproxy_second})(?![0-9])
    HAPROXYDATE %{MONTHDAY:haproxy_monthday}/%{MONTH:haproxy_month}/%{YEAR:haproxy_year}:%{HAPROXYTIME:haproxy_time}.%{INT:haproxy_milliseconds}
    HAPROXYCAPTUREDREQUESTHEADERS %{DATA:captured_request_headers}
    HAPROXYCAPTUREDRESPONSEHEADERS %{DATA:captured_response_headers}
    HAPROXYHTTP %{SYSLOGTIMESTAMP:syslog_timestamp} %{IPORHOST:syslog_server} %{SYSLOGPROG}: %{IP:client_ip}:%{INT:client_port} \[%{HAPROXYDATE:accept_date}\] %{NOTSPACE:frontend_name} %{NOTSPACE:backend_name}/%{NOTSPACE:server_name} %{INT:time_request}/%{INT:time_queue}/%{INT:time_backend_connect}/%{INT:time_backend_response}/%{NOTSPACE:time_duration} %{INT:http_status_code} %{NOTSPACE:bytes_read} %{DATA:captured_request_cookie} %{DATA:captured_response_cookie} %{NOTSPACE:termination_state} %{INT:actconn}/%{INT:feconn}/%{INT:beconn}/%{INT:srvconn}/%{NOTSPACE:retries} %{INT:srv_queue}/%{INT:backend_queue} (\{%{HAPROXYCAPTUREDREQUESTHEADERS}\})?( )?(\{%{HAPROXYCAPTUREDRESPONSEHEADERS}\})?( )?"%{WORD:http_verb} (%{URIPROTO:http_proto}://)?(?:%{USER:http_user}(?::[^@]*)?@)?(?:%{URIHOST:http_host})?(?:%{URIPATHPARAM:http_request})?( HTTP/%{NUMBER:http_version}")?
 '''

Restart Telegraf

Logs

Jan 31 16:09:52 test telegraf: 2017/01/31 16:09:52 I! Starting Telegraf (version 1.1.2)
Jan 31 16:09:52 test telegraf: 2017/01/31 16:09:52 I! Loaded outputs: influxdb
Jan 31 16:09:52 test telegraf: 2017/01/31 16:09:52 I! Loaded inputs: inputs.processes inputs.net inputs.cpu inputs.disk inputs.kernel inputs.mem inputs.logparser inputs.diskio inputs.swap inputs.system inputs.haproxy
Jan 31 16:09:52 test telegraf: 2017/01/31 16:09:52 I! Tags enabled: env=production host=test server=test server_type=loadbalancer
Jan 31 16:09:52 test telegraf: 2017/01/31 16:09:52 I! Agent Config: Interval:10s, Quiet:false, Hostname:"test.localdomain", Flush Interval:10s
Jan 31 16:09:52 test telegraf: 2017/01/31 16:09:52 Seeked /var/log/haproxy.log - &{Offset:0 Whence:2}

The measurement metrics_haproxy_logs is not created instead of other measurement (cpu, memory,..) are populated.

If i put only those custom_patterns, it's works :

custom_patterns = '''
    HAPROXYTIME (?!<[0-9])%{HOUR:haproxy_hour}:%{MINUTE:haproxy_minute}(?::%{SECOND:haproxy_second})(?![0-9])
    HAPROXYDATE %{MONTHDAY:haproxy_monthday}/%{MONTH:haproxy_month}/%{YEAR:haproxy_year}:%{HAPROXYTIME:haproxy_time}.%{INT:haproxy_milliseconds}
    HAPROXYCAPTUREDREQUESTHEADERS %{DATA:captured_request_headers}
    HAPROXYCAPTUREDRESPONSEHEADERS %{DATA:captured_response_headers}
    HAPROXYHTTP %{SYSLOGTIMESTAMP:syslog_timestamp} %{IPORHOST:syslog_server} %{SYSLOGPROG}: %{IP:client_ip}:%{INT:client_port}
 '''

Any idea about this trouble ?

sparrc commented 7 years ago

this is because of #2178, Go doesn't support negative lookaheads

JulienChampseix commented 7 years ago

Thanks @sparrc for your quick response ! (as always ;) ) FYI, i have opened an issue on go github (hope to have a workaround soon). Don't hesitate to add more details if you can provide more infos.

influxdata / telegraf