search

influxdata / telegraf

Agent for collecting, processing, aggregating, and writing metrics, logs, and other arbitrary data.
https://influxdata.com/telegraf
MIT License
14.52k stars 5.56k forks source link

grok pattern not working with log parser #2645

Closed abhishekbuyt closed 7 years ago

abhishekbuyt commented 7 years ago

My nginx log format is

'"$remote_addr" "$http_x_forwarded_for" "$remote_user" "$request_time [$time_local]" "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent"' ' $request_time $upstream_response_time $pipe'; my grok pattern is

"%{IP:lb}" "(?%{IPORHOST:origin}(?:, %{IPORHOST})*)" "%{NGUSER}" "%{BASE10NUM:} [%{HTTPDATE:timestamp}]" "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} (?:%{NUMBER:bytes}|-) (?:"(?:%{URI:referrer}|-)"|%{QS:referrer}) %{QS:agent} %{BASE10NUM:request_duration} %{BASE10NUM:processing_duration} %{NOTSPACE:pipe}

i have used the grok debugger here. --https://grokdebug.herokuapp.com/

it works. on the debugger

Example log lines ---

"192.168.1.248" "175.140.230.26, 203.106.85.218, 96.17.72.149" "-" "0.014 [03/Apr/2017:18:02:33 +0530]" "GET /openx/ads/request.php?bannerid=24306&campaignid=4047&zoneid=8005&np_zone=4389&t=nads&context=cat%3Dhome%3BnpKey%3Dfilmitam%3BclientId%3D765125760%3Bclient%3Dandroid%3BappVer%3D8.3.29%3B&zone=storypage&client=android&resolution=1080x1776&clientId=765125760&appVer=8.3.29&osVersion=5.1.1&brand=Newshunt&long=109.09999849274755&lat=5.079999938607216&cellid=7729479---11701----Gsm&lang=ta&conn=w&bannercount=1&udid=359652054517805&gaid=f5b7a672-d77e-4b96-bb0e-5d7001528f11&density=3.0&pagetype=sources&newsItemId=65931385&debug=1&connectionSpeed=AVERAGE&slowConnectionMode=false&pageReferrer=HEADLINES&excludeBanners=&logmd5=f0342948f50edeb393f76114bf63bb60ZZ8005&clickmd5=9ed5f0712b43024dbe24beb7a68e9d43ZZ8005&debug=1 HTTP/1.1" 200 5 "-" "Dalvik/2.1.0 (Linux; U; Android 5.1.1; C6903 Build/14.6.A.1.236)" 0.014 0.014 . "192.168.1.248" "47.9.132.10, 124.124.128.13" "-" "0.053 [03/Apr/2017:18:02:33 +0530]" "POST /openx/ads/index.php?t=nads&zone=card-p1&client=android&resolution=720x1280&clientId=876745430&appVer=8.3.17&osVersion=5.1.1&brand=Newshunt&long=0.0&lat=0.0&cellid=682000-405-868-45----Gsm&lang=hi&conn=4G&bannercount=1&udid=911510505044269&gaid=f4cd7797-2b14-4e63-9bd7-7bb902b38c76&density=2.0&pagetype=headlines&debug=1&connectionSpeed=AVERAGE&slowConnectionMode=false HTTP/1.1" 200 2574 "-" "Dalvik/2.1.0 (Linux; U; Android 5.1.1; LS-5016 Build/LMY47V)" 0.053 0.053 . "192.168.1.248" "112.110.91.148" "-" "0.003 [03/Apr/2017:18:02:33 +0530]" "GET /openx/ads/log.php?md5=-1&log_id=a3053e6ab3fc50ae4ce59532f7e9f358ZZ8000&clientId=876266491&oxlg_cnt=8&oxlg_bannerid=4904&oxlg_campaignid=1233&oxlg_zoneid=8000&oxlg_loc=http%3A%2F%2F67031bab42d324.newshuntads.com%2Fopenx%2Fads%2Findex.php%3Ft%3Dnads%26client%3Dandroid%26res%3D720x1280%26clientId%3D876266491%26appVer%3D4.62.59%26osVersion%3D5.1.1%26brand%3DNewsHunt%26long%3D76.1420292%26lat%3D11.711316%26cellid%3D52811-404-19-414----Gsm%26isReg%3DN%26imgFmt%3D7%26featureMask%3D7340031%26langMask%3D4095%26selLang%3D512%26udid%3D869774020183422%26conn%3D2G%26groupKey%3Dentertain%26readerMask%3D1%26acqFKey%3Dhunt&oxlg_cb=0879d64a78&oxlg_req_oi=36&oxlg_req_time=03042017+12%3A32%3A12&oxlg_lgts=1491222732-69883200&ccsf=1&np_zone=3000 HTTP/1.1" 200 43 "-" "Mozilla/5.0 (Linux; Android 5.1.1; Redmi 3 Build/LMY47V; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/56.0.2924.87 Mobile Safari/537.36" 0.003 0.003 . "192.168.1.248" "157.48.5.172, 49.44.116.210, 125.56.222.156" "-" "0.013 [03/Apr/2017:18:02:33 +0530]" "POST /openx/ads/index.php?t=nads&zone=card-p1&client=android&resolution=720x1280&clientId=888835284&appVer=8.3.29&osVersion=6.0.1&brand=Newshunt&long=78.86345429&lat=14.48093892&cellid=2216465-404-49-86----Gsm&lang=en,hi,te&conn=4G&bannercount=1&udid=863914030827453&gaid=a9873935-ffaf-421c-aacf-4ed5b9d55399&density=2.0&pagetype=location&locationkey=38&debug=1&connectionSpeed=AVERAGE&slowConnectionMode=false HTTP/1.1" 200 1732 "-" "Dalvik/2.1.0 (Linux; U; Android 6.0.1; vivo 1610 Build/MMB29M)" 0.013 0.013 .

my efforts in dubugging poing to the x-http-forwarded-for in the 2nd column. Any help will be appreciated. Thanks in advance.

danielnelson commented 7 years ago

We try to reserve GitHub Issues for actionable bug reports or feature requests. Please ask general questions at the InfluxData Community site.

It will be helpful if you can try to narrow it down to a minimal example, additionally you should include the telegraf.conf you are using so that the escaping can be checked.