search

influxdata / telegraf

Agent for collecting, processing, aggregating, and writing metrics, logs, and other arbitrary data.
https://influxdata.com/telegraf
MIT License
14.89k stars 5.6k forks source link

Add ability to parse other text encodings to tail/logparser input #6775

Open rameshs10 opened 4 years ago

rameshs10 commented 4 years ago

Relevant telegraf.conf:

[[inputs.logparser]] files = ["C:\Users\jen.tomm\Desktop\applicationlog_2019-11-13_07-43-25\sample.log"] from_beginning = false [inputs.logparser.grok] measurement = "applicationlog"
patterns = ["^%{TIMESTAMP_ISO8601:timestamp},%{WORD:LogLevel},%{NUMBER:ThreadId},%{WORD:LoggerName},%{DATA:ServiceId},%{DATA:RequestId},%{DATA:UserId},%{DATA:RemoteDate},%{DATA:RemoteServer},%{DATA:RemoteProcessId},%{DATA:RemoteThreadId},%{GREEDYDATA:message}"]

System info:

windows telegraf 1.13

Steps to reproduce:

  1. parsing logs through log parser with grok pattern
  2. using grafana log panel plugin to get the log metrics

Expected behavior:

logs have to display with proper format but getting space between the letters

Actual behavior:

getting space between the logs

2 0 1 9-1 1-1 3 0 4 : 5 4 : 5 4 . 8 6 1 0 0 0 , D E B U G, 6 1 0 0 , a p p l i c a t i o n S e r v i c e , R e n d e r W o r k e r , , , , , , , " R e n d e r Q u e u e H a n d l e r R e a d B o d y : W o r k e r < E U D - V - 0 A 6 F 5 5 1 8 0 > r e q u e s t < G e t J o b > "

Additional info:

Date,LogLevel,ThreadId,LoggerName,ServiceId,RequestId,UserId,RemoteDate,RemoteServer,RemoteProcessId,RemoteThreadId,Message 2019-11-13 04:54:54.861000,DEBUG,6100,applicationService,RenderWorker,,,,,,,"RenderQueueHandler_ReadBody: Worker request " 2019-11-13 04:54:54.920000,NOTICE,4592,applicationService,EUD-V-0A6F55180|55ECC331,,jen.tomm,,,,,"QueueHandler::GrowServicePoolThreads: Growing thread pool count to <9>"

rameshs10 commented 4 years ago

Kindly advice me if is the problem with grafana plugin issue.

danielnelson commented 4 years ago

Let's check what Telegraf is parsing, this will run for 60 seconds and print the output:

telegraf --input-filter logparser --test --test-wait 60
rameshs10 commented 4 years ago

in telegraf it is writing right format

2019-11-13 04:54:54.861000,DEBUG,6100,applicationService,RenderWorker,,,,,,,"RenderQueueHandler_ReadBody: Worker request "

danielnelson commented 4 years ago

I would check the data in InfluxDB next using the influx command line tool, if it is looking okay there then it must be related to Grafana. Can you ask any follow-up questions at the InfluxData Community site?

rameshs10 commented 4 years ago

I would check the data in InfluxDB next using the influx command line tool, if it is looking okay there then it must be related to Grafana. Can you ask any follow-up questions at the InfluxData Community site?

Thanks daniel ,

i have logs in UCS-2 LE BOM encoding and getting space between the letters while running telegraf test command as well but if i changed to UTF-8 it is fine . is there any way to get the result in same UCS-2 LE BOM encoding ?

danielnelson commented 4 years ago

It's not possible right now to parse encodings other than utf-8.