search

influxdata / telegraf

Agent for collecting, processing, aggregating, and writing metrics, logs, and other arbitrary data.
https://influxdata.com/telegraf
MIT License
14.63k stars 5.58k forks source link

csv_timestamp_format - Does not accept a timestamp without a year "Stamp" #8948

Closed VictorRobellini closed 12 months ago

VictorRobellini commented 3 years ago

Relevant telegraf.conf:

[[inputs.tail]]
    files = ["/var/log/pfblockerng/ip_block.log"]
    name_suffix = "_ipblock"
    data_format = "csv"
    csv_delimiter = ","
    from_beginning = true
    csv_timestamp_column = "timestamp"
    csv_timestamp_format = "Jan _2 15:04:05"
    csv_tag_columns = ["geoip_code","feed_name","src_ip"]
    csv_column_names =    ["timestamp","rulenum","interface","friendlyname","action","ip_version","protocolid","protocol","src_ip","dest_ip","src_port","dest_port","direction","geoip_code","ip_alias_name","ip_evaluated","feed_name","resolvedhostname","clienthostname","duplicateeventstatus"]

Logfile

    Mar 6 02:00:15,1770011016,igb2.11,VLAN_11_IOT,block,4,1,ICMP,10.0.11.105,8.8.8.8,request,2836,out,US,pfB_DoH_IP_v4,8.8.8.8,TheGreatWall_DoH_IP_v4,dns.google,Google-Home-Mini,null,-
    Mar 6 02:00:15,1770011016,igb2.11,VLAN_11_IOT,block,4,1,ICMP,10.0.11.105,8.8.8.8,request,2836,out,US,pfB_DoH_IP_v4,8.8.8.8,TheGreatWall_DoH_IP_v4,dns.google,Google-Home-Mini,null,-
    Mar 6 02:00:15,1770011016,igb2.11,VLAN_11_IOT,block,4,1,ICMP,10.0.11.105,8.8.8.8,request,2836,out,US,pfB_DoH_IP_v4,8.8.8.8,TheGreatWall_DoH_IP_v4,dns.google,Google-Home-Mini,null,-
    Mar 6 02:00:15,1770011016,igb2.11,VLAN_11_IOT,block,4,1,ICMP,10.0.11.105,8.8.8.8,request,2836,out,US,pfB_DoH_IP_v4,8.8.8.8,TheGreatWall_DoH_IP_v4,dns.google,Google-Home-Mini,null,-
    Mar 6 02:00:15,1770011016,igb2.11,VLAN_11_IOT,block,4,1,ICMP,10.0.11.105,8.8.8.8,request,2836,out,US,pfB_DoH_IP_v4,8.8.8.8,TheGreatWall_DoH_IP_v4,dns.google,Google-Home-Mini,null,-

telegraf --test --config /usr/local/etc/telegraf.conf

    > tail_ipblock,feed_name=TheGreatWall_DoH_IP_v4,geoip_code=US,host=pfSense.home,path=/var/log/pfblockerng/ip_block.log,src_ip=10.0.11.105 action="block",clienthostname="Google-Home-Mini",dest_ip="8.8.8.8",dest_port=1296i,direction="out",duplicateeventstatus="null",friendlyname="VLAN_11_IOT",interface="igb2.11",ip_alias_name="pfB_DoH_IP_v4",ip_evaluated="8.8.8.8",ip_version=4i,protocol="ICMP",protocolid=1i,resolvedhostname="dns.google",rulenum=1770011016i,src_port="request" -6821420594871345152
    > tail_ipblock,feed_name=TheGreatWall_DoH_IP_v4,geoip_code=US,host=pfSense.home,path=/var/log/pfblockerng/ip_block.log,src_ip=10.0.11.105 action="block",clienthostname="Google-Home-Mini",dest_ip="8.8.8.8",dest_port=1296i,direction="out",duplicateeventstatus="null",friendlyname="VLAN_11_IOT",interface="igb2.11",ip_alias_name="pfB_DoH_IP_v4",ip_evaluated="8.8.8.8",ip_version=4i,protocol="ICMP",protocolid=1i,resolvedhostname="dns.google",rulenum=1770011016i,src_port="request" -6821420594871345152

Influx results

    > precision rfc3339
    > select * from tail_ipblock limit 1
    name: tail_ipblock
    time                           action clienthostname dest_ip        dest_port direction duplicateeventstatus feed_name    friendlyname geoip_code host                     interface ip_alias_name ip_evaluated ip_version path                              protocol protocolid resolvedhostname        rulenum    src_ip       src_port
    ----                           ------ -------------- -------        --------- --------- -------------------- ---------    ------------ ---------- ----                     --------- ------------- ------------ ---------- ----                              -------- ---------- ----------------        -------    ------       --------             
    1753-11-02T03:27:53.128654848Z block  wan            11.111.111.111 2059      in        null                 CINS_army_v4 4WAN         GB         pfSense.home igb0      pfB_PRI1_v4   80.82.78.100 4          /var/log/pfblockerng/ip_block.log UDP      17         security.criminalip.com 1770009194 80.82.78.100 57291

System info:

Telegraf 1.17.0 Influxdb 1.8.3 pfSense 2.5.0

Steps to reproduce:

Use the above telegraf config to parse the log file posted in this issue report.

Expected behavior:

The date would be properly parsed

Actual behavior:

Somehow I end up time traveling.

Additional info:

This timestamp is referenced as "Stamp" in the go docs and is referenced in many other documents. https://github.com/golang/go/blob/0fdc3801bfd43d6f55e4ea5bf095e1ea55430339/src/time/format.go#L86

Go sample

https://play.golang.org/p/ihlq_i2rB4a

ssoroka commented 3 years ago

Very interesting. It makes sense. I guess you're intending it to default to the current year? Seems that might not be too reliable when processing data around dec/jan.

VictorRobellini commented 3 years ago

I completely agree but it's how the logs are delivered. The format is called ts-syslog.

ts-syslog ("Jan 02 15:04:05", parsed time is set to the current year)

https://github.com/influxdata/telegraf/blob/master/plugins/parsers/grok/README.md

powersj commented 12 months ago

Hi,

Looking through older issues.

Fundamentally, you need to specify a year at some point. You cannot assume the use of the current of year. While in the middle of the year it might be obvious to use the current year, but this isn't always the case. For example on new years, if you start putting 2024 on everything that could be wrong while parsing logs from Dec 31, 2023.

The suggestion then is for users to not parse the timestamp, capture it as a a field, and use starlark to set the year based on their situation and then update the timestamp with that new field. This is a bit messy, but when you try to parse without a year, you will eventually need to set one and knowing what to set it to is not easily known or set.