send a log message with a full timestamp: <167>2020-07-30T20:00:59.090Z Esxi01 Vpxa: verbose vpxa[C6BCB70] [Originator@6876 sub=VpxaHalCnxHostagent opID=WFU-1b1ac72d] Completed WaitForUpdatesDone callback
send a log message with a short timestamp: <86>Jul 30 22:14:56 ubnt sudo: root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/opt/vyatta/sbin/ubnt_vtysh -c show ip route summary json
The first message causes logstash to mark the timestamp property as date, but logstash fails to supply a date when handling the second message:
logstash_1 | [WARN ] 2020-07-30 22:06:21.253 [[beats]>worker3] elasticsearch - Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"rsyslog-2020.07.30", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x7fb19247>], :response=>{"index"=>{"_index"=>"rsyslog-2020.07.30", "_type"=>"_doc", "_id"=>"1RnDoXMB-RYw4kgdDFU1", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [timestamp] of type [date] in document with id '1RnDoXMB-RYw4kgdDFU1'. Preview of field's value: 'Jul 30 22:14:56'", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"failed to parse date field [Jul 30 22:14:56] with format [strict_date_optional_time||epoch_millis]", "caused_by"=>{"type"=>"date_time_parse_exception", "reason"=>"Failed to parse with all enclosed parsers"}}}}}}
Logstash is unable to handle syslog input from systems with different time formats going into the same ES index.
Version: 7.8.1
Operating System: Official docker containers
Config File (if you have sensitive info, please remove it):
Steps to Reproduce:
send a log message with a full timestamp:
<167>2020-07-30T20:00:59.090Z Esxi01 Vpxa: verbose vpxa[C6BCB70] [Originator@6876 sub=VpxaHalCnxHostagent opID=WFU-1b1ac72d] Completed WaitForUpdatesDone callback
send a log message with a short timestamp:
<86>Jul 30 22:14:56 ubnt sudo: root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/opt/vyatta/sbin/ubnt_vtysh -c show ip route summary json
The first message causes logstash to mark the
timestamp
property asdate
, but logstash fails to supply a date when handling the second message:logstash_1 | [WARN ] 2020-07-30 22:06:21.253 [[beats]>worker3] elasticsearch - Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"rsyslog-2020.07.30", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x7fb19247>], :response=>{"index"=>{"_index"=>"rsyslog-2020.07.30", "_type"=>"_doc", "_id"=>"1RnDoXMB-RYw4kgdDFU1", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [timestamp] of type [date] in document with id '1RnDoXMB-RYw4kgdDFU1'. Preview of field's value: 'Jul 30 22:14:56'", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"failed to parse date field [Jul 30 22:14:56] with format [strict_date_optional_time||epoch_millis]", "caused_by"=>{"type"=>"date_time_parse_exception", "reason"=>"Failed to parse with all enclosed parsers"}}}}}}