Unclear and very verbose status 400 messages when Logstash hits a Elasticsearch mapping error during inserts

elvarb commented 8 years ago

Upgraded one Logstash pipeline to version 2.1.1 that is running on Windows. The config for that pipeline includes the throttle filter so when Logstash starts this is written to the log.

{:timestamp=>"2015-12-21T14:52:31.833000+0000", :message=>"Defaulting filter worker threads to 1 because there are some filters that might not work with multiple worker threads", :count_was=>2, :filters=>["throttle"], :level=>:warn}

I also added a second elasticsearch output to a new Elasticsearch 2 cluster and after that change the logs started to fill with these errors for probably every message.

{:timestamp=>"2015-12-21T14:42:42.814000+0000", :message=>"Failed action. ", :status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-2015.12.21", :_type=>"logs", :_routing=>nil}, #<LogStash::Event:0x45de7e3c @metadata_accessors=#<LogStash::Util::Accessors:0x4e15819b @store={}, @lut={}>, @cancelled=false, @data={"@version"=>"1", "@timestamp"=>"2015-12-21T14:39:39.000Z", .....
{:timestamp=>"2015-12-21T14:42:40.404000+0000", :message=>"Failed action. ", :status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-2015.12.21", :_type=>"logs", :_routing=>nil}, #<LogStash::Event:0x6f953673 @metadata_accessors=#<LogStash::Util::Accessors:0x71d04f7d @store={}, @lut={}>, @cancelled=false, @data={"@version"=>"1", "@timestamp"=>"2015-12-21T14:39:24.000Z", .....

This is the only Logstash instance I have running that has this problem, also the only one giving the worker thread warning.

elvarb commented 8 years ago

Noticed the same for a different instance but only for parts of the configs, not much volume so I missed it earlier.

That instance has 12 different config files, each for specific type of logs and each includes everything needed for that log type (input, filter and output).

2 of 12 were failing

Filters used kv, translate, mutate
Filters used, json

By removing one elasticsearch output stopped the errors.

elvarb commented 8 years ago

I think I have figured out what is the issue, mapping errors between servers. Hard to notice because how the log message is built. Also I think there is something wrong with how the log message is written

This is an example of a single line in the logstash log.

The problems I see are two

The source of the error is at the end of the line, it should be as close to the start of the line as possible. Should this be a new issue?
What is with the repeated full content of the event again and again?

{
    :timestamp=>"2015-12-22T14:02:14.372000+0000",
    :message=>"Failed action. ",
    :status=>400,
    :action=>[
        "index",
        {
            :_id=>nil,
            :_index=>"logstash-2015.12.22",
            :_type=>"logs",
            :_routing=>nil
        },
        #<LogStash::Event:0x38af72e8 
        @metadata_accessors=#<LogStash::Util::Accessors:0x52358b52 
        @store={},
        @lut={}>,
        @cancelled=false,
        @data={
            "@version"=>"1",
            "@timestamp"=>"2015-12-22T13:59:24.000Z",
            "host"=>"172.25.243.3",
            "tags"=>["winserv"],
            "winevent"=>{
                "EventTime"=>"2015-12-22 13:59:24",
                "Hostname"=>"hostname01.domain",
                "Keywords"=>-9223372036854775808,
                "EventType"=>"INFO",
                "SeverityValue"=>2,
                "Severity"=>"INFO",
                "EventID"=>36867,
                "SourceName"=>"Schannel",
                "ProviderGuid"=>"{1F678132-5938-4686-9FDC-C8FF68F15C85}",
                "Version"=>0,
                "Task"=>0,
                "OpcodeValue"=>0,
                "RecordNumber"=>253714,
                "ProcessID"=>488,
                "ThreadID"=>9428,
                "Channel"=>"System",
                "Domain"=>"NT AUTHORITY",
                "AccountName"=>"SYSTEM",
                "UserID"=>"SYSTEM",
                "AccountType"=>"User",
                "Message"=>"Creating an SSL client credential.",
                "Opcode"=>"Info",
                "Type"=>"client",
                "EventReceivedTime"=>"2015-12-22 13:59:26",
                "SourceModuleName"=>"eventlog",
                "SourceModuleType"=>"im_msvistalog"}
            },
        @metadata={},
        @accessors=#<LogStash::Util::Accessors:0x24eba1 
        @store={
            "@version"=>"1",
            "@timestamp"=>"2015-12-22T13:59:24.000Z",
            "host"=>"172.25.243.3",
            "tags"=>["winserv"],
            "winevent"=>{
                "EventTime"=>"2015-12-22 13:59:24",
                "Hostname"=>"hostname01.domain",
                "Keywords"=>-9223372036854775808,
                "EventType"=>"INFO",
                "SeverityValue"=>2,
                "Severity"=>"INFO",
                "EventID"=>36867,
                "SourceName"=>"Schannel",
                "ProviderGuid"=>"{1F678132-5938-4686-9FDC-C8FF68F15C85}",
                "Version"=>0,
                "Task"=>0,
                "OpcodeValue"=>0,
                "RecordNumber"=>253714,
                "ProcessID"=>488,
                "ThreadID"=>9428,
                "Channel"=>"System",
                "Domain"=>"NT AUTHORITY",
                "AccountName"=>"SYSTEM",
                "UserID"=>"SYSTEM",
                "AccountType"=>"User",
                "Message"=>"Creating an SSL client credential.",
                "Opcode"=>"Info",
                "Type"=>"client",
                "EventReceivedTime"=>"2015-12-22 13:59:26",
                "SourceModuleName"=>"eventlog",
                "SourceModuleType"=>"im_msvistalog"}
            },
        @lut={
            "[tags]"=>[
                {
                    "@version"=>"1",
                    "@timestamp"=>"2015-12-22T13:59:24.000Z",
                    "host"=>"172.25.243.3",
                    "tags"=>["winserv"],
                    "winevent"=>{
                        "EventTime"=>"2015-12-22 13:59:24",
                        "Hostname"=>"hostname01.domain",
                        "Keywords"=>-9223372036854775808,
                        "EventType"=>"INFO",
                        "SeverityValue"=>2,
                        "Severity"=>"INFO",
                        "EventID"=>36867,
                        "SourceName"=>"Schannel",
                        "ProviderGuid"=>"{1F678132-5938-4686-9FDC-C8FF68F15C85}",
                        "Version"=>0,
                        "Task"=>0,
                        "OpcodeValue"=>0,
                        "RecordNumber"=>253714,
                        "ProcessID"=>488,
                        "ThreadID"=>9428,
                        "Channel"=>"System",
                        "Domain"=>"NT AUTHORITY",
                        "AccountName"=>"SYSTEM",
                        "UserID"=>"SYSTEM",
                        "AccountType"=>"User",
                        "Message"=>"Creating an SSL client credential.",
                        "Opcode"=>"Info",
                        "Type"=>"client",
                        "EventReceivedTime"=>"2015-12-22 13:59:26",
                        "SourceModuleName"=>"eventlog",
                        "SourceModuleType"=>"im_msvistalog"}
                    },
                    "tags"
                ],
            "message"=>[
                {
                    "@version"=>"1",
                    "@timestamp"=>"2015-12-22T13:59:24.000Z",
                    "host"=>"172.25.243.3",
                    "tags"=>["winserv"],
                    "winevent"=>{
                    "EventTime"=>"2015-12-22 13:59:24",
                        "Hostname"=>"hostname01.domain",
                        "Keywords"=>-9223372036854775808,
                        "EventType"=>"INFO",
                        "SeverityValue"=>2,
                        "Severity"=>"INFO",
                        "EventID"=>36867,
                        "SourceName"=>"Schannel",
                        "ProviderGuid"=>"{1F678132-5938-4686-9FDC-C8FF68F15C85}",
                        "Version"=>0,
                        "Task"=>0,
                        "OpcodeValue"=>0,
                        "RecordNumber"=>253714,
                        "ProcessID"=>488,
                        "ThreadID"=>9428,
                        "Channel"=>"System",
                        "Domain"=>"NT AUTHORITY",
                        "AccountName"=>"SYSTEM",
                        "UserID"=>"SYSTEM",
                        "AccountType"=>"User",
                        "Message"=>"Creating an SSL client credential.",
                        "Opcode"=>"Info",
                        "Type"=>"client",
                        "EventReceivedTime"=>"2015-12-22 13:59:26",
                        "SourceModuleName"=>"eventlog",
                        "SourceModuleType"=>"im_msvistalog"}
                    },
                    "message"],
                    "winevent"=>[
                        {
                            "@version"=>"1",
                            "@timestamp"=>"2015-12-22T13:59:24.000Z",
                            "host"=>"172.25.243.3",
                            "tags"=>["winserv"],
                            "winevent"=>{
                                "EventTime"=>"2015-12-22 13:59:24",
                                "Hostname"=>"hostname01.domain",
                                "Keywords"=>-9223372036854775808,
                                "EventType"=>"INFO",
                                "SeverityValue"=>2,
                                "Severity"=>"INFO",
                                "EventID"=>36867,
                                "SourceName"=>"Schannel",
                                "ProviderGuid"=>"{1F678132-5938-4686-9FDC-C8FF68F15C85}",
                                "Version"=>0,
                                "Task"=>0,
                                "OpcodeValue"=>0,
                                "RecordNumber"=>253714,
                                "ProcessID"=>488,
                                "ThreadID"=>9428,
                                "Channel"=>"System",
                                "Domain"=>"NT AUTHORITY",
                                "AccountName"=>"SYSTEM",
                                "UserID"=>"SYSTEM",
                                "AccountType"=>"User",
                                "Message"=>"Creating an SSL client credential.",
                                "Opcode"=>"Info",
                                "Type"=>"client",
                                "EventReceivedTime"=>"2015-12-22 13:59:26",
                                "SourceModuleName"=>"eventlog",
                                "SourceModuleType"=>"im_msvistalog"
                            }
                        },
                        "winevent"
                    ],
                    "[winevent][Channel]"=>[
                        {
                            "EventTime"=>"2015-12-22 13:59:24",
                            "Hostname"=>"hostname01.domain",
                            "Keywords"=>-9223372036854775808,
                            "EventType"=>"INFO",
                            "SeverityValue"=>2,
                            "Severity"=>"INFO",
                            "EventID"=>36867,
                            "SourceName"=>"Schannel",
                            "ProviderGuid"=>"{1F678132-5938-4686-9FDC-C8FF68F15C85}",
                            "Version"=>0,
                            "Task"=>0,
                            "OpcodeValue"=>0,
                            "RecordNumber"=>253714,
                            "ProcessID"=>488,
                            "ThreadID"=>9428,
                            "Channel"=>"System",
                            "Domain"=>"NT AUTHORITY",
                            "AccountName"=>"SYSTEM",
                            "UserID"=>"SYSTEM",
                            "AccountType"=>"User",
                            "Message"=>"Creating an SSL client credential.",
                            "Opcode"=>"Info",
                            "Type"=>"client",
                            "EventReceivedTime"=>"2015-12-22 13:59:26",
                            "SourceModuleName"=>"eventlog",
                            "SourceModuleType"=>"im_msvistalog"
                        },
                        "Channel"
                    ],
                    "[winevent][EventTime]"=>[
                        {
                            "EventTime"=>"2015-12-22 13:59:24",
                            "Hostname"=>"hostname01.domain",
                            "Keywords"=>-9223372036854775808,
                            "EventType"=>"INFO",
                            "SeverityValue"=>2,
                            "Severity"=>"INFO",
                            "EventID"=>36867,
                            "SourceName"=>"Schannel",
                            "ProviderGuid"=>"{1F678132-5938-4686-9FDC-C8FF68F15C85}",
                            "Version"=>0,
                            "Task"=>0,
                            "OpcodeValue"=>0,
                            "RecordNumber"=>253714,
                            "ProcessID"=>488,
                            "ThreadID"=>9428,
                            "Channel"=>"System",
                            "Domain"=>"NT AUTHORITY",
                            "AccountName"=>"SYSTEM",
                            "UserID"=>"SYSTEM",
                            "AccountType"=>"User",
                            "Message"=>"Creating an SSL client credential.",
                            "Opcode"=>"Info",
                            "Type"=>"client",
                            "EventReceivedTime"=>"2015-12-22 13:59:26",
                            "SourceModuleName"=>"eventlog",
                            "SourceModuleType"=>"im_msvistalog"
                        },
                        "EventTime"
                    ],
                    "@timestamp"=>[
                        {
                            "@version"=>"1",
                            "@timestamp"=>"2015-12-22T13:59:24.000Z",
                            "host"=>"172.25.243.3",
                            "tags"=>["winserv"],
                            "winevent"=>{
                                "EventTime"=>"2015-12-22 13:59:24",
                                "Hostname"=>"hostname01.domain",
                                "Keywords"=>-9223372036854775808,
                                "EventType"=>"INFO",
                                "SeverityValue"=>2,
                                "Severity"=>"INFO",
                                "EventID"=>36867,
                                "SourceName"=>"Schannel",
                                "ProviderGuid"=>"{1F678132-5938-4686-9FDC-C8FF68F15C85}",
                                "Version"=>0,
                                "Task"=>0,
                                "OpcodeValue"=>0,
                                "RecordNumber"=>253714,
                                "ProcessID"=>488,
                                "ThreadID"=>9428,
                                "Channel"=>"System",
                                "Domain"=>"NT AUTHORITY",
                                "AccountName"=>"SYSTEM",
                                "UserID"=>"SYSTEM",
                                "AccountType"=>"User",
                                "Message"=>"Creating an SSL client credential.",
                                "Opcode"=>"Info",
                                "Type"=>"client",
                                "EventReceivedTime"=>"2015-12-22 13:59:26",
                                "SourceModuleName"=>"eventlog",
                                "SourceModuleType"=>"im_msvistalog"
                            }
                        },
                        "@timestamp"
                    ],
                    "[winevent][LogonType]"=>[
                        {
                            "EventTime"=>"2015-12-22 13:59:24",
                            "Hostname"=>"hostname01.domain",
                            "Keywords"=>-9223372036854775808,
                            "EventType"=>"INFO",
                            "SeverityValue"=>2,
                            "Severity"=>"INFO",
                            "EventID"=>36867,
                            "SourceName"=>"Schannel",
                            "ProviderGuid"=>"{1F678132-5938-4686-9FDC-C8FF68F15C85}",
                            "Version"=>0,
                            "Task"=>0,
                            "OpcodeValue"=>0,
                            "RecordNumber"=>253714,
                            "ProcessID"=>488,
                            "ThreadID"=>9428,
                            "Channel"=>"System",
                            "Domain"=>"NT AUTHORITY",
                            "AccountName"=>"SYSTEM",
                            "UserID"=>"SYSTEM",
                            "AccountType"=>"User",
                            "Message"=>"Creating an SSL client credential.",
                            "Opcode"=>"Info",
                            "Type"=>"client",
                            "EventReceivedTime"=>"2015-12-22 13:59:26",
                            "SourceModuleName"=>"eventlog",
                            "SourceModuleType"=>"im_msvistalog"
                        },
                        "LogonType"
                    ],
                    "[winevent][SourceName]"=>[
                        {
                            "EventTime"=>"2015-12-22 13:59:24",
                            "Hostname"=>"hostname01.domain",
                            "Keywords"=>-9223372036854775808,
                            "EventType"=>"INFO",
                            "SeverityValue"=>2,
                            "Severity"=>"INFO",
                            "EventID"=>36867,
                            "SourceName"=>"Schannel",
                            "ProviderGuid"=>"{1F678132-5938-4686-9FDC-C8FF68F15C85}",
                            "Version"=>0,
                            "Task"=>0,
                            "OpcodeValue"=>0,
                            "RecordNumber"=>253714,
                            "ProcessID"=>488,
                            "ThreadID"=>9428,
                            "Channel"=>"System",
                            "Domain"=>"NT AUTHORITY",
                            "AccountName"=>"SYSTEM",
                            "UserID"=>"SYSTEM",
                            "AccountType"=>"User",
                            "Message"=>"Creating an SSL client credential.",
                            "Opcode"=>"Info",
                            "Type"=>"client",
                            "EventReceivedTime"=>"2015-12-22 13:59:26",
                            "SourceModuleName"=>"eventlog",
                            "SourceModuleType"=>"im_msvistalog"
                        },
                        "SourceName"
                    ],
                    "[winevent][SourceGroup]"=>[
                        {
                            "EventTime"=>"2015-12-22 13:59:24",
                            "Hostname"=>"hostname01.domain",
                            "Keywords"=>-9223372036854775808,
                            "EventType"=>"INFO",
                            "SeverityValue"=>2,
                            "Severity"=>"INFO",
                            "EventID"=>36867,
                            "SourceName"=>"Schannel",
                            "ProviderGuid"=>"{1F678132-5938-4686-9FDC-C8FF68F15C85}",
                            "Version"=>0,
                            "Task"=>0,
                            "OpcodeValue"=>0,
                            "RecordNumber"=>253714,
                            "ProcessID"=>488,
                            "ThreadID"=>9428,
                            "Channel"=>"System",
                            "Domain"=>"NT AUTHORITY",
                            "AccountName"=>"SYSTEM",
                            "UserID"=>"SYSTEM",
                            "AccountType"=>"User",
                            "Message"=>"Creating an SSL client credential.",
                            "Opcode"=>"Info",
                            "Type"=>"client",
                            "EventReceivedTime"=>"2015-12-22 13:59:26",
                            "SourceModuleName"=>"eventlog",
                            "SourceModuleType"=>"im_msvistalog"
                        },
                        "SourceGroup"
                    ],
                    "[winevent][Category]"=>[
                        {
                            "EventTime"=>"2015-12-22 13:59:24",
                            "Hostname"=>"hostname01.domain",
                            "Keywords"=>-9223372036854775808,
                            "EventType"=>"INFO",
                            "SeverityValue"=>2,
                            "Severity"=>"INFO",
                            "EventID"=>36867,
                            "SourceName"=>"Schannel",
                            "ProviderGuid"=>"{1F678132-5938-4686-9FDC-C8FF68F15C85}",
                            "Version"=>0,
                            "Task"=>0,
                            "OpcodeValue"=>0,
                            "RecordNumber"=>253714,
                            "ProcessID"=>488,
                            "ThreadID"=>9428,
                            "Channel"=>"System",
                            "Domain"=>"NT AUTHORITY",
                            "AccountName"=>"SYSTEM",
                            "UserID"=>"SYSTEM",
                            "AccountType"=>"User",
                            "Message"=>"Creating an SSL client credential.",
                            "Opcode"=>"Info",
                            "Type"=>"client",
                            "EventReceivedTime"=>"2015-12-22 13:59:26",
                            "SourceModuleName"=>"eventlog",
                            "SourceModuleType"=>"im_msvistalog"
                        },
                        "Category"
                    ],
                    "[custom_object_that_should_not_appear_in_this_event][Level]"=>nil,
                    "[winevent][EventID]"=>[
                        {
                            "EventTime"=>"2015-12-22 13:59:24",
                            "Hostname"=>"hostname01.domain",
                            "Keywords"=>-9223372036854775808,
                            "EventType"=>"INFO",
                            "SeverityValue"=>2,
                            "Severity"=>"INFO",
                            "EventID"=>36867,
                            "SourceName"=>"Schannel",
                            "ProviderGuid"=>"{1F678132-5938-4686-9FDC-C8FF68F15C85}",
                            "Version"=>0,
                            "Task"=>0,
                            "OpcodeValue"=>0,
                            "RecordNumber"=>253714,
                            "ProcessID"=>488,
                            "ThreadID"=>9428,
                            "Channel"=>"System",
                            "Domain"=>"NT AUTHORITY",
                            "AccountName"=>"SYSTEM",
                            "UserID"=>"SYSTEM",
                            "AccountType"=>"User",
                            "Message"=>"Creating an SSL client credential.",
                            "Opcode"=>"Info",
                            "Type"=>"client",
                            "EventReceivedTime"=>"2015-12-22 13:59:26",
                            "SourceModuleName"=>"eventlog",
                            "SourceModuleType"=>"im_msvistalog"
                        },
                        "EventID"
                    ],
                    "[custom_object_that_should_not_appear_in_this_event][Environment]"=>nil,
                    "type"=>[
                        {
                            "@version"=>"1",
                            "@timestamp"=>"2015-12-22T13:59:24.000Z",
                            "host"=>"172.25.243.3",
                            "tags"=>["winserv"],
                            "winevent"=>{
                                "EventTime"=>"2015-12-22 13:59:24",
                                "Hostname"=>"hostname01.domain",
                                "Keywords"=>-9223372036854775808,
                                "EventType"=>"INFO",
                                "SeverityValue"=>2,
                                "Severity"=>"INFO",
                                "EventID"=>36867,
                                "SourceName"=>"Schannel",
                                "ProviderGuid"=>"{1F678132-5938-4686-9FDC-C8FF68F15C85}",
                                "Version"=>0,
                                "Task"=>0,
                                "OpcodeValue"=>0,
                                "RecordNumber"=>253714,
                                "ProcessID"=>488,
                                "ThreadID"=>9428,
                                "Channel"=>"System",
                                "Domain"=>"NT AUTHORITY",
                                "AccountName"=>"SYSTEM",
                                "UserID"=>"SYSTEM",
                                "AccountType"=>"User",
                                "Message"=>"Creating an SSL client credential.",
                                "Opcode"=>"Info",
                                "Type"=>"client",
                                "EventReceivedTime"=>"2015-12-22 13:59:26",
                                "SourceModuleName"=>"eventlog",
                                "SourceModuleType"=>"im_msvistalog"
                            }
                        },
                        "type"
                    ]
                }>>],
:response=>{
    "create"=>{
        "_index"=>"logstash-2015.12.22",
        "_type"=>"logs",
        "_id"=>"AVHJ_2LzY9l0n1eA4GZe",
        "status"=>400,
        "error"=>{
            "type"=>"mapper_parsing_exception",
            "reason"=>"failed to parse [winevent.EventTime]",
            "caused_by"=>{
                "type"=>"number_format_exception",
                "reason"=>"For input string: \"2015-12-22 13:59:24\""
            }
        }
    }
},
:level=>:warn}

elvarb commented 8 years ago

Also In the message this line shows up

 "[custom_object_that_should_not_appear_in_this_event][Level]"=>nil,

This field is sometimes used for a custom application we have. The message content of the eventlog is a json document that is converted to a custom object in Logstash. Each of those events has 10-20 objects and Level is one of them. So its odd to see one of those objects specified in this error event.

loekvangool commented 7 years ago

This is still an issue:

When I use an illegal template from Logstash using manage_template, Logstash shows an unhelpful message instead of passing through the error from Elasticsearch that allows me to go ahead and fix it.

Version: 5.2
Operating System: OSX

Config File (if you have sensitive info, please remove it):

manage_template => true
template_overwrite => true
template_name => "flights"
template => "./flights_mapping.json"

The Logstash error is non-specific: Got response code '400' contacting Elasticsearch

Full message:

[2017-02-23T09:11:01,281][INFO ][logstash.outputs.elasticsearch] Installing elasticsearch template to _template/flights
[2017-02-23T09:11:01,465][ERROR][logstash.outputs.elasticsearch] Failed to install template. {:message=>"Got response code '400' contacting Elasticsearch at URL 'https://***.eu-west-1.aws.found.io:9243/_template/flights'", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError", :backtrace=>["/usr/local/Cellar/logstash/5.2.1/libexec/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-6.2.6-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:76:in `perform_request'", "/usr/local/Cellar/logstash/5.2.1/libexec/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-6.2.6-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:273:in `perform_request_to_url'", "/usr/local/Cellar/logstash/5.2.1/libexec/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-6.2.6-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:261:in `perform_request'", "/usr/local/Cellar/logstash/5.2.1/libexec/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-6.2.6-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:351:in `with_connection'", "/usr/local/Cellar/logstash/5.2.1/libexec/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-6.2.6-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:260:in `perform_request'", "/usr/local/Cellar/logstash/5.2.1/libexec/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-6.2.6-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:268:in `put'", "/usr/local/Cellar/logstash/5.2.1/libexec/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-6.2.6-java/lib/logstash/outputs/elasticsearch/http_client.rb:303:in `template_put'", "/usr/local/Cellar/logstash/5.2.1/libexec/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-6.2.6-java/lib/logstash/outputs/elasticsearch/http_client.rb:79:in `template_install'", "/usr/local/Cellar/logstash/5.2.1/libexec/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-6.2.6-java/lib/logstash/outputs/elasticsearch/template_manager.rb:29:in `install'", "/usr/local/Cellar/logstash/5.2.1/libexec/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-6.2.6-java/lib/logstash/outputs/elasticsearch/template_manager.rb:9:in `install_template'", "/usr/local/Cellar/logstash/5.2.1/libexec/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-6.2.6-java/lib/logstash/outputs/elasticsearch/common.rb:54:in `install_template'", "/usr/local/Cellar/logstash/5.2.1/libexec/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-6.2.6-java/lib/logstash/outputs/elasticsearch/common.rb:21:in `register'", "/usr/local/Cellar/logstash/5.2.1/libexec/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:8:in `register'", "/usr/local/Cellar/logstash/5.2.1/libexec/logstash-core/lib/logstash/output_delegator.rb:37:in `register'", "/usr/local/Cellar/logstash/5.2.1/libexec/logstash-core/lib/logstash/pipeline.rb:234:in `start_workers'", "org/jruby/RubyArray.java:1613:in `each'", "/usr/local/Cellar/logstash/5.2.1/libexec/logstash-core/lib/logstash/pipeline.rb:234:in `start_workers'", "/usr/local/Cellar/logstash/5.2.1/libexec/logstash-core/lib/logstash/pipeline.rb:188:in `run'", "/usr/local/Cellar/logstash/5.2.1/libexec/logstash-core/lib/logstash/agent.rb:302:in `start_pipeline'"]}

When trying this directly in Elasticsearch with PUT _template, it gives errors that are telling me what to fix:

{
  "error": {
    "root_cause": [
      {
        "type": "mapper_parsing_exception",
        "reason": "Mapping definition for [Arrival] has unsupported parameters:  [WeatherDelay : {type=integer}] [WheelsOn : {type=keyword}]"
      }
    ],
    "type": "mapper_parsing_exception",
    "reason": "Failed to parse mapping [flights]: Mapping definition for [Arrival] has unsupported parameters:  [WeatherDelay : {type=integer}] [WheelsOn : {type=keyword}]",
    "caused_by": {
      "type": "mapper_parsing_exception",
      "reason": "Mapping definition for [Arrival] has unsupported parameters:  [WeatherDelay : {type=integer}] [WheelsOn : {type=keyword}]"
    }
  },
  "status": 400
}

omegagx commented 7 years ago

This is still an issue for me as well. Is there an ETA on a fix?

woodywalton commented 5 years ago

Still an issue communicating to Cloud

logstash-plugins / logstash-output-elasticsearch

Unclear and very verbose status 400 messages when Logstash hits a Elasticsearch mapping error during inserts #333