failed to parse field as object, but found a concrete value

[cdenneen]

Logstash 6.4.2 logstash-codec-cloudtrail (3.0.5) still has same problem (#15).

responseElements and requestParameters (and their children like requestParameters.iamInstanceProfile)

[2018-10-23T20:26:49,297][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-cloudtrail-write", :_type=>"_doc", :_routing=>nil}, #<LogStash::Event:0x2b804bc4>], :response=>{"index"=>{"_index"=>"logstash-cloudtrail-000004", "_type"=>"_doc", "_id"=>"H-2bomYBuhFkxAoaxCUs", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"object mapping for [responseElements.role] tried to parse field [role] as object, but found a concrete value"}}}}
[2018-10-23T20:28:28,793][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-cloudtrail-write", :_type=>"_doc", :_routing=>nil}, #<LogStash::Event:0x4cc6643b>], :response=>{"index"=>{"_index"=>"logstash-cloudtrail-000004", "_type"=>"_doc", "_id"=>"gO6domYBuhFkxAoaSECE", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"object mapping for [requestParameters.iamInstanceProfile] tried to parse field [iamInstanceProfile] as object, but found a concrete value"}}}}

[cdenneen]

@jsvd @robbavey Looks like we've had issues with these fields being parsed and mapped properly for > 3 years now. Can we have someone look into way to fix this? I'll even POST a custom template/mapping to fix these issues if that's the only solution.

[kares]

Just to chime in for anyone landing here, the codec parses JSON records and does a few tricks to adjust the layout.

One of the ways forward here would be to adapt the plugin to Elastic Common Schema and leverage a predictable flat layout (similar to the one Beats has been using). This would be a separate plugin execution mode, similar to other codecs, as changing the current event layout might break backwards compatibility others might be relying upon.

The plugin mostly relies on support from the community and the last related update has been done by a plugin user.