Field [_id] is a metadata field and cannot be added inside a document

[TaurusD]

I'm using Elasticsearch 5.4, Logstash 5.4.2, MongoDB 3.4.5. When I import from MongoDB to Elasticsearch through Logstash, I found the following error:

[2017-07-11T16:57:06,180][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"mongodb-2017.07.11", :_type=>"logs", :_routing=>nil}, 2017-07-11T08:57:03.744Z %{message}], :response=>{"index"=>{"_index"=>"mongodb-2017.07.11", "_type"=>"logs", "_id"=>"AV0w3bobj2yUz3xTJdU4", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"Field [_id] is a metadata field and cannot be added inside a document. Use the index API request parameters."}}}}

It mentioned that there was a conflict between the name of [_id] meta field in Kibana and [_id] meta column of MongoDB. How should I solve this conflict in my logstash pipeline config? I think this is a critical problem between ES and Mongo, how did this plugin work before?

Thank you!

[TaurusD]

Well, I'm back for this short moment cause I solved the above problem by just changing the column name from [_id] to other names with mutate filter. And I found another problem after I imported data. Please let me close this issue and open another one for a different topic. Thanks.

[manojsb]

@TaurusD Can you explain please how did you make it work ? and how did you setup this ?

[TaurusD]

@manojsb Hi, I just used mutate filter like this:

filter { mutate{ rename => ["_id", "Obj_id"] } }

This may import the data from _id in Mongo to Obj_id field in Elasticsearch, so there won't be any conflict with the meta field [_id] of Elasticsearch. Obj_id is a field you can define by yourself. In fact, I noticed that this mongo-input plugin do copy the _id column to a new field called [mongo_id] while importing, but it may 'forget' to remove import the original _id so that it caused conflict.

[manojsb]

@TaurusD thank you so much for explanation. Just one more question are using it in production?

[TaurusD]

@manojsb no, I just use it for some testing. no production

[manojsb]

@TaurusD thanks for feedback

[ansballard]

Just ran into this as well. parse_method => "dig" correctly ignores the _id field, since it has an explicit check at Line 317. A fix would be pretty straightforward, just need to add the same check to the flatten and simple parse_methods. For now I'm going to use dig, since I'm mostly researching at this point.