uken / fluent-plugin-elasticsearch

Apache License 2.0
890 stars 310 forks source link

bulk api filter_path=items.*.error support #975

Closed jiachinzhao closed 2 years ago

jiachinzhao commented 2 years ago

from here, es bulk api, support filter_path to filter response which some docs are insert correct when bulk inserted failed

examples a error response Could not bulk insert to Data Stream: nginx-main-ext-access-log-v1 {"took"=>4, "errors"=>true, "items"=>[{"create"=>{"_index"=>".ds-nginx-main-ext-access-log-v1-2022.05.16-000001", "_type"=>"_doc", "_id"=>"XwX54IABKbDZYVhbWSKH", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [upstream_status] of type [integer] in document with id 'XwX54IABKbDZYVhbWSKH'. Preview of field's value: '-'", "caused_by"=>{"type"=>"number_format_exception", "reason"=>"For input string: \"-\""}}}}, {"create"=>{"_index"=>".ds-nginx-main-ext-access-log-v1-2022.05.16-000001", "_type"=>"_doc", "_id"=>"YAX54IABKbDZYVhbWSKH", "_version"=>1, "result"=>"created", "_shards"=>{"total"=>2, "successful"=>2, "failed"=>0}, "_seq_no"=>56602, "_primary_term"=>1, "status"=>201}}]} doc_id: XwX54IABKbDZYVhbWSKH has error doc_id: YAX54IABKbDZYVhbWSKH insert ok

when bulk size is large, error response is also too large. so response maybe just show infos which docs insert failed.

github-actions[bot] commented 2 years ago

@jiachinzhao this issue was automatically closed because it did not follow the issue template.