elasticsearch bulk insert response with errors: true is truncated

[FarhadF]

Bug Report

Describe the bug

elasticsearch bulk insert response with errors: true is truncated

Example truncated response:

{"took":95,"errors":true,"items":[{"index":{"_index":"cwl-2020.05.04","_type":"_doc","_id":"wwBj4HEBjqrEiEstJBXR","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":14288318,"_primary_term":2,"status":201}},{"index":{"_index":"cwl-2020.05.04","_type":"_doc","_id":"xABj4HEBjqrEiEstJBXR","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":14294117,"_primary_term":2,"status":201}},{"index":{"_index":"cwl-2020.05.04","_type":"_doc","_id":"xQBj4HEBjqrEiEstJBXR","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":14297702,"_primary_term":2,"status":201}},{"index":{"_index":"cwl-2020.05.04","_type":"_doc","_id":"xgBj4HEBjqrEiEstJBXR","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":14294127,"_primary_term":2,"status":201}},{"index":{"_index":"cwl-2020.05.04","_type":"_doc","_id":"xwBj4HEBjqrEiEstJBXR","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":14285118,"_primary_term":2,"status":201}},{"index":{"_index":"cwl-2020.05.04","_type":"_doc","_id":"yABj4HEBjqrEiEstJBXR","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":14288323,"_primary_term":2,"status":201}},{"index":{"_index":"cwl-2020.05.04","_type":"_doc","_id":"yQBj4HEBjqrEiEstJBXR","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":14276391,"_primary_term":2,"status":201}},{"index":{"_index":"cwl-2020.05.04","_type":"_doc","_id":"ygBj4HEBjqrEiEstJBXR","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":14294131,"_primary_term":2,"status":201}},{"index":{"_index":"cwl-2020.05.04","_type":"_doc","_id":"ywBj4HEBjqrEiEstJBXR","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":14288326,"_primary_term":2,"status":201}},{"index":{"_index":"cwl-2020.05.04","_type":"_doc","_id":"zABj4HEBjqrEiEstJBXR","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":14276402,"_primary_term":2,"status":201}},{"index":{"_index":"cwl-2020.05.04","_type":"_doc","_id":"zQBj4HEBjqrEiEstJBXR","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":14276405,"_primary_term":2,"status":201}},{"index":{"_index":"cwl-2020.05.04","_type":"_doc","_id":"zgBj4HEBjqrEiEstJBXR","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":14288338,"_primary_term":2,"status":201}},{"index":{"_index":"cwl-2020.05.04","_type":"_doc","_id":"zwBj4HEBjqrEiEstJBXR","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":14285120,"_primary_term":2,"status":201}},{"index":{"_index":"cwl-2020.05.04","_type":"_doc","_id":"0ABj4HEBjqrEiEstJBXR","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":14276408,"_primary_term":2,"status":201}},{"index":{"_index":"cwl-2020.05.04","_type":"_doc","_id":"0QBj4HEBjqrEiEstJBXR","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":14288341,"_primary_term":2,"status":201}},{"index":{"_index":"cwl-2020.05.04","_type":"_doc","_id":"0gBj4HEBjqrEiEstJBXR","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":14288343,"_primary_term":2,"status":201}},{"index":{"_index":"cwl-2020.05.04","_type":"_doc","_id":"0wBj4HEBjqrEiEstJBXR","_version":1,"result":"created","_s
2020-05-04 17:52:06
[2020/05/04 15:52:06] [ warn] [engine] failed to flush chunk '1-1588607521.163150373.flb', retry in 11 seconds: task_id=0, input=forward.3 > output=es.0

Example error:

[error] [output:es:es.0] could not pack/validate JSON response

Config:

[INPUT]
    Name forward
    unix_path /var/run/fluent.sock
[INPUT]
    Name tcp
    Tag firelens-healthcheck
    Listen 127.0.0.1
    Port 8878
[SERVICE]
    Parsers_File /fluent-bit/parsers/parsers.conf
    Grace 30[FILTER]
    Name parser
    Match ${environment}-${task_name}-firelens*
    Key_Name log
    Parser json
    Reserve_Data True
[OUTPUT]
    Name es
    Match ${environment}-${task_name}-firelens*
    Host ${es_endpoint}
    Logstash_Prefix cwl-${environment}-${task_name}
    Logstash_Format On
    Type _doc
    Buffer_Size 16KB
[OUTPUT]
    Name null
    Match firelens-healthcheck

Expected behavior After increasing the buffer_size i assume it should log the whole response.

[FarhadF]

@edsiper could you please advise?

[edsiper]

@FarhadF increase the value of buffer_size so you retrieve a larger response from Elastic:

https://docs.fluentbit.io/manual/pipeline/outputs/elasticsearch#configuration-parameters

[edsiper]

(increase it even more :)

[FarhadF]

Thanks for the feedback, I tried the solution but I only get [2020/07/29 15:59:16] [ warn] [engine] failed to flush chunk '1-1596038351.4012953.flb', retry in 9 seconds: task_id=1, input=forward.3 > output=es.0 logs and no more the stated logs in this issue. And they get to the point that they cant be retried:

[ warn] [engine] chunk '1-1596038516.195649786.flb' cannot be retried: task_id=1, input=forward.3 > output=es.0

[OUTPUT]
    Name es
    Match *
    Buffer_Size 512KB
    Host XXX
    Logstash_Prefix XXX
    Logstash_Format On
    Type _doc

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.