[out_es] Elasticsearch error

[doker-kg]

Hello! Deployed EFK and Curator in Kubernetes 1.10. Externally, everything is fine, but in the fluent-bit logs - [2018/06/05 12:21:00] [warn] [outes] Elasticsearch error {"took": 38, "errors": true, "items": [{"index": {" index": "logstash-2018.06.05", "_ type": "flbtype", " id": "Sjvkz2MBucitTSdgq3rn "," status ": 400," error ": {" type ":" mapper_parsing_exception "," reason ":" failed to parse "," caused_by ": {" type ":" i_o_exception "," reason ":" Duplicate field '@timestamp' \ n at [Source: org.elasticsearch.common.bytes.BytesReference$MarkSupportingStreamInputWrapper@31b917f0; line: 1, column: 1257] "}}}}]} and there is no data in the ES. root@kuberm2:/# kubectl get all -n logging NAME READY STATUS RESTARTS AGE pod/curator-1527984060-2fq6n 0/1 Completed 0 2d pod/curator-1528070460-6fkmn 0/1 Completed 0 1d pod/curator-1528156860-t6tgz 0/1 Completed 0 12h pod/es-client-db5cc477b-h4z8s 1/1 Running 0 6d pod/es-client-db5cc477b-x7khj 1/1 Running 0 6d pod/es-data-0 1/1 Running 0 5d pod/es-data-1 1/1 Running 0 5d pod/es-data-2 1/1 Running 0 5d pod/es-master-6985c67c44-lsz6n 1/1 Running 0 6d pod/es-master-6985c67c44-rhtnp 1/1 Running 0 6d pod/es-master-6985c67c44-w5sx4 1/1 Running 0 6d pod/fluent-bit-6csqn 1/1 Running 0 6d pod/fluent-bit-qxprd 1/1 Running 0 6d pod/fluent-bit-t8td7 1/1 Running 0 6d pod/fluent-bit-v2g4h 1/1 Running 0 6d pod/kibana-8497d64b4d-fvkz9 1/1 Running 0 5d pod/nettools-pod 1/1 Running 0 6d

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/elasticsearch ClusterIP 10.101.217.123 9200/TCP 6d service/elasticsearch-discovery ClusterIP None 9300/TCP 6d service/kibana ClusterIP 10.110.156.223 80/TCP 6d

NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/fluent-bit 4 4 4 4 4 6d

NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE deployment.apps/es-client 2 2 2 2 6d deployment.apps/es-master 3 3 3 3 6d deployment.apps/kibana 1 1 1 1 5d

NAME DESIRED CURRENT READY AGE replicaset.apps/es-client-db5cc477b 2 2 2 6d replicaset.apps/es-master-6985c67c44 3 3 3 6d replicaset.apps/kibana-8497d64b4d 1 1 1 5d

NAME DESIRED CURRENT AGE statefulset.apps/es-data 3 3 5d

NAME DESIRED SUCCESSFUL AGE job.batch/curator-1527984060 1 1 2d job.batch/curator-1528070460 1 1 1d job.batch/curator-1528156860 1 1 12h

NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE cronjob.batch/curator 1 0 * False 0 12h 4d

Help me please

[doker-kg]

curl -X DELETE 'http://10.101.217.123:9200/_all' did not help this is fluent-bit from the workers node, errors only on worker node

[pires]

The error is clear.

Duplicate field '@timestamp'

Whatever is producing to Elasticsearch is doing it wrongly. You should look for issues in that fluentbit component.