Mixed log entities in logtrail

Hi! i have 1 pod who generate logs. this pod based on BusyBox and execute following shell command i=0; while true; do echo "$i: $(date)"; i=$((i+1)); sleep 1 ; done

so when i open kubectl logs & docker logs command the output have strong order of numbers

0: Thu Jun 20 10:13:35 UTC 2019
1: Thu Jun 20 10:13:36 UTC 2019
2: Thu Jun 20 10:13:37 UTC 2019
3: Thu Jun 20 10:13:38 UTC 2019
4: Thu Jun 20 10:13:39 UTC 2019
5: Thu Jun 20 10:13:40 UTC 2019
6: Thu Jun 20 10:13:41 UTC 2019
7: Thu Jun 20 10:13:42 UTC 2019
8: Thu Jun 20 10:13:43 UTC 2019
9: Thu Jun 20 10:13:44 UTC 2019
10: Thu Jun 20 10:13:45 UTC 2019
11 etc....

but when i see logs from my pod in kibana, the logs appears in very very mixed way

and it looks like

79: Thu Jun 20 10:14:55 UTC 2019
 89: Thu Jun 20 10:15:05 UTC 2019
 91: Thu Jun 20 10:15:07 UTC 2019
120: Thu Jun 20 10:15:36 UTC 2019
 118: Thu Jun 20 10:15:34 UTC 2019
80: Thu Jun 20 10:14:56 UTC 2019
 39: Thu Jun 20 10:14:14 UTC 2019
  50: Thu Jun 20 10:14:26 UTC 2019
33: Thu Jun 20 10:14:08 UTC 2019
 8: Thu Jun 20 10:13:43 UTC 2019
 30: Thu Jun 20 10:14:05 UTC 2019
 15: Thu Jun 20 10:13:50 UTC 2019
 17: Thu Jun 20 10:13:52 UTC 2019
14: Thu Jun 20 10:13:49 UTC 2019
 25: Thu Jun 20 10:14:00 UTC 2019
 4: Thu Jun 20 10:13:39 UTC 2019
1: Thu Jun 20 10:13:36 UTC 2019
 3: Thu Jun 20 10:13:38 UTC 2019

so u can see it mixed, and how to solve this problem? i use ElasticSearch 6.2.3 and logtrail 6.2.3 and Fluent-bit 1.1.2

my logtrail.json looks like

{
  "index_patterns" : [
    {
      "es": {
        "default_index": "logstash*",
        "allow_url_parameter": false
      },
      "tail_interval_in_seconds": 2,
      "es_index_time_offset_in_seconds": 0,
      "display_timezone": "local",
      "display_timestamp_format": "MMM DD HH:mm:ss",
      "max_buckets": 500,
      "nested_objects" : true,
      "default_time_range_in_days" : 0,
      "fields" : {
        "mapping" : {
            "timestamp" : "@timestamp",
            "display_timestamp" : "@timestamp",
            "program": "kubernetes.container_name",
            "message": "log"
        }
      }
    }
  ]
}

P.S I think this is not fluent-bit or fluentd issue because i was try both, and the problem still exist

sivasamyk / logtrail

Mixed log entities in logtrail #354