Missing request body when using bulk_message_request_threshold

[OranShuster]

• [X] read the contribution guideline

Problem

When using bulk_message_request_threshold to set a max on bulk request size, a few requests seems to not have a body my guess is that the buffer only contains records bigger than bulk_message_request_threshold so the request is empty

the following error message shows up in the logs

2022-10-12 14:51:33 +0000 [warn]: #0 [elasticearch_output] failed to flush the buffer. retry_times=3 next_retry_time=2022-10-12 14:51:42 +0000 chunk="5ead6ab0ace58a1ac1712bba5a3a4fcc" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster (fluentd-k8s-master-audit): [400] {\"error\":{\"root_cause\":[{\"type\":\"parse_exception\",\"reason\":\"request body is required\"}],\"type\":\"parse_exception\",\"reason\":\"request body is required\"},\"status\":400}"

Another problem with this is that this request is considered recoverable (like 413) so a lot of networks requests are going to be "wasted" on this

Steps to replicate

  <match **>
    @type elasticsearch_data_stream
    @id elasticearch_output
    data_stream_name fluentd-k8s-master-audit
    data_stream_template_name fluentd-k8s-master-audit-template
    data_stream_ilm_policy default-ilm
    hosts "#{ENV['ES_HOSTS']}"
    logstash_format false
    verify_es_version_at_startup false
    default_elasticsearch_version 7
    include_timestamp true
    request_timeout 20s
    bulk_message_request_threshold 40894464
    <buffer>
      @type file
      path /usr/share/fluentd-k8s-master/buffers/k8s-master-audit
      flush_thread_count 3
      chunk_limit_size 40mb
    </buffer>
  </match>

Expected Behavior or What you need to ask

All requests should have a body

Using Fluentd and ES plugin versions

• OS version N/A
• Kubernetes 1.20
• Fluentd v1.15
• ES plugin 5.2.4
• ES version 7.16.2