Elasticsearch_data_stream: Fluentd hangs if no Elasticsearch connection available during start

[moondly]

Problem

When using elasticsearch_data_stream the workaround provided in https://github.com/uken/fluent-plugin-elasticsearch/issues/596 does not work and Fluentd hangs after the error below:

2021-12-28 11:16:27 +0000 [error]: config error file="/fluentd/etc/td-agent.conf" error_class=Fluent::ConfigError error="Failed to create data stream: <elastic_other> connect_write timeout reached"

Steps to replicate

<match **>
  @type copy
  <store>
    @type stdout
  </store>
  <store ignore_error>
    @id elastic_other
    @type elasticsearch_data_stream
    data_stream_name nco-logs-other
    host localhost
    port 1234
    verify_es_version_at_startup false
    default_elasticsearch_version 7
    max_retry_get_es_version 1
    max_retry_putting_template 1
     <buffer>
       @type file
     </buffer>
  </store>
</match>

Expected Behavior

It should send logs to stdout while trying to connect to elasticsearch and do not force Fluentd to exit

Using Fluentd and ES plugin versions

Image created with the following Dockerfile:

FROM docker.io/fluent/fluentd:v1.14-debian

# Use root account to use apt
USER root

# below RUN includes plugin as examples elasticsearch is not required
# you may customize including plugins as you wish
RUN buildDeps="sudo make gcc g++ libc-dev" \
  && apt-get update \
  && apt-get install -y --no-install-recommends $buildDeps \
  && sudo gem install --no-document \
  fluent-plugin-elasticsearch:5.1.4 \
  fluent-plugin-prometheus:2.0.2 \
  fluent-logger:0.9.0 \
  elasticsearch-api:7.16.1 \
  elasticsearch-xpack:7.16.1 \
  fluent-plugin-record-modifier:2.1.0 \
  && sudo gem sources --clear-all \
  && SUDO_FORCE_REMOVE=yes \
  apt-get purge -y --auto-remove \
  -o APT::AutoRemove::RecommendsImportant=false \
  $buildDeps \
  && rm -rf /var/lib/apt/lists/* \
  && rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem

USER fluent