Fluentd file buffering does not work with logz plugin

[ekhaydarov]

Describe the bug: By default logz_buffering is enabled. However, /buffers persistent volumes are created by default too which idles. Manually setting buffer to file starts writing buffers to the /buffers volume. However, data that is sent is not then flushed so the files just build up. Having turned log level to debug there is no error or warning in the /fluentd/logs/out.

Expected behaviour: Default chart template should either be memory buffers and persistent volume disabled or file buffers enable and persistent volume enabled. Then file buffers should send to logz and flush the .buffer and .buffer.meta files

Steps to reproduce the bug: Install logging-operator-logging 3.9.2. then set file buffers for logz clusteroutput

clusterOutputs:
  - name: logz
    spec:
      logz:
        buffer:
          type: file
        endpoint:
          url: https://listener-nl.logz.io
          port: 8071

Additional context: Add any other context about the problem here.

Environment details:

• Kubernetes version (e.g. v1.15.2): 1.20.2
• Cloud-provider/provisioner (e.g. AKS, GKE, EKS, PKE etc): AKS ( i know sorry)
• logging-operator version (e.g. 2.1.1): 3.9.2
• Install method (e.g. helm or static manifests): helm
• Logs from the misbehaving component (and any other relevant logs):

  # Logfile created on 2021-04-27 09:51:50 +0000 by logger.rb/v1.4.2
  2021-04-27 09:51:50 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
  2021-04-27 09:51:50 +0000 [info]: gem 'fluent-mixin-config-placeholders' version '0.4.0'
  2021-04-27 09:51:50 +0000 [info]: gem 'fluent-plugin-aws-elasticsearch-service' version '2.4.0'
  2021-04-27 09:51:50 +0000 [info]: gem 'fluent-plugin-azure-storage-append-blob' version '0.2.1'
  2021-04-27 09:51:50 +0000 [info]: gem 'fluent-plugin-cloudwatch-logs' version '0.13.4'
  2021-04-27 09:51:50 +0000 [info]: gem 'fluent-plugin-concat' version '2.4.0'
  2021-04-27 09:51:50 +0000 [info]: gem 'fluent-plugin-datadog' version '0.13.0'
  2021-04-27 09:51:50 +0000 [info]: gem 'fluent-plugin-dedot_filter' version '1.0.0'
  2021-04-27 09:51:50 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.13'
  2021-04-27 09:51:50 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.0.2'
  2021-04-27 09:51:50 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '4.3.3'
  2021-04-27 09:51:50 +0000 [info]: gem 'fluent-plugin-enhance-k8s-metadata' version '2.0.0'
  2021-04-27 09:51:50 +0000 [info]: gem 'fluent-plugin-gcs' version '0.4.0'
  2021-04-27 09:51:50 +0000 [info]: gem 'fluent-plugin-gelf-hs' version '1.0.8'
  2021-04-27 09:51:50 +0000 [info]: gem 'fluent-plugin-geoip' version '1.3.2'
  2021-04-27 09:51:50 +0000 [info]: gem 'fluent-plugin-grafana-loki' version '1.2.16'
  2021-04-27 09:51:50 +0000 [info]: gem 'fluent-plugin-kafka' version '0.16.0'
  2021-04-27 09:51:50 +0000 [info]: gem 'fluent-plugin-kinesis' version '3.3.0'
  2021-04-27 09:51:50 +0000 [info]: gem 'fluent-plugin-kubernetes-metadata-filter' version '2.5.3'
  2021-04-27 09:51:50 +0000 [info]: gem 'fluent-plugin-kubernetes-sumologic' version '2.0.0'
  2021-04-27 09:51:50 +0000 [info]: gem 'fluent-plugin-label-router' version '0.2.4'
  2021-04-27 09:51:50 +0000 [info]: gem 'fluent-plugin-logdna' version '0.4.0'
  2021-04-27 09:51:50 +0000 [info]: gem 'fluent-plugin-logzio' version '0.0.21'
  2021-04-27 09:51:50 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
  2021-04-27 09:51:50 +0000 [info]: gem 'fluent-plugin-newrelic' version '1.1.10'
  2021-04-27 09:51:50 +0000 [info]: gem 'fluent-plugin-oss' version '0.0.2'
  2021-04-27 09:51:50 +0000 [info]: gem 'fluent-plugin-parser-logfmt' version '0.0.0'
  2021-04-27 09:51:50 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.0.0'
  2021-04-27 09:51:50 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.0'
  2021-04-27 09:51:50 +0000 [info]: gem 'fluent-plugin-redis' version '0.3.5'
  2021-04-27 09:51:50 +0000 [info]: gem 'fluent-plugin-remote-syslog' version '1.1'
  2021-04-27 09:51:50 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
  2021-04-27 09:51:50 +0000 [info]: gem 'fluent-plugin-s3' version '1.5.1'
  2021-04-27 09:51:50 +0000 [info]: gem 'fluent-plugin-splunk-hec' version '1.2.5'
  2021-04-27 09:51:50 +0000 [info]: gem 'fluent-plugin-sumologic_output' version '1.7.2'
  2021-04-27 09:51:50 +0000 [info]: gem 'fluent-plugin-syslog_rfc5424' version '0.9.0.rc.7'
  2021-04-27 09:51:50 +0000 [info]: gem 'fluent-plugin-tag-normaliser' version '0.1.1'
  2021-04-27 09:51:50 +0000 [info]: gem 'fluent-plugin-throttle' version '0.0.5'
  2021-04-27 09:51:50 +0000 [info]: gem 'fluent-plugin-webhdfs' version '1.4.0'
  2021-04-27 09:51:50 +0000 [info]: gem 'fluentd' version '1.11.5'
  2021-04-27 09:51:51 +0000 [info]: using configuration file: <ROOT>
  <system>
  rpc_endpoint "127.0.0.1:24444"
  log_level info
  workers 1
  </system>
  <source>
  @type prometheus
  port 2020
  metrics_path "/api/v1/metrics/prometheus"
  </source>
  <source>
  @type prometheus_monitor
  </source>
  <source>
  @type prometheus_output_monitor
  </source>
  <source>
  @type forward
  @id main_forward
  bind "0.0.0.0"
  port 24240
  </source>
  <match **>
  @type label_router
  @id main
  metrics true
  <route>
    @label "@0ec98e263bfb977f32f22041399e7a66"
    metrics_labels {"id":"clusterflow:data-monitoring:all"}
    <match>
      negate false
    </match>
  </route>
  </match>
  <label @0ec98e263bfb977f32f22041399e7a66>
  <filter **>
    @type record_modifier
    @id clusterflow:data-monitoring:all:0
    <record>
      cluster_name we1-dev-data-k8s-services
    </record>
  </filter>
  <match **>
    @type logzio_buffered
    @id clusterflow:data-monitoring:all:clusteroutput:data-monitoring:logz
    endpoint_url "https://listener-nl.logz.io:8071?token=xxx"
    <buffer tag,time>
      @type "file"
      flush_mode immediate
      path "/buffers/clusterflow:data-monitoring:all:clusteroutput:data-monitoring:logz.*.buffer"
      retry_forever true
      timekey 10m
      timekey_wait 10m
    </buffer>
  </match>
  </label>
  <label @ERROR>
  <match **>
    @type null
    @id main-fluentd-error
  </match>
  </label>
  <match **>
  @type null
  @id main-no-output
  </match>
  <label @FLUENT_LOG>
  <match fluent.*>
    @type null
    @id main-fluentd-log
  </match>
  </label>
  </ROOT>
  2021-04-27 09:51:51 +0000 [info]: starting fluentd-1.11.5 pid=6 ruby="2.7.2"
  2021-04-27 09:51:51 +0000 [info]: spawn command to main:  cmdline=["/usr/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/bin/fluentd", "-o", "/fluentd/log/out", "--log-rotate-age", "10", "--log-rotate-size", "10485760", "-c", "/fluentd/etc/fluent.conf", "-p", "/fluentd/plugins", "--under-supervisor"]
  2021-04-27 09:51:52 +0000 [info]: adding filter in @0ec98e263bfb977f32f22041399e7a66 pattern="**" type="record_modifier"
  2021-04-27 09:51:52 +0000 [info]: adding match in @0ec98e263bfb977f32f22041399e7a66 pattern="**" type="logzio_buffered"
  2021-04-27 09:51:52 +0000 [info]: adding match in @FLUENT_LOG pattern="fluent.*" type="null"
  2021-04-27 09:51:52 +0000 [info]: adding match in @ERROR pattern="**" type="null"
  2021-04-27 09:51:52 +0000 [info]: adding match pattern="**" type="label_router"
  2021-04-27 09:51:52 +0000 [info]: adding match pattern="**" type="null"
  2021-04-27 09:51:52 +0000 [info]: adding source type="prometheus"
  2021-04-27 09:51:52 +0000 [info]: adding source type="prometheus_monitor"
  2021-04-27 09:51:52 +0000 [info]: adding source type="prometheus_output_monitor"
  2021-04-27 09:51:52 +0000 [info]: adding source type="forward"
  2021-04-27 09:51:52 +0000 [warn]: parameter 'metrics' in <match **>
  @type label_router
  @id main
  metrics true
  <route>
  @label "@0ec98e263bfb977f32f22041399e7a66"
  metrics_labels {"id":"clusterflow:data-monitoring:all"}
  <match>
    negate false
  </match>
  </route>
  </match> is not used.
  2021-04-27 09:51:52 +0000 [warn]: parameter 'metrics_labels' in <route>
  @label "@0ec98e263bfb977f32f22041399e7a66"
  metrics_labels {"id":"clusterflow:data-monitoring:all"}
  <match>
  negate false
  </match>
  </route> is not used.
  2021-04-27 09:51:52 +0000 [info]: #0 starting fluentd worker pid=15 ppid=6 worker=0
  2021-04-27 09:52:07 +0000 [info]: #0 [main_forward] listening port port=24240 bind="0.0.0.0"
  2021-04-27 09:52:07 +0000 [info]: #0 fluentd worker is now running worker=0

• Resource definition (possibly in YAML format) that caused the issue, without sensitive data:

/kind bug

[pepov]

There are two separate issues here:

• if buffers are not defined for an output, the logging operator doesn't help setting up the buffer automatically to use filesystem (instead defaulting to memory)
• the above buffer config doesn't remove buffer chunks and metadata after flushed

As of the first, I would create a separate issue. As of the second I beleive that is caused by the timekey/timekey_wait config messing with flush_mode immediate. If you set flush_mode immediate I would recommend setting timekey 1s and timekey_wait 0s or something like that explicitly.