kube-logging / logging-operator

Logging operator for Kubernetes
https://kube-logging.dev
Apache License 2.0
1.53k stars 326 forks source link

Logzio plugin: Not rendering output_include_tags nor output_include_time #1723

Open kloudwrangler opened 4 months ago

kloudwrangler commented 4 months ago

Describe the bug: When creating a clusteroutput of type logz, I explicitly set the following fields

output_include_tags: false
output_include_time: false

However, the configuration produced by this CRD does not set these fields. This causes a problem as these fields default to true when not set.

https://github.com/logzio/fluent-plugin-logzio/blob/master/lib/fluent/plugin/out_logzio_buffered.rb#L15-L16

Here is the CRD

apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterOutput
metadata:
  name: logzio
  namespace: cattle-logging-system
spec:
  logz:
    buffer:
      chunk_limit_size: 16m
      flush_interval: 5s
      flush_mode: interval
      flush_thread_count: 4
      queue_limit_length: 4096
      type: file
    endpoint:
      port: 8071
      token:
        valueFrom:
          secretKeyRef:
            key: token
            name: logz-token
      url: https://listener-eu.logz.io
    output_include_tags: false
    output_include_time: false

Here is the relevant part of output

<match **>
    @type logzio_buffered
    @id clusterflow:cattle-logging-system:webapp:clusteroutput:cattle-logging-system:logzio
    endpoint_url https://listener-eu.logz.io:8071
    <buffer tag,time>
      @type file
      chunk_limit_size 16m
      flush_interval 5s
      flush_mode interval
      flush_thread_count 4
      path /buffers/clusterflow:cattle-logging-system:webapp:clusteroutput:cattle-logging-system:logzio.*.buffer
      queue_limit_length 4096
      retry_forever true
      timekey 10m
      timekey_wait 1m
    </buffer>
  </match>

Expected behaviour: When explicitly setting these fields in the ClusterOutput, it should produce a fluentd configuration with those fields explicitly set e.g.

<match **>
    @type logzio_buffered
    @id clusterflow:cattle-logging-system:webapp:clusteroutput:cattle-logging-system:logzio
    endpoint_url https://listener-eu.logz.io:8071
    output_include_tags false
    output_include_time false

Also, if not set, then it shall leave it blank as these fields will default to true if not set

Steps to reproduce the bug:

# Create a fake token
kubectl apply -f - <<EOF
apiVersion: v1
data:
  token: ZmFrZXRva2Vu
kind: Secret
metadata:
  name: logz-token
  namespace: cattle-logging-system
type: Generic
EOF

# Create a clusterflow of type logz.io 
kubectl apply -f - <<EOF
apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterOutput
metadata:
  name: logzio
  namespace: cattle-logging-system
spec:
  logz:
    buffer:
      chunk_limit_size: 16m
      flush_interval: 5s
      flush_mode: interval
      flush_thread_count: 4
      queue_limit_length: 4096
      type: file
    endpoint:
      port: 8071
      token:
        valueFrom:
          secretKeyRef:
            key: token
            name: logz-token
      url: https://listener-eu.logz.io
    output_include_tags: false
    output_include_time: false

EOF

# After some time, look at the produced configuration by looking at the secret
kubectl get secret logging-operator-fluentd-app -o jsonpath="{.data['fluentd\.conf']}" | base64 --decode | grep "output_include_tags"

Additional context: None

Environment details:

/kind bug

stale[bot] commented 2 months ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions!

stale[bot] commented 3 weeks ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions!