'chunk bytes limit exceeds for an emitted event stream' with chunk_limit_size set

[kefiras]

Bugs should be filed for issues encountered whilst operating logging-operator. You should first attempt to resolve your issues through the community support channels, e.g. Slack, in order to rule out individual configuration errors. #logging-operator Please provide as much detail as possible.

Describe the bug: I am seeing a lot of 'chunk bytes limit exceeds for an emitted event stream' even when the chunk_limit_size is set. I suspect this may happen in certain circumstances but I don't understand when and why.

Expected behaviour: message size shouldn't be more than chunk_limit_size set

Steps to reproduce the bug: Run logging operator with Kafka output.

Additional context: N/A

Environment details:

• Kubernetes version (e.g. v1.15.2): 1.28.9
• Cloud-provider/provisioner (e.g. AKS, GKE, EKS, PKE etc): AKS
• logging-operator version (e.g. 2.1.1): 4.6.0
• Install method (e.g. helm or static manifests): helm
• Logs from the misbehaving component (and any other relevant logs):
• Resource definition (possibly in YAML format) that caused the issue, without sensitive data:

clusterOutputs:
  - name: kafka
    spec:
      kafka:
        brokers: eventhub.servicebus.windows.net:9093
        buffer:
          chunk_limit_size: 512KB
          disable_chunk_backup: true
          flush_thread_count: 8
          retry_forever: false
          retry_timeout: 3h
          timekey: 30s
          timekey_wait: 0s
        default_topic: aks
        format:
          type: json
        password:
          value: password
        sasl_over_ssl: false
        ssl_ca_certs_from_system: true
        ssl_verify_hostname: false
        topic_key: aks
        username:
          value: $ConnectionString

/kind bug

[pepov]

In the above configuration you buffer messages until 30 seconds until they are written into a single chunk. The default for chunk_limit_size is 8MB even for a memory buffer, 256MB for file buffer. In logging operator version 4.6.0 the setting for file buffers was also set to 8MB, but that was removed in 4.7.0 with https://github.com/kube-logging/logging-operator/pull/1729 which will use fluentd's default 256MB.

I would recommend setting chunk_limit_size to something bigger than 8MB or use a smaller time window by reducing the timekey value.

[kefiras]

Thanks for reply. The thing is that Azure Event Hub maximum message size is 1MB. Looks like the only thing that can be done is to set relatively small time window.