[BUG] [parser /container] Error with Reduced max_log_size in OpenTelemetry Collector's Filelog Receiver

[Masmiiadm]

Component(s)

pkg/stanza

What happened?

Description

I based the new log parsing setup for the OpenTelemetry Collector on this documentation, and everything worked perfectly with the default configuration.

However, when I tried reducing the max_log_size parameter in the filelogreceiver configuration from its default value of 1 MiB to 40 KiB, I encountered the following error "data_type": "logs", "component": "fileconsumer", "path": "/var/log/pods/kiali-operator_kiali-kiali-operator-5f5f969b89-7vqcx_bd93551f-ff0e-413e-95b1-4fd8420f2690/operator/0.log", "error": "failed to detect a valid container log format: entry cannot be parsed as container logs" "deployment\": {\".\": {}, \"f:instanceName\": {}, \"f:namespace\": {}}, \"f:environment\": {\".\": {}, \"f:isKubernetes\": {}}, \"f....

Expected Result

the fileconsumer parses these logs successfully like the 1MiB default configuration

Actual Result

the filelog receiver and th container parser failed to detect a valid container log format when configure max_log_size 40 KiB

Collector version

0.111.0

Environment information

No response

OpenTelemetry Collector configuration

apiVersion: v1
data:
  relay: |
    exporters:
      debug: {}
    extensions:
      health_check:
        endpoint: ${env:MY_POD_IP}:13133
    processors:
      batch: {}
      k8sattributes:
        extract:
          metadata:
          - k8s.namespace.name
          - k8s.deployment.name
          - k8s.statefulset.name
          - k8s.daemonset.name
          - k8s.cronjob.name
          - k8s.job.name
          - k8s.node.name
          - k8s.pod.name
          - k8s.pod.uid
          - k8s.pod.start_time
        filter:
          node_from_env_var: K8S_NODE_NAME
        passthrough: false
        pod_association:
        - sources:
          - from: resource_attribute
            name: k8s.pod.ip
        - sources:
          - from: resource_attribute
            name: k8s.pod.uid
        - sources:
          - from: connection
      memory_limiter:
        check_interval: 5s
        limit_percentage: 80
        spike_limit_percentage: 25
    receivers:
      filelog:
        exclude:
        - /var/log/pods/*/opentelemetry-collector/*.log
        include:
        - /var/log/pods/*/*/*.log
        include_file_name: false
        include_file_path: true
        max_log_size: 40KiB
        operators:
        - id: container-parser
          type: container
        retry_on_failure:
          enabled: true
        start_at: end
    service:
      extensions:
      - health_check
      pipelines:
        logs:
          exporters:
          - debug
          processors:
          - transform
          - k8sattributes
          - memory_limiter
          - resource
          - batch
          receivers:
          - filelog
      telemetry:
        logs:
          level: info
        metrics:
          address: ${env:MY_POD_IP}:8888

Log output

2024-10-28T10:46:19.065Z  error reader/reader.go:148  process: %w {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/var/log/pods/kiali-operator_kiali-kiali-operator-5f5f969b89-7vqcx_bd93551f-ff0e-413e-95b1-4fd8420f2690/operator/0.log", "error": "failed to detect a valid container log format: entry cannot be parsed as container logs: [\\\"openid\\\", \\\"profile\\\", \\\"email\\\"], \\\"username_claim\\\": \\\"sub\\\"}, \\\"openshift\\\": {\\\"client_id_prefix\\\": \\\"kiali\\\"}, \\\"strategy\\\": \\\"anonymous\\\"}, \\\"custom_dashboards\\\": [], \\\"deployment\\\": {\\\"accessible_namespaces\\\": [\\\"**\\\"], \\\"affinity\\\": {\\\"node\\\": {}, \\\"pod\\\": {}, \\\"pod_anti\\\": {}}, \\\"hpa\\\":

Additional context

No response

[github-actions[bot]]

Pinging code owners:

• pkg/stanza: @djaglowski

  See Adding Labels via Comments if you do not have permissions to add labels yourself.

[djaglowski]

cc @ChrsMark

[ChrsMark]

Hey @Masmiiadm, could you share a sample log line that causes that failure? From what I can see in the Log Output section of the issue, the entry does not really look like a valid format? Or what I see is the trimmed log line?

I assume that when the max_log_size is set the log lines can be trimmed. In that case the container format might have been corrupted and seeing the parser failing actually makes sense.

[Masmiiadm]

Hi @ChrsMark . you will find the file 0.log attached 0.log

[ChrsMark]

Hi @ChrsMark . you will find the file 0.log attached 0.log

The link does not work for me. Just a single log line in a comment will be fine, no need to share the whole log file :).