Filelog receiver log mount points

[paraniraja]

Hi,

Using opentelmetry-operator, how to mount /var/log/pods to daemonset?

Tried with version "0.68.1", only conf file is mounted.

Thanks

[dosubot[bot]]

Hey @paraniraja, I'm here to help you with your coding conundrums! Let's squash those bugs together! 🤖

To mount /var/log/pods to a DaemonSet using the OpenTelemetry operator, you need to modify the DaemonSet configuration to include the necessary volume and volumeMounts. Here is an example configuration:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: opentelemetry-collector
  namespace: default
spec:
  selector:
    matchLabels:
      name: opentelemetry-collector
  template:
    metadata:
      labels:
        name: opentelemetry-collector
    spec:
      containers:
        - name: opentelemetry-collector
          image: otel/opentelemetry-collector:0.68.1
          volumeMounts:
            - name: varlogpods
              mountPath: /var/log/pods
              readOnly: true
            - name: varlibdockercontainers
              mountPath: /var/lib/docker/containers
              readOnly: true
      volumes:
        - name: varlogpods
          hostPath:
            path: /var/log/pods
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers

This configuration mounts the /var/log/pods directory from the host to the /var/log/pods directory in the container, allowing the OpenTelemetry collector to access the logs.

Additionally, you may want to configure the Filelog Receiver to collect logs from these mounted directories. Here is an example configuration for the Filelog Receiver:

receivers:
  filelog:
    include:
      - /var/log/pods/*/*/*.log
    exclude:
      - /var/log/pods/*/otel-collector/*.log
    start_at: beginning
    include_file_path: true
    include_file_name: false
    operators:
      - type: router
        id: get-format
        routes:
          - output: parser-docker
            expr: 'body matches "^\\{"'
          - output: parser-crio
            expr: 'body matches "^[^ Z]+ "'
          - output: parser-containerd
            expr: 'body matches "^[^ Z]+Z"'
      - type: regex_parser
        id: parser-crio
        regex: '^(?P<time>[^ Z]+) (?P<stream>stdout|stderr) (?P<logtag>[^ ]*) ?(?P<log>.*)$'
        output: extract_metadata_from_filepath
        timestamp:
          parse_from: attributes.time
          layout_type: gotime
          layout: '2006-01-02T15:04:05.999999999Z07:00'
      - type: regex_parser
        id: parser-containerd
        regex: '^(?P<time>[^ ^Z]+Z) (?P<stream>stdout|stderr) (?P<logtag>[^ ]*) ?(?P<log>.*)$'
        output: extract_metadata_from_filepath
        timestamp:
          parse_from: attributes.time
          layout: '%Y-%m-%dT%H:%M:%S.%LZ'
      - type: json_parser
        id: parser-docker
        output: extract_metadata_from_filepath
        timestamp:
          parse_from: attributes.time
          layout: '%Y-%m-%dT%H:%M:%S.%LZ'
      - type: move
        from: attributes.log
        to: body
      - type: regex_parser
        id: extract_metadata_from_filepath
        regex: '^.*\/(?P<namespace>[^_]+)_(?P<pod_name>[^_]+)_(?P<uid>[a-f0-9\-]{36})\/(?P<container_name>[^\._]+)\/(?P<restart_count>\d+)\.log$'
        parse_from: attributes["log.file.path"]
        cache:
          size: 128
      - type: move
        from: attributes.stream
        to: attributes["log.iostream"]
      - type: move
        from: attributes.container_name
        to: resource["k8s.container.name"]
      - type: move
        from: attributes.namespace
        to: resource["k8s.namespace.name"]
      - type: move
        from: attributes.pod_name
        to: resource["k8s.pod.name"]
      - type: move
        from: attributes.restart_count
        to: resource["k8s.container.restart_count"]
      - type: move
        from: attributes.uid
        to: resource["k8s.pod.uid"]

This configuration ensures that the Filelog Receiver collects logs from the specified directories and processes them appropriately ^[1][2].

To continue talking to Dosu, mention @dosu.

[vc43]

@paraniraja Have you found a solution?