SumoLogic / fluentd-kubernetes-sumologic

FluentD plugin to extract logs from Kubernetes clusters, enrich and ship to Sumo logic.
Apache License 2.0
61 stars 65 forks source link

LOG_FORMAT is ignored? #3

Closed mfornasa closed 7 years ago

mfornasa commented 7 years ago

Maybe I'm doing this wrong... I tried to apply LOG_FORMAT=text in the daemonset sample definintion but the logs are still sent in JSON.

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: fluentd
  labels:
    app: fluentd
    version: v1
spec:
  template:
    metadata:
      labels:
        name: fluentd
    spec:
      volumes:
      - name: pos-files
        emptyDir: {}
      - name: host-logs
        hostPath:
          path: /var/log/
      - name: docker-logs
        hostPath:
          path: /var/lib/docker
      containers:
      - image: sumologic/fluentd-kubernetes-sumologic:latest
        name: fluentd
        imagePullPolicy: Always
        volumeMounts:
        - name: host-logs
          mountPath: /mnt/log/
          readOnly: true
        - name: docker-logs
          mountPath: /var/lib/docker/
          readOnly: true
        - name: pos-files
          mountPath: /mnt/pos/
        env:
        - name: LOG_FORMAT
          value: text
        - name: COLLECTOR_URL
          valueFrom:
            secretKeyRef:
              name: sumologic
              key: collector-url
stevezau commented 7 years ago

That should work. I'll test it myself and get back to you.

mfornasa commented 7 years ago

More info. I tried to put the value "invalid" in the LOG_FORMAT var, and the container refuses to start:

2017-01-26 12:15:17 +0000 [error]: config error file="/fluentd/etc/fluent.conf" error="Invalid log_format invalid must be text, json or json_merge"

When reverting back to text, I get the JSON log back in Sumologic. I am using the hosted service.

mfornasa commented 7 years ago

In the output plugin, here the log_format contained in sumo_metadata owerwrites the log_format provided to the output plugin directly, which comes from here. So, the env var does not work.

stevezau commented 7 years ago

ah, i see the issue.

I believe by adding the below to source.containers.conf it should set the _sumo_metadata field correctly for the output plugin.

log_format "#{ENV['LOG_FORMAT']}"

Can you please let me know if you still have issues?

mfornasa commented 7 years ago

Works now.