fluent-plugins-nursery / fluent-plugin-remote_syslog

Fluentd plugin for output to remote syslog serivce (e.g. Papertrail)
https://github.com/dlackty/fluent-plugin-remote_syslog
MIT License
69 stars 54 forks source link

problem with ${tag_parts[]} ? #17

Closed widgetpl closed 7 years ago

widgetpl commented 8 years ago

My docker-compose manifest looks like

version: '2'
services:

  fluentd:
    build: fluentd/
    container_name: fluentd
    hostname: "env-192.168.0.101"
    ports:
      - "24224:24224"
      - "24284:24284"
    volumes:
      - ./fluentd_logs:/fluentd/log:rw
      - ./fluentd/rsyslog.conf:/fluentd/etc/fluent.conf
    links:
      - rsyslog:rsyslog

  logger:
    image: widgetpl/logger:v0.1
    hostname: "logger"
    container_name: logger
    logging:
      driver: fluentd
      options:
        fluentd-address: "localhost:24224"
        tag: "docker.{{(.Hostname)}}"
    environment:
      SLEEP: 1
      ENV1: value1
      ENV2: value2
    links:
      - fluentd:fluentd

  rsyslog:
    #image: widgetpl/rsyslog:v0.1
    build: ../STACK-282/rsyslog/
    container_name: rsyslog
    volumes:
      - ./rsyslog/template.conf:/etc/rsyslog.d/docker.conf
      - ./rsyslog/logs:/var/log/logs:rw
    ports:
      - "1515:515"
      - "1514:514"

and I have tried two differnt fluentd configs. First

  <match docker.**>
    @type remote_syslog
    @log_level debug
    host rsyslog
    port 515
    severity debug
    facility local7
    tag ${tag}
    hostname ${hostname}
  </match>

and second

  <match docker.**>
    @type remote_syslog
    @log_level debug
    host rsyslog
    port 515
    severity debug
    facility local7
    tag ${tag_parts[0]}
    hostname ${tag_parts[1]}
  </match>

and I have this setup for rsyslog

template (name="DynFile" type="string" string="/var/log/logs/%HOSTNAME%/%PROGRAMNAME%.log")

local7.* ?DynFile

When I use First config of fluentd i get

hostname: env-192.168.0.101
source: env-192.168.0.101
fromhost: fluentd.stack261_default
fromhost-ip: 172.18.0.3
syslogtag: docker.michal-Latitude-E6540:
programname: docker.michal-Latitude-E6540

and the logs are in in /var/log/logs/env-192.168.0.101/docker.michal-Latitude-E6540.log

and whit second config

hostname: fluentd.stack261_default
source: fluentd.stack261_default
fromhost: fluentd.stack261_default
fromhost-ip: 172.18.0.3
syslogtag: ${tag_parts[1]}
programname: ${tag_parts

and the logs are in /var/log/logs/fluentd.stack261_default/${tag_parts.log

widgetpl commented 7 years ago

It will not work as hostname does not support ${tag_parts}.

rootvgusr commented 7 years ago

I am having this same issue with the td-agent enterprise edition. My config file works just fine with the open source version, not the commercial version. I am trying to use ${tag_parts[x]} to set sourcetype and index in the splunk hec output plugin.

<match {us,last_chance_predix}> @type splunk_hec host splunkserver.com port 8088 token D76AA0AC-EE4886867-YTYTY-974646 default_host "#{Socket.gethostname}" default_source fluentd default_index ${tag_parts[0]} sourcetype ${tag_parts[1]}

default_index = last_chance_predix

sourcetype new:test