search

SumoLogic / fluentd-output-sumologic

Fluentd output plugin to deliver logs or metrics to Sumo Logic.
https://rubygems.org/gems/fluent-plugin-sumologic_output
Apache License 2.0
29 stars 28 forks source link
fluentd-output-plugin sumologic sumologic-collector

readme

Build Status Gem Version contributions welcome

fluent-plugin-sumologic_output, a plugin for Fluentd

This plugin has been designed to output logs or metrics to SumoLogic via a HTTP collector endpoint

License

Released under Apache 2.0 License.

Installation

gem install fluent-plugin-sumologic_output

Configuration

Configuration options for fluent.conf are:

NOTE: * Placeholders are supported

Example Configuration

Reading from the JSON formatted log files with in_tail and wildcard filenames:

<source>
  @type tail
  format json
  time_key time
  path /path/to/*.log
  pos_file /path/to/pos/ggcp-app.log.pos
  time_format %Y-%m-%dT%H:%M:%S.%NZ
  tag appa.*
  read_from_head false
</source>

<match appa.**>
 @type sumologic
 endpoint https://collectors.sumologic.com/receiver/v1/http/XXXXXXXXXX
 log_format json
 source_category prod/someapp/logs
 source_name AppA
 open_timeout 10
</match>

Sending metrics to Sumo Logic using in_http:

<source>
  @type http
  port 8888
  bind 0.0.0.0
</source>

<match test.carbon2>
    @type sumologic
    endpoint https://endpoint3.collection.us2.sumologic.com/receiver/v1/http/ZaVnC4dhaV1hYfCAiqSH-PDY6gUOIgZvO60U_-y8SPQfK0Ks-ht7owrbk1AkX_ACp0uUxuLZOCw5QjBg1ndVPZ5TOJCFgNGRtFDoTDuQ2hzs3sn6FlfBSw==
    data_type metrics
    metric_data_format carbon2
    flush_interval 1s
</match>

<match test.graphite>
    @type sumologic
    endpoint https://endpoint3.collection.us2.sumologic.com/receiver/v1/http/ZaVnC4dhaV1hYfCAiqSH-PDY6gUOIgZvO60U_-y8SPQfK0Ks-ht7owrbk1AkX_ACp0uUxuLZOCw5QjBg1ndVPZ5TOJCFgNGRtFDoTDuQ2hzs3sn6FlfBSw==
    data_type metrics
    metric_data_format graphite
    flush_interval 1s
</match>

Example input/output

Assuming following inputs are coming from a log file named /var/log/appa_webserver.log

{"asctime": "2016-12-10 03:56:35+0000", "levelname": "INFO", "name": "appa", "funcName": "do_something", "lineno": 29, "message": "processing something", "source_ip": "123.123.123.123"}

Then output becomes as below within SumoLogic

{
    "timestamp":1481343785000,
    "asctime":"2016-12-10 03:56:35+0000",
    "levelname":"INFO",
    "name":"appa",
    "funcName":"do_something",
    "lineno":29,
    "message":"processing something",
    "source_ip":"123.123.123.123"
}

Dynamic Configuration within log message

The plugin supports overriding SumoLogic metadata and log_format parameters within each log message by attaching the field _sumo_metadata to the log message.

NOTE: The _sumo_metadata field will be stripped before posting to SumoLogic.

Example

{
  "name": "appa",
  "source_ip": "123.123.123.123",
  "funcName": "do_something",
  "lineno": 29,
  "asctime": "2016-12-10 03:56:35+0000",
  "message": "processing something",
  "_sumo_metadata": {
    "category": "new_sourceCategory",
    "source": "override_sourceName",
    "host": "new_sourceHost",
    "log_format": "merge_json_log"
  },
  "levelname": "INFO"
}

Retry Mechanism

retry_min_interval, retry_max_interval, retry_timeout, retry_max_times are not the buffer retries parameters. Due to technical reason, this plugin implements it's own retrying back-off exponential mechanism. It is disabled by default, but we recommend to enable it by setting use_internal_retry to true.

TLS 1.2 Requirement

Sumo Logic only accepts connections from clients using TLS version 1.2 or greater. To utilize the content of this repo, ensure that it's running in an execution environment that is configured to use TLS 1.2 or greater.