search

pcfens / puppet-filebeat

Apache License 2.0
61 stars 179 forks source link
filebeat hacktoberfest puppet

readme

puppet-filebeat

Build Status

Table of Contents

Description

The filebeat module installs and configures the filebeat log shipper maintained by elastic.

Setup

What filebeat affects

By default filebeat adds a software repository to your system, and installs filebeat along with required configurations.

Upgrading to Filebeat 7.x

To upgrade to Filebeat 7.x, simply set $filebeat::major_version to 7 and $filebeat::package_ensure to latest (or whichever version of 7.x you want, just not present).

You'll also need to change instances of filebeat::prospector to filebeat::input when upgrading to version 4.x of this module.

Setup Requirements

The filebeat module depends on puppetlabs/stdlib, and on puppetlabs/apt on Debian based systems.

Beginning with filebeat

filebeat can be installed with puppet module install pcfens-filebeat (or with r10k, librarian-puppet, etc.)

The only required parameter, other than which files to ship, is the outputs parameter.

Usage

All of the default values in filebeat follow the upstream defaults (at the time of writing).

To ship files to elasticsearch:

class { 'filebeat':
  outputs => {
    'elasticsearch' => {
     'hosts' => [
       'http://localhost:9200',
       'http://anotherserver:9200'
     ],
     'loadbalance' => true,
     'cas'         => [
        '/etc/pki/root/ca.pem',
     ],
    },
  },
}

To ship log files through logstash:

class { 'filebeat':
  outputs => {
    'logstash'     => {
     'hosts' => [
       'localhost:5044',
       'anotherserver:5044'
     ],
     'loadbalance' => true,
    },
  },
}

Shipper and logging options can be configured the same way, and are documented on the elastic website.

Adding an Input

Inputs are processes that ship log files to elasticsearch or logstash. They can be defined as a hash added to the class declaration (also used for automatically creating input using hiera), or as their own defined resources.

At a minimum, the paths parameter must be set to an array of files or blobs that should be shipped. doc_type is what logstash views as the type parameter if you'd like to apply conditional filters.

filebeat::input { 'syslogs':
  paths    => [
    '/var/log/auth.log',
    '/var/log/syslog',
  ],
  doc_type => 'syslog-beat',
}

Multiline Logs

Filebeat inputs can handle multiline log entries. The multiline parameter accepts a hash containing pattern, negate, match, max_lines, and timeout as documented in the filebeat configuration documentation.

JSON Logs

Filebeat inputs (versions >= 5.0) can natively decode JSON objects if they are stored one per line. The json parameter accepts a hash containing message_key, keys_under_root, overwrite_keys, and add_error_key.

Depending on the version, expand_keys, document_id and ignore_decoding_error may be supported as well.

See the filebeat configuration documentation for details.

Inputs in Hiera

Inputs can be defined in hiera using the inputs parameter. By default, hiera will not merge input declarations down the hiera hierarchy. That behavior can be changed by configuring the lookup_options flag.

inputs can be a Hash that will follow all the parameters listed on this documentation or an Array that will output as is to the input config file.

Usage of filebeat modules

Filebeat ships with modules which contain pipelines and dashboards for common software. Filebeat needs to be setup to ship directly into elasticsearch that it's possible that filebeat will setup pipelines and dashboards automatically.

If your setup includes logstash or some other service between filebeat and elasticsearch the following settings might not work as expected.

The following should be a minimal example to get filebeat::module::* to create the required config and push pipeline and dashboards into your elasticsearch & kibana.

class { 'filebeat::module::system':
  syslog_enabled => true,
  auth_enabled => true,
}

class { 'filebeat':
  enable_conf_modules => true,
  overwrite_pipelines => true,
  setup => {
    dashboards => {
      enabled => true
    },
    kibana => {
      host => 'http://kibana.example.com:5601',
    }
  }
}

Usage on Windows

When installing on Windows, this module will download the windows version of Filebeat from elastic to C:\Temp by default. The directory can be overridden using the tmp_dir parameter. tmp_dir is not managed by this module, but is expected to exist as a directory that puppet can write to.

Processors

Filebeat 5.0 and greater includes a new libbeat feature for filtering and/or enhancing all exported data through processors before being sent to the configured output(s). They can be defined as a hash added to the class declaration (also used for automatically creating processors using hiera), or as their own defined resources.

To drop the offset and input_type fields from all events:

class {'filebeat':
  processors => [
    {
      'drop_fields' => {
        'fields' => ['input_type', 'offset'],
      }
    }
  ],
}

To drop all events that have the http response code equal to 200: input

class {'filebeat':
  processors => [
    {
      'drop_event' => {
        'when' => {'equals' => {'http.code' => 200}}
      }
    }
  ],
}

Now to combine these examples into a single definition:

class {'filebeat':
  processors => [
    {
      'drop_fields' => {
        'params'   => {'fields' => ['input_type', 'offset']},
        'priority' => 1,
      }
    },
    {
      'drop_event' => {
        'when'     => {'equals' => {'http.code' => 200}},
        'priority' => 2,
      }
    }
  ],
}

For more information please review the documentation here.

Processors in Hiera

Processors can be declared in hiera using the processors parameter. By default, hiera will not merge processor declarations down the hiera hierarchy. That behavior can be changed by configuring the lookup_options flag.

Index Lifecycle Management

You can override the default filebeat ILM policy by specifying ilm.policy hash in filebeat::setup parameter:

filebeat::setup:
  ilm.policy:
    phases:
      hot:
        min_age: "0ms"
        actions:
          rollover:
            max_size: "10gb"
            max_age: "1d"

Reference

Public Classes

Class: filebeat

Installs and configures filebeat.

Parameters within filebeat

Private Classes

Class: filebeat::config

Creates the configuration files required for filebeat (but not the inputs)

Class: filebeat::install

Calls the correct installer class based on the kernel fact.

Class: filebeat::params

Sets default parameters for filebeat based on the OS and other facts.

Class: filebeat::repo

Installs the yum or apt repository for the system package manager to install filebeat.

Class: filebeat::service

Configures and manages the filebeat service.

Class: filebeat::install::linux

Install the filebeat package on Linux kernels.

Class: filebeat::install::windows

Downloads, extracts, and installs the filebeat zip file in Windows.

Public Defines

Define: filebeat::input

Installs a configuration file for a input.

Be sure to read the filebeat configuration details to fully understand what these parameters do.

Parameters for filebeat::input

Define: filebeat::module

Base resource used to implement filebeat module support in this puppet module and can be useful if you have custom filebeat modules.

Parameters for filebeat::module

Limitations

This module doesn't load the elasticsearch index template into elasticsearch (required when shipping directly to elasticsearch).

When installing on Windows, there's an expectation that C:\Temp already exists, or an alternative location specified in the tmp_dir parameter exists and is writable by puppet. The temp directory is used to store the downloaded installer only.

Generic template

By default, a generic, open ended template is used that simply converts your configuration into a hash that is produced as YAML on the system. To use a template that is more strict, but possibly incomplete, set conf_template to filebeat/filebeat.yml.erb.

Debian Systems

Filebeat 5.x and newer requires apt-transport-https, but this module won't install it for you.

Using config_file

There are a few very specific use cases where you don't want this module to directly manage the filebeat configuration file, but you still want the configuration file on the system at a different location. Setting config_file will write the filebeat configuration file to an alternate location, but it will not update the init script. If you don't also manage the correct file (/etc/filebeat/filebeat.yml on Linux, C:/Program Files/Filebeat/filebeat.yml on Windows) then filebeat won't be able to start.

If you're copying the alternate config file location into the real location you'll need to include some metaparameters like

file { '/etc/filebeat/filebeat.yml':
  ensure  => file,
  source  => 'file:///etc/filebeat/filebeat.special',
  require => File['filebeat.yml'],
  notify  => Service['filebeat'],
}

to ensure that services are managed like you might expect.

Logging on systems with Systemd and with version filebeat 7.0+ installed

With filebeat version 7+ running on systems with systemd, the filebeat systemd service file contains a default that will ignore the logging hash parameter

Environment="BEAT_LOG_OPTS=-e`

to overide this default, you will need to set the systemd_beat_log_opts_override parameter to empty string

example:

class {'filebeat':
  logging => {
    'level'     => 'debug',
    'to_syslog' => false,
    'to_files'  => true,
    'files'     => {
      'path'        => '/var/log/filebeat',
      'name'        => 'filebeat',
      'keepfiles'   => '7',
      'permissions' => '0644'
    },
  systemd_beat_log_opts_override => "",
}

this will only work on systems with puppet version 6.1+. On systems with puppet version < 6.1 you will need to systemctl daemon-reload. This can be achived by using the camptocamp-systemd

include systemd::systemctl::daemon_reload

class {'filebeat':
  logging => {
...
    },
  systemd_beat_log_opts_override => "",
  notify  => Class['systemd::systemctl::daemon_reload'],
}

Development

Pull requests and bug reports are welcome. If you're sending a pull request, please consider writing tests if applicable.