search

example42 / puppet-tp

Tiny Puppet - The Universal Installer
http://tiny-puppet.com
Apache License 2.0
68 stars 21 forks source link
automation configuration-management cross-platform

readme

Tiny Puppet

Coverage Status Codacy Badge

Table of Contents

Module description

Example42's tp (short for Tiny Puppet) module can manage every application (relevant tinydata is needed) on every Operating System (Linux flavours, Solaris, macOS, Windows) using different methods (native packages, packages from upstream repos, release tarballs, git source, docker container).

It permits allows this as code, via Puppet user defined types:

via the command line tool tp, that allows to:

via Bolt tasks, that permits to orchestrate the above operations on remote nodes.

Features

The main features of tp module are:

Use cases

Tiny Puppet can be considered a supplement or a replacement of standard component modules.

It's particularly useful when there are to manage packages, services and configuration files, for more complex and application-specific resources a dedicated module is probably preferable.

It can be used in the following cases:

The intended users can be any of the following:

To see real-world usage of tp defines give a look to:

Setup

TP can be installed as any other module:

Once tp module is added to the modulepath the (optional) tp command can be installed to a node in the following ways:

What tp affects

Getting started with tp

Starting from version 3.8.0, a technology preview of tp 4 features is available by specifying the use_v4 parameter:

tp::use_v4: true

Here follows an example of tp resources used inside a custom profile where the content of a configuration file is based on a template with custom values.

class profile::openssh (
  String $server_template = 'profile/openssh/sshd_config.erb',
  String $client_template = 'profile/openssh/ssh_config.erb',
  Hash $options    = {},
) {

  # OpenSSH installation
  tp::install { 'openssh': }

  # Configuration of sshd_config server configuration file (main config file)
  tp::conf { 'openssh':
    template     => $server_template,
    options_hash => $options,
  }

  # Configuration of ssh_config client configuration file
  tp::conf { 'openssh::ssh_config':
    template     => $client_template,
    options_hash => $options,
  }
}

The above class, once included, will:

In the defined templates key-values set in the $options hash can be accessed via <%= @options['key_name'] %> (example for an erb template), so, with hieradata as follows:

profile::openssh::options:
  StrictHostKeyChecking: yes

we can have in the used templates lines as follows:

StrictHostKeyChecking: <%= @options['StrictHostKeyChecking'] %>

Usage in Puppet code

The user defined types (or defines, or user defines) provided by tp module can be used in Puppet manifests to manage installation and configuration of applications. They can be straightforward and essential ( tp:: { 'Application name': }) but provide several parameters which can be used to customise and fine-tune the managed resources as needed.

Common uses

Install an application with default settings (package installed, service started)

tp::install { 'redis': }

Configure the application main configuration file a custom erb template which uses data from a custom $options_hash:

tp::conf { 'rsyslog':
  template     => 'site/rsyslog/rsyslog.conf.erb',
  options_hash => lookup('rsyslog::options_hash'),
}

Populate any custom directory from a Git repository (it requires Puppet Labs' vcsrepo module):

tp::dir { '/opt/apps/my_app': 
  source      => 'https://git.example.42/apps/my_app/',
  vcsrepo     => 'git',
}

Installing packages - tp::install

Some parameters are available to manage tp::install automation:

Other parameters are available to manage integrations:

Some specific params are to handle repos:

These parameters allow to skip management of packages or services:

Some parameters allow to configure tp::conf and tp::dir resources directly from tp::install (inheriting the same settings and options):

Parameters are also available to customise the tiny data settings which affect package and service names, repos settings, file paths and so on. The params are common to all the tp defines, check Updating tiny data and using alternative data sources section for details.

An example with a custom test for the rabbitmq service:

tp::install { 'rabbitmq':
  cli_enable    => true,
  test_enable   => true,
  test_template => 'profile/rabbitmb/tp_test.erb',
  options_hash  => { 'server' => "rabbitmq.${::domain}" }
}

It's possible to specify the version of the package to install (the provided version must be available in the configured repos):

tp::install { 'postfix':
  ensure => '2.10.1-9',
}

To uninstall an application, there are two alternatives:

tp::uninstall { 'redis': }
tp::install { 'redis': ensure => absent }

Installation alternatives - tp::stdmod

To handle packages installations and configuration files, there's also the tp::stdmod define to manage an application using stdmod compliant parameters.

Note that tp::stdmod is alternative to tp::install (both of them manage packages and services) and may be complementary to tp::conf (you can configure files with both).

tp::stdmod { 'redis':
  config_file_template => 'site/redis/redis.conf',
}

If you wonder what's better, use tp::install + tp::conf rather than tp::stdmod.

Managing configurations - tp::conf

The tp::conf define is a wrapper over a file resource which makes it easy and fast to manage configuration files for an application, handling the correct permissions, paths and owners for the underlying OS. Be aware anyway that the contents of the managed files are entirely up to you: Tiny Puppet does not have any awareness of the configuration options available for the managed applications.

If as the title is passed just the name of the application, Tiny Puppet tries to configure its main configuration file. There are various ways to manage other configuration files related to the application as detailed in the section tp::conf file paths conventions.

It's possible to manage files with different methods, for example directly providing its content:

tp::conf { 'redis':
  content => 'my content is king',
}

or providing a custom template with custom options:

tp::conf { 'openssh::ssh_config':
  template     => 'profile/openssh/ssh_config.erb',
  options_hash => {
    UsePAM        => 'yes',
    X11Forwarding => 'no',
  } 
}

Via the template parameter with can both specify files .erb suffix (used as content => template($template)) or with .epp suffix (used as content => epp($template)). If not .erb or .epp suffix is present in the template value, then it's treates as and erb (content => template($template)).

In the profile/templates/openssh/ssh_config.erb template you will have the contents you want and use the above options with something like (note you can use both the @options and the @options_hash variable):

[...]
UsePAM <%= @options['UsePAM'] %>
X11Forwarding <%= @options['X11Forwarding'] %>
[...]

Alternatively a custom epp template with Puppet code instead of Ruby (used as content => epp($epp)) directly with the epp parameter:

tp::conf { 'redis:
  epp   => 'site/redis/redis.conf.epp',
}

also it's possible to provide the source to use, instead of managing it with the content argument:

tp::conf { 'redis':
  source      => [ "puppet:///modules/site/redis/redis.conf-${hostname}" ,
                   'puppet:///modules/site/redis/redis.conf' ] ,
}

For applications for which it exists the setting 'config_file_format' you can just pass the hash of options_hash of settings to configure and tp::conf creates a valid configuration file for the application:

tp::conf { 'filebeat':
  options_hash => {
    filebeat.modules => ['module: system']
    syslog => {
      enabled   => true,
      var.paths => ["/var/log/syslog*","/var/log/messages"],
    }
  }
}

This example makes much more sense if based on Hiera data (see Configuring tp resources via Hiera section for details):

tp::conf_hash:
  filebeat:
    options_hash:
      filebeat.modules:
      - module: system
      syslog:
        enabled: true
        var.paths:
          - "/var/log/syslog*"
          - "/var/log/messages"

By default, configuration files managed by tp::conf automatically notify the service(s) and require the package(s) installed via tp::install. If you use tp::conf without a relevant tp::install define and have dependency cycle problems or references to non existing resources, you can disable these automatic relationships:

tp::conf { 'bind':
  config_file_notify  => false,
  config_file_require => false,
}

You can also set custom resource references to point to actual resources you declare in your manifests:

tp::conf { 'bind':
  config_file_notify  => 'Service[bind9]',
  config_file_require => 'Package[bind9-server]',
}

Tiny Puppet even validates the syntax of the managed configuration files before applying them, if the setting 'validate_cmd' is present in Tiny Data. To disable this validation, which prevents Puppet from changing a file if its syntax is wrong, set validate_syntax parameter to false.

tp::conf file paths conventions

Tp:conf has some conventions on the actual configuration file managed.

By default, if you just specify the application name, the file managed is the "main" configuration file of that application (in case this is not evident or might be questionable, check the config_file_path value in the tinydata files for the used application).

# This manages /etc/ssh/sshd_config
tp::conf { 'openssh':
  [...]
}

If you specify a file name after the application name in the title, separated by ::, and you don't specify any alternative base_file, then that file is placed in the "base" configuration dir (config_dir_path in tinydata):

# This manages /etc/ssh/ssh_config
tp::conf { 'openssh::ssh_config':
  [...]
}

If you specify the parameter base_file then the path is the one of the specified base_file and the title does not provide any information about the managed file path (it still needs the relevant app in the first part, before ::, and it needs to be unique across the catalog). For example, if base_file => 'init' the path used is the value of the init_file_path key in the relevant tinydata.

# This manages /etc/default/puppetserver on Debian or /etc/sysconfig/puppetserver on RedHat
tp::conf { 'puppetserver::init':
  base_file => 'init',
  [...]
}

If you explicitly set a path, that path is used and the title is ignored (be sure, anyway, to refer to a supported application and is not duplicated in your catalog):

# This manages /usr/local/bin/openssh_check
tp::conf { 'openssh::ssh_check':
  path => '/usr/local/bin/openssh_check',
  [...]
}

Managing directories - tp::dir

Manage a whole configuration directory:

tp::dir { 'redis':
  source      => 'puppet:///modules/site/redis/',
}

Manage a specific directory type. Currently defined directories types are:

Note that some of these directory types might not be defined for every application.

tp::dir { 'apache':
  base_dir => 'data',
  source   => 'puppet:///modules/site/apache/default_site',
}

Clone a whole configuration directory from a Git repository (it requires Puppet Labs' vcsrepo module):

tp::dir { 'redis':
  source      => 'https://git.example.42/puppet/redis/conf/',
  vcsrepo     => 'git',
}

Populate any custom directory from a Subversion repository (it requires Puppet Labs' vcsrepo module):

tp::dir { 'my_app': # The title is irrilevant, when path argument is used 
  path        => '/opt/apps/my_app',
  source      => 'https://svn.example.42/apps/my_app/',
  vcsrepo     => 'svn',
}

Provide a data directory (the default DocumentRoot, for apache) from a Git repository (it requires Puppet Labs' vcsrepo module) :

tp::dir { 'apache':
  # base_dir is a tag that defines the type of directory for the specified application.
  # Default: config. Other possible dir types: 'data', 'log', 'confd', 'lib'
  # or any other name defined in the application data with a format like: ${base_dir}_dir_path
  base_dir    => 'data' 
  source      => 'https://git.example.42/apps/my_app/',
  vcsrepo     => 'git',
}

Continuous deployment the tiny way: automatically deploy the latest version of an app from a git repo whenever Puppet runs:

tp::dir { 'my_app':
  ensure  => latest,
  path    => '/opt/apps/my_app',
  source  => 'https://git.example.42/apps/my_app/',
  vcsrepo => 'git',
}

Managing repositories - tp::repo

Currently, Tiny Puppet supports applications' installation only via the OS native packaging system or Chocolatey on Windows and HomeBrew on MacOS. In order to cope with software which may not be provided by default on an OS, TP provides the tp::repo define that manages YUM and APT repositories for RedHat and Debian based Linux distributions.

The data about a repository is managed like all the other data of Tiny Puppet. Find here an example for managing Apt repositories and here one for Yum ones.

Generally, you don't have to use the tp::repo define directly, as, when the repository data is present, it's automatically added from the tp::install one.

When it's present the relevant data for an application, it's possible to install it using different alternative repos. For example you can use this synatx to install the repo for the Elastic stack version 6.x:

tp::install { 'elastic_repo':
  repo => '6.x',
}

In some cases, where for the given application name there are no packages, the following commands have exactly the same effect:

tp::install { 'epel': }  # Installs Epel repository on RedHat derivatives. Does nothing on other OS.
tp::repo { 'epel': }     # Same effect of tp::install since no application package (other than the release one) is actually installed

If, for whatever reason, you don't want to automatically manage a repository for an application, you can set to false the auto_repo parameter, and, eventually you can manage the repository in a custom dependency class:

tp::install { 'elasticsearch':
  auto_repo => false,
}

Tinydata has information about various useful repos, both general or application/vendor specific. They are a tp::install away, all the following ones add repos for RedHat systems and derivatives:

There is also Tiny Data for some vendors repos, and sometimes they are directly in the relevant application data, sometimes in a dedicated namespace:

Automated and easy testing - tp::test

The tp module provides the following functionality for writing easy and quick checks:

To automatically add testing for an application installed via tp use the follwoing parameters:

tp::install { 'redis':
  test_enable => true,  # Default: false
  cli_enable  => true,  # Default: false. This also installs the tp command on the system
}

If you want these functionality enabled by default, in your control-repo's main manifests/site.pp you can add the following resource defaults:

Tp::Install {
  test_enable => true,
  cli_enable  => true,
}

To add a custom test for an application you can either specify the template to use for the test script of that application, in tp::install:

tp::install { 'rabbitmq':
  test_enable   => true,
  cli_enable    => true,
  test_template => 'profile/rabbimq/tptest.erb',
}

Or you can use a specific tp::test define:

tp::test { 'rabbitmq':
  template => 'profile/rabbimq/tptest.erb',
  options_hash => {
    port => '11111',
    host => 'localhost',
  },
}

All the keys set via the $options_hash parameter can be used in the erb template with sopmething like:

port_to_check=<%= @options_hash['port'] >

The tp::test define has the following parameters to manage the content of the test script (placed under /etc/tp/test/$title):

Starting from tp version 3.1.0 it's also possible to add any custom test script also unrelated to a specific application. You can you these to check general system status of some web application status or whatever it may make sense to test. You can legerage on the tp test command or the tp::test define to automate infrastructure testing on your CI/CD pipelines.

Examples:

tp::test { 'system_health':
  source => 'puppet:///modules/profile/base/system_health',
}

tp::test { 'my_web_app':
  source => 'puppet:///modules/profile/my_web_app/test',
}

The scripts can be in any language, they just need to have an exit code 0 in case of success, and a different exit code in case of failure.

The tp test command and the tp::test task will exit with 0 if all the test have been successfull, and exit with 1 if any of the test have returned an error.

Configuring tp resources via Hiera

The main and unique class of this module, tp, installs the tp CLI command (set tp::cli_enable to false to avoid that) and offers parameters which allow to configure via Hiera what tp resources to manage.

For each of these parameters (example: install_hash) it's possible to set on hiera:

Where * is any of install, conf, dir, puppi, stdmod, concat and repo.

An example to install kubernetes and sysdig, adding the management of the required dependencies:

tp::install_hash:
  kubernetes:
    auto_prereq: true
  sysdig:
    auto_prereq: true

This is an example of tp::dir hash (with the ensure latest for a git repo for "Tiny Continuous Deployment"):

tp::dir_hash:
  apache::openskills.info:
    vcsrepo: git
    source: git@git.alvagante.com:web/openskills.info.git
    path: /var/www/html/openskills.info
  apache::abnormalia.com:
    ensure: latest
    vcsrepo: git
    source: git@git.alvagante.org:web/abnormalia.com.git
    path: /var/www/html/abnormalia.com

There are also analogue parameters to handle resources Hashes based on the clients' OS Family for tp::install (tp::osfamily_install_hash, tp::osfamily_install_hash_merge_behaviour, tp::osfamily_install_defaults) and tp::conf (tp::osfamily_conf_hash, tp::osfamily_conf_hash_merge_behaviour, tp::osfamily_conf_defaults)

Here is an example of OS based install_hash (note the usage of Yaml merge keys to avoid data duplication for RedHat and Debian families): 

linux_tp_install: &linux_tp_install
  filebeat:
    auto_prereq: true
  metricbeat: {}
  auditbeat: {}
  heartbeat-elastic:
    ensure: absent

tp::osfamily_install_hash:
  RedHat:
    <<: *linux_tp_install
  Debian:
    <<: *linux_tp_install
  windows:
    chocolateygui: {}
    docker-desktop: {}
    powertoys: {}
    MobaXTerm: {}
    Sysinternals: {}

Updating tiny data and using alternative data sources

By default Tiny Puppet uses the tinydata module to retrieve data for different applications, but it's possible to override its settings in two ways:

The settings_hash parameter, present in all tp defines, allows the override of specific settings coming from tiny data.

The names of the available settings are defined in the tp::settings data type. Usage can be as follows:

tp::install { 'redis':
  settings_hash => {
    'package_name'     => 'my_redis',
    'config_file_path' => '/opt/etc/redis',
  },
}

The data_module parameter allows to use a totally different module for tinydata:

tp::install { 'apache':
  data_module => 'my_data', # Default: tinydata
}

In this custom data module, we have to reproduce the structure of tinydata to make it work with tp.

If we want to use our own data module for all our applications, we might prefer to set the following resource defaults in our main manifest (manifest/site.pp, typically):

Tp::Install {
  data_module  => 'my_data',
}
Tp::Conf {
  data_module  => 'my_data',
}
Tp::Dir {
  data_module  => 'my_data',
}

Starting from version 2.3.0 (with tinydata version > 0.3.0) tp can even install applications for which there's actually no tinydata defined. In this case just the omonimous package is installed and a warning about missing tinydata is shown. In these cases other defines like tp::conf don't work.

Usage on the command line

Tiny Puppet adds the tp command to Puppet. Just have it in your modulepath and install the tp command with:

sudo puppet module install example42-tp
sudo puppet tp setup

With the tp command you can install on the local OS the given application, taking care of naming differences, additional repos or prerequisites.

tp install <application>    # Install an application
tp uninstall <application>  # Uninstall an application
tp test [application]       # Test one or all the applications
tp log [application]        # Tail the logs of one or all applications
tp info [application]       # Show info on one or all applications
tp debug [application]      # Troubleshoot one or all applications
tp version [application]    # Show version of one or all applications
tp source <application>     # Clone the source of an applications
tp version [application]    # Show version of one or all applications

tp desktop                  # Show tp desktop options
tp desktop init             # Initialise a new tp desktop repostiory
tp desktop list             # List the available desktops
tp desktop show <desktop>   # Show the details of a desktop
tp desktop prevew <desktop> # Preview what desktop apply would do
tp desktop apply <desktop>  # Apply a desktop configuration

Check this asciicast to see tp cli in action.

Reference

The tp module provides the following resources.

Classes

Defined types

Types

Functions

Tasks

Refer to in-code documentation for full reference.

Check Puppetmodule.info for the online version.

Prerequisites and limitations

Starting from version 3 Tiny Puppet requires Hiera data in module, available from Puppet 4.9.

Version 2.x of Tiny Puppet is compatible with Puppet 4.4 or later and PE 2016.1.1 or later.

Version 1.x is compatible also with Puppet 3, using the 3.x compatible defines (with the 3 suffix, like tp::install3).

Version 0.x of Tiny Puppet is compatible by default with Puppet 3 (tp::install) and have Puppet 4 / future parser version, with the 4 suffix, like tp::install4).

If tp doesn't correctly install a specific application on the OS you want, please TELL US.

It's very easy and quick to add new apps or support for a new OS in tinydata.

Currently most of the applications are supported on RedHat and Debian derivatives Linux distributions, but as long as you provide a valid installable package name, tp can install any application given in the title, even if there's no specific Tinydata for it.

Tiny Puppet requires these Puppet modules:

In order to work on some OS you need some additional modules and software:

If you use the relevant defines, other dependencies are needed:

Additional info

You can experiment and play with Tiny Puppet and see a lot of usage examples on Example42's PSICK control-repo and the psick module.

Tiny Puppet has a website.

The following blog posts, newest first, have been written on Tiny Puppet, older ones might contain not updated information: