search

tobami / littlechef

Cook with Chef without a Chef Server
Apache License 2.0
473 stars 70 forks source link

readme

LittleChef

With LittleChef you will be able to get started more quickly cooking with Chef, the excellent Configuration Management System.

Overview

You may think of this like a pocket Chef that doesn't need a Chef Server. Just your local kitchen with all your cookbooks, roles data bags and nodes, which will get rsynced to a node each time you start a Chef Solo configuration run with the bundled fix command.

It also adds features to Chef Solo that are currently only available for Chef Server users: data bag search, and node search.

How it all works

It all starts in the kitchen, which you should keep under version control:

Whenever you start a Chef Solo configuration run with the local fix command, all cookbooks, roles and databags are rsynced to the /tmp/chef-solo/ directory, together with the /etc/chef/node.json and /etc/chef/solo.rb files, and chef-solo is executed at the remote node.

The result is that you can configure your nodes exactly when and how you want, all without needing a Chef Server. And all your infrastructure, including your nodes, will be in code, revision controlled.

Environments

Chef 10 does not support Environments in Solo mode, but LittleChef will automatically add a cookbook library that will let you define chef_environment in a role or node.

If you use Chef 11, all environment features except cookbook versioning work out of the box.

Data bag Search

Chef Solo does not support data bag search. To be able to use search, the chef-solo-search library needs to be present in your cookbooks directory.

Most examples in the [chef search documentation][] are possible, including stuff like: search(:users, "married:true AND age:35").

Node Search

Node search is achieved by creating a "node" data bag on the fly for every run, with the data from each node defined in nodes/, but with the attribute values being the result from merging cookbook, node and role attributes, following the standard Chef attribute preference rules. Some automatic attributes are also added.

munin_servers = search(:node, "role:#{node['munin']['server_role']} AND chef_environment:#{node.chef_environment}")

Logs

Chef Solo output for a configuration run will be found at the node's /var/log/chef/solo.log, and the previous configuration run will be moved to solo.log.1.

metadata.rb and ruby roles

LittleChef depends on the JSON versions of the cookbook metadata and roles to properly merge attributes. You can still use the ruby versions, and generate the JSON versions when you make changes. If you have knife locally installed, it will even be done automatically on every run if a changed metadata.rb is detected. Ruby roles are not yet automatically converted, but an implementation is planned.

Plugins

You can define your own LittleChef tasks as Python plugin modules. They should be located in the plugins directory. The filename will be the plugin name and the module docstring the description. Each plugin should define an execute function, which will then be executed when applying a plugin on a node (the Cooking section describes how to run a plugins).

You can find example plugins in the repository plugins directory

Getting Big

Beyond a small number of nodes it becomes more and more difficult to keep track of everything. To solve that problem we created Kitchen, an incredibly useful dashboard that provies an overview of all nodes and a graphical visualization of role dependencies.

Installation

Desktop support

Tested on all major operating systems: Linux, Mac OS X, Windows and FreeBSD

Requirements

The best way to install LittleChef is using pip. Required packages are installed by typing:

$ sudo apt-get install python-pip python-dev

for Debian and Ubuntu, or

$ yum install python-pip python-devel

for RHEL and CentOS.

Installation

You can install LittleChef directly from the PyPI:

$ pip install littlechef

Usage

Disclaimer

Careful what you do with your nodes!:

A certain famous Chef: What do I always say? Anyone can cook. LittleChef: Yeah. Anyone can, that doesn't mean that anyone should.

Local Setup

To create a new kitchen, type:

$ fix new_kitchen

This will create a few files and directories inside the current directory which LittleChef needs to be able to cook: littlechef.cfg, environments, roles/, data_bags/, nodes/, cookbooks/ and site-cookbooks/. You can create and have as many kitchens as you like on your computer.

Authentication

To be able to issue commands to remote nodes, you need to enter a user and a password with sudo rights. new_kitchen will have created a file named littlechef.cfg. You can edit it now to enter needed authentication data. There are several possibilities:

The last one allows the most flexibility, as it allows you to define different usernames, passwords and/or keypair-files per hostname. LittleChef will look at ~/.ssh/config by default, but you can always specify another path in littlechef.cfg:

[userinfo]
user = myusername
password = mypassword
ssh-config = /path/to/config/file

An example ~/.ssh/config file:

Host www.cooldomain.com
    HostName www.cooldomain.com
    IdentityFile ~/.ssh/prod_rsa
    User produser
Host *.devdomain.com
    IdentityFile ~/.ssh/dev_rsa
    User devuser

Berkshelf support

littlechef supports Berkshelf. If given file exists littlechef will execute berks vendor berksfile-cookbooks-directory, if use do not provide berksfile_cookbooks_directory then random directory in tmp is used. If user want's to upload some cookbooks which are not tracked in Berskfile, they have to place them to cookbooks directory.

[kitchen]
berksfile = Berksfile
berksfile_cookbooks_directory = berks-cookbooks

Other Configuration Options

You can also optionally override the directory being used on the nodes to sync your kitchen to:

[kitchen]
node_work_path = /tmp/chef-solo

You can use encrypted data bags. Create secret keys, Use knife-solo_data_bag Gem to create encrypted data bags, and specify a path for the encrypted_data_bag_secret file:

[userinfo]
encrypted_data_bag_secret = ~/path/to/encrypted_data_bag_secret

This will put the encrypted_data_bag_secret in /etc/chef/encrypted_data_bag_secret with permissions root:root with perms 0600. Chef-solo will automatically use it wherever you use Chef::EncryptedDataBagItem.load in your recipes. It will also remove the /etc/chef/encrypted_data_bag_secret file from the node at the end of the run.

If you are not comfortable about leaving data bags in the remote node, which will leak sensitive information, you can tell littlechef to delete them all at the end of the run:

[userinfo]
remove_data_bags=true
[userinfo]
autodeploy_chef=true

if set to true, a check will be performed before each deployment for chef-solo, and if it is not present it will be installed using the omnibus method.

[connection]
gateway = hub.example.com

If you want to use http/https proxy with chef_solo run. You have to add following entries to config file. They will create solo.rb config file with http/https proxy configured.

[connection]
http_proxy = "http://172.14.1.12:8888"
https_proxy = "http://172.14.1.12:8888"

The sync-packages section allows you to define remote and local directories, which will then be synchronized at every run.

[sync-packages]
dest-dir = /srv/packages
local-dir = ./packages

Deploying chef-solo

For convenience, there is a command that allows you to deploy chef-solo to a node.

The best way is to use the omnibus method getchef: fix node:MYNODE deploy_chef:version=11.12

You can also install Chef Solo without asking for confirmation: fix node:MYNODE deploy_chef:ask=no

Note that if you already have Chef Solo installed on your nodes, you won't need this. Also, if you previously installed Chef using any other procedure, please don't use the deploy_chef installation method, removing chef first might be a good idea.

Multihop littlechef setup

If your nodes are not directly accessible, you might want to specify a gateway host. The fix command will connect to the host specified and issue all following connections from this host. All ssh communication will be tunneled through this gateway connection. This can be used if your nodes are behind a firewall and only one host is accessible from your current network location.

[connection]
gateway = hub.example.com

After issuing a fix command, this will connect to hub.example.com. All further node connections will be done from hub.example.com.

To be able to properly connect to hub.example.com same host should be defined in ssh-config. Because final connection is made from a hub.example.com any required ssh keys must be available on it or user needs to setup ssh-agent forwarding.

Cooking

Note: Don't cook outside of a kitchen!

List of commands:

Options:

Once a node has a config file, the command you will be using most often is fix node:MYNODE, which allows you to repeatedly tweak the recipes and attributes for a node and rerun the configuration.

Configuring nodes in parallel

By default LittleChef configures nodes serially however it can also use Fabric's parallel SSH support to configure multiple nodes in parallel. All commands are supported (node, nodes_with_role, ssh, role, and recipe)

Consulting the inventory

Using LittleChef as a library

You can import littlechef.py into your own Python project. The following script is equivalent to using the fix orders:

from littlechef import runner as lc
lc.env.user = 'MyUsername'
lc.env.password = 'MyPassword'
lc.env.host_string = 'MyHostnameOrIP'
lc.deploy_chef(gems='yes', ask='no')

lc.recipe('MYRECIPE') #Applies <MYRECIPE> to <MyHostnameOrIP>
lc.node('MyHostnameOrIP') #Applies the saved nodes/MyHostnameOrIP.json configuration

Performance Tips

You can greatly reduce the SSH connection setup time by reusing existing connections. On Unix systems, you can do so by adding the ControlMaster directive to your ssh config:

#~/.ssh/config
Host *
  ControlMaster auto
  ControlPath /tmp/ssh-%r@%h:%p

Other tutorial material

Getting help

For help regarding the use of LittleChef, or to share any ideas or suggestions you may have, please post on LittleChef's discussion group

Reporting bugs

Build Status

If you find bugs please report it on https://github.com/tobami/littlechef/issues

Happy cooking!