See our blog post for the philosophy behind Runbook and an overview of its features.
Runbook provides a DSL for specifying a series of steps to execute an operation. Once your runbook is specified, you can use it to generate a formatted representation of the book or to execute the runbook interactively. For example, you can export your runbook to markdown or use the same runbook to execute commands on remote servers.
Runbook has two modes for evaluating your runbook. The first mode, view mode, allows you to export your runbook into various formats such as markdown. The second mode, run mode, allows you to execute behavior based on the statements in your runbook.
Runbook can be integrated into existing infrastructure in many different ways. It can be integrated into existing projects to add orchestration functionality, installed on systems as a stand-alone executable, or runbooks can be defined as self-executable scripts. In addition to being useful for automating common tasks, runbooks are a perfect bridge for providing operations teams with step-by-step instructions to handle common issues (especially when solutions cannot be easily automated).
Lastly, Runbook provides an extendable interface for augmenting the DSL and defining your own behavior.
Though Runbook can solve a myriad of problems, it is best used for removing the need for repeated, rote developer operations. Runbook allows developers to execute processes at a higher level than that of individual command-line commands. Additionally, Runbook provides features to simply and safely execute operations in mission-critical environments.
Runbook is not intended to replace more special-purpose automation solutions such as configuration management solutions (Puppet, Chef, Ansible, Salt), deployment solutions (Capistrano, Kubernetes, Docker Swarm), monitoring solutions (Nagios, Datadog), or local command execution (shell scripts, Rake tasks, Make). Instead Runbook is best used as a glue when needing to accomplish a task that cuts across these domains.
Add this line to your application's Gemfile:
gem 'runbook'
And then execute:
$ bundle
Or install it yourself as:
$ gem install runbook
Generate a runbook using the Runbook Generator:
$ runbook generate runbook my_first_runbook
Execute the runbook:
$ runbook exec my_first_runbook.rb
When setting up Runbook, you can install it at a system level, create a dedicated runbook project, or incorporate Runbook into an existing project.
Install runbook at a system level using gem
:
$ gem install runbook
Set any Runbook configuration in /etc/runbook.conf
.
Generate runbook files using runbook generate runbook
. Execute runbook generate help runbook
for more details.
Installing Runbook at a system level can be useful for executing runbooks on remote hosts or within docker containers. One disadvantage of installing Runbook at a system level is that there is no built-in solution for dependency management.
Install runbook using gem
:
$ gem install runbook
Generate a new runbook project:
$ runbook generate project <PROJECT_NAME>
This will generate a new runbook project. Cd into your project directory and initialize its dependencies:
$ cd <PROJECT_NAME> && bin/setup
Add this line to your project's Gemfile:
gem 'runbook'
Install the Runbook gem:
$ bundle install
Initialize Runbook in your project:
$ bundle exec runbook init
Below is an example of a runbook:
Runbook.book "Restart Nginx" do
description <<-DESC
This is a simple runbook to restart nginx and verify
it starts successfully
DESC
section "Restart Nginx" do
server "app01.prod"
user "root"
step "Stop Nginx" do
note "Stopping Nginx..."
command "service nginx stop"
assert %q{service nginx status | grep "not running"}
end
step { wait 5 }
step "Start Nginx" do
note "Starting Nginx..."
command "service nginx start"
assert %q{service nginx status | grep "is running"}
confirm "Nginx is taking traffic"
notice "Make sure to report why you restarted nginx"
end
end
end
Hierarchically, a runbook looks like this:
A runbook is composed of entities, statements, and setters. Runbook entities contain either other entities or statements. Examples of entities include Books, Sections, and Steps. They define the structure of the runbook and can be considered the "nodes" of the tree structure. As entities are the nodes of the tree structure, statements are the "leaves" of the structure and comprise the various behaviors or commands of the runbook. Setters, typically referenced from within steps, associate state with the node, which can be accessed by its children.
Entities are composed of a title and a list of items which are their children. Each entity can be rendered with a specific view or executed with a specific run.
Books are the root of a runbook. They are initialized as follows:
Runbook.book "Unbalance node" do
end
Every book requires a title. Books can have description, layout, section, and step children. Descriptions describe the book and are declared with the description
keyword.
A book is broken up into sections. Every section requires a title. Sections can have descriptions, other sections, or steps as children.
Steps hold state and group together a set of statements. Steps do not require titles or children. This allows runbooks to be very flexible. You can fill out steps as needed, or be terse when the behavior of the step is self-evident. Steps without titles will not prompt to continue when running in paranoid mode.
Setup is a special entity that is always executed. It is not skipped when starting or resuming execution in the middle of a runbook. A prompt is never presented to determine if you should or should not execute the setup section. The setup section is similar to the step entity in that it shares the same DSL. In other words, any keywords available within steps are also available within the setup section.
The setup section provides two important use cases. First, it allows you ensure any dependent values are defined when executing your runbook. If skipping the initial steps of your runbook and starting in the middle, you can be sure that any initialization steps have been executed. For example, presume you have a runbook that prompts for a list of servers, stops the servers, and then starts them. It would be advantageous to define the prompting server logic in a setup section, so you can start the runbook at the start server step and know that the list of servers is defined.
Second, if you dynamically define the sections and steps in your runbook based on user input, then doing this in the setup section allows you start the runbook in the middle of the dynamically defined steps.
Because the setup section is always executed, it's execution should be idempotent. In other words, the setup section should be able to be executed multiple times in a row and produce the same result.
It may be ideal to ensure user input is only asked for once when executing a setup section.
Runbook.book "Make Pizza" do
setup do
ruby_command do
@toppings ||= ENV["TOPPINGS"]
ask "What toppings would you like?", into: :toppings, default: "cheese" unless @toppings
end
end
end
The above example will set @toppings
from a passed-in environment variable if present, otherwise it will ask the user to set @toppings
. If toppings have already has been defined from a previous execution, it will not prompt the user for the value again. Because this logic references a value that is defined at runtime (@toppings
), it must be wrapped in a ruby_command
.
Any entity can be associated with arbitrary tags or labels. Once tags or labels are assigned, entity behavior can be modified using hooks.
Runbook.book "Bounce Nodes", :untested do
step "Disable monitoring", :skip do
confirm "Have you disabled health monitoring?"
end
step "Restart nodes", :aws_only, :mutator, labels: {rails_env: :production} do
confirm "Have you restarted the nodes?"
end
end
Runbook::Runs::SSHKit.register_hook(:warn_for_untested_runbook, :before, Runbook::Entities::Book) do |object, metadata|
warning = "This runbook has not yet been tested. Beware of bugs!"
metadata[:toolbox].warn(warning) if object.tags.include?(:untested)
end
Runbook::Runs::SSHKit.register_hook(:skip_skippable_entities, :around, Runbook::Entity) do |object, metadata, block|
next if object.tags.include?(:skip)
next if object.labels[:rails_env] && object.labels[:rails_env] != ENV["RAILS_ENV"]
block.call
end
Statements are the workhorses of runbooks. They comprise all the behavior runbooks execute. Runbook comes with the following statements:
Prompts the user for a string and stores its value on the containing step entity. Once this statement is executed, its value is accessed as an instance variable under the into
parameter. This value can be referenced in later statements such as the ruby_command
statement. An optional default
value can be specified. An optional echo
parameter can be specified to indicate whether typed input should be echoed to the screen.
ask "What percentage of requests are failing?", into: :failing_request_percentage, default: "100", echo: true
ruby_command do
note "Failing request percentage: #{@failing_request_percentage}"
end
In the above example, the note
statement must be wrapped in a ruby_command
statement. Without wrapping note
in a ruby_command
, it would be evaluated at compile time but the user will only be asked for input when the runbook is executed (so @failing_request_percentage
would not have a value). If you find yourself wrapping many or all runbook statements in ruby commands it may make sense to set these values at compile time using environment variables.
Runs the provided cmd
repeatedly until it returns true. A timeout and maximum number of attempts can be set. When either the attempt or timeout limit is hit, a command can be specified that will be run. If no command is specified, the process will fail. Commands can optionally be specified as raw
. This tells SSHKit to not perform auto-wrapping of the commands, but execute the exact string on the remote server. See SSHKit's documentation for more details.
assert(
'service nginx status | grep "is running"',
cmd_ssh_config: {servers: ["host1.prod"], parallelization: {strategy: :parallel}},
cmd_raw: false,
interval: 3, # How often, in seconds, to wait between tries
timeout: 300, # Total time, in seconds, to keep trying command, after which it will fail
attempts: 3, # Total number of attempts after which the process will fail
abort_statement: Runbook::Statements::Command.new(
"echo 'help' | mail -s 'need help' page-me@page-me.com",
ssh_config: {servers: [:local], parallelization: {strategy: :parallel}},
raw: false
)
)
Runs the provided cmd
and captures its output into into
. Once captured, this value can be referenced in later statements such as the ruby_command
statement. An optional ssh_config
can be specified to configure how the capture command gets run. Capture commands take an optional strip
parameter that indicates if the returned output should have leading and trailing whitespace removed. Capture commands also take an optional raw
parameter that tells SSHKit whether the command should be executed as is, or to include the auto-wrapping of the ssh_config.
capture %Q{wc -l file.txt | cut -d " " -f 1}, into: :num_lines, strip: true, ssh_config: {user: "root"}
Accepts the same parameters as capture
, but returns a hash of server names to capture results. capture_all
should be used whenever multiple servers are specified because the returned result of capture
is non-deterministic when specifying multiple servers.
capture_all %Q{wc -l file.txt | cut -d " " -f 1}, into: :num_lines, strip: true, ssh_config: {servers: ["host1.stg", "host2.stg"]}
Runs the provided cmd
. An optional ssh_config
can be specified to configure how the command gets run. Commands also take an optional raw
parameter that tells SSHKit whether the command should be executed as is, or to include the auto-wrapping of the ssh_config.
command "service nginx start", ssh_config: {servers: ["host1.prod", "host2.prod"], parallelization: {strategy: :groups}}
Proposes the prompt to the user and exits if the user does not confirm the prompt.
confirm "Asset requests have started trickling to the box"
Prints the description in an unformatted manner to the user
description <<-DESC
This message will print directly to the user as written, without
additional formatting.
DESC
Downloads the specified file to to
. An optional ssh_config
can be specified to configure how the download command gets run, for example specifying the remote host and remote directory to download from. Optional options
can be specified that get passed down to the underlying sshkit implementation
download '/home/pblesi/rad_file.txt', to: my_rad_file.txt, ssh_config: {servers: ["host1.prod"]}, options: {log_percent: 10}
Defines a tmux layout to be used by your runbook. When executing the runbook, the specified layout will be initialized. This statement can only be specified at the book level. See Tmux Layouts for more details.
layout [[
[:runbook, :deploy],
[:monitor_1, :monitor_2, :monitor_3],
]]
Prints a short note to the user.
note "This operation kills all zombie processes"
Prints out an important message to the user.
notice "There be dragons!"
Executes its block in the context of the parent step. The block is passed the ruby_command statement, the execution metadata, and the run as arguments.
ruby_command do |rb_cmd, metadata, run|
if (failure_rate = rb_cmd.parent.failing_request_percentage) > 25
`echo 'Help! failure rate at #{failure_rate}' | mail -s 'High failure rate!' page-me@page-me.com`
else
`echo "Experienced failure rate of #{failure_rate}" | mail -s 'Help me eventually' not-urgent@my_site.com`
end
notice "Email sent!"
end
Metadata at execution time is structured as follows:
{
book_title: "Restart Nginx", # The title of the current runbook
depth: 1, # The depth within the tree (book starts at depth 1)
index: 0, # The index of the item in terms of it's parent's children (starts at 0 for first child)
position: "1.1", # A string representing your current position within the tree
noop: false, # A boolean indicating if you are running in noop mode. ruby_command blocks are never evaluated in noop mode
auto: false, # A boolean indicating if you are running in auto mode
paranoid: true, # A boolean indicating if you are running in paranoid mode (prompting before each step)
keep_panes: false, # A boolean indicating whether panes should be kept open after completion
start_at: 0, # A string representing the step where nodes should start being processed
toolbox: Runbook::Toolbox.new, # A collection of methods to invoke side-effects such as printing and collecting input
layout_panes: {}, # A map of pane names to pane ids. `layout_panes` is used by the `tmux_command` to identify which tmux pane to send the command to
repo: {}, # A repository for storing data and retrieving it between ruby_commands. Any data stored in the repo is persisted if a runbook is stopped and later resumed.
}
Additional methods that the ruby_command
block has access to are:
metadata[:toolbox].prompt
: A TTY::Prompt
for retrieving input from the usermetadata[:toolbox].ask(msg)
: retrieve user inputmetadata[:toolbox].yes?(msg)
: provide the user with a yes/no promptmetadata[:toolbox].output(msg)
: output text to the usermetadata[:toolbox].warn(msg)
: output warning text to the usermetadata[:toolbox].error(msg)
: output error text to the usermetadata[:toolbox].exit(return_value)
: exit the process with the specified response codeRuns the provided cmd
in the specified pane
.
tmux_command "tail -Fn 100 /var/log/nginx.log", :monitor_1
Uploads the specified file to to
. An optional ssh_config
can be specified to configure how the upload command gets run, for example specifying the remote host and remote directory to upload to. Optional options
can be specified that get passed down to the underlying sshkit implementation
upload my_secrets.yml, to: secrets.yml, ssh_config: {servers: ["host1.prod"]}, options: {log_percent: 10}
Sleeps for the specified amount of time (in seconds)
wait 5
Runbook provides native support for defining tmux layouts and executing commands in separate tmux panes. Layouts are specified by passing an array or hash to the layout
statement in book blocks.
Runbook.book "My Book" do
layout [
:left,
{name: :middle, directory: "/var/log", command: "tail -Fn 100 auth.log"},
[:top_right, {name: :bottom_right, runbook_pane: true}]
]
end
When layout is passed as an array, each element of the array represents a pane stacked side-by-side with the other elements. Elements of the array can be symbols, hashes, or arrays.
Symbols and hashes represent panes. Hash keys for a pane include name
, directory
, command
, and runbook_pane
. name
is the identifier used for the pane. This is used when specifying what pane you want to execute tmux commands in. directory
indicates the starting directory of the pane. command
is the initial command to execute in the pane when it is created. runbook_pane
indicates which pane in the layout should hold the executing runbook. Only one pane should be designated as the runbook_pane
and the runbook pane should not have a directory or command specified.
Arrays nested underneath the initial array split the pane from top to bottom. Arrays nested under these arrays split the pane from side to side, ad infinitum. You can start spliting panes from top to bottom as opposed to side-by-side by immediately nesting an array.
Runbook.book "Stacked Layout" do
layout [[
:top,
:middle,
:bottom,
]]
end
When a hash is passed to layout
, the keys of the hash represent window names and the values represent pane layouts.
Runbook.book "Multi Window Layout" do
layout({
:web_monitor => [
:left, :middle, :right,
],
:db_monitor => [[
:top, :middle, :bottom,
]]
})
end
Notice in the example that parenthesis are used to wrap the hash. Ruby will raise a syntax error if layout
's argument is not wrapped in parenthesis when passing a hash. Runbook expects that it is running in the last window in a tmux session. If you are running a runbook that uses a multi-window layout, the layout will not work unless runbook is running in the last window in the session.
If you want panes to be un-evenly spaced, you can replace the array of panes with a hash where the keys are panes and the values are numbers. The panes will be spaced according to the specified numbers.
Runbook.book "Uneven Layout" do
layout [[
{:left => 20, {name: :middle, runbook_pane: true} => 60, :right => 20},
{:bottom_left => 5, :bottom_right => 5},
]]
end
Tmux layouts are persisted between runs of the same runbook. As long as none of the panes initially created by the runbook are closed, running the same runbook in the same pane will not recreate the tmux layout, but will reuse the existing layout. This is helpful when a runbook does not complete and must be restarted. When a runbook finishes, it asks if you want to close all opened panes. If your runbook is running in auto mode it will automatically close all panes when finished.
Setters set state on the parent item, typically the containing step. Runbook comes with the following setters:
parallelization: Specifies the SSHKit parallelization parameters for all commands in the entity. The default parallelization strategy is :parallel
. Other strategies include :sequence
and :groups
. See SSHKit for more details on these options.
parallelization strategy: :parallel, limit: 2, wait: 2
server: Specifies the server to use for all commands in the entity. This command in conjunction with servers
are declarative and overwrite each other. So if you specify server
once, servers
twice and finally, server
again, only the last designation will be used to run the commands.
server "db01.qa"
servers: Used to specify a list of servers for the entity. All commands contained in this entity will be run against this list of servers (unless they have been overridden by a lower config.)
servers "app01.qa", "app02.qa"
path: Specify the path from which commands in this step will execute.
path "/home/sholmes"
user: Specify the user that the command will be run as
user "root"
group: Specify the effective group the commands will be run as
group "devs"
env: Specify the environment for the commands
env {rails_env: :production}
umask: Specify the umask the commands will be run with
umask "077"
Additionally, Step
provides an ssh_config
helper method for generating ssh_configs that can be passed to command statements.
step do
cmd_ssh_config = ssh_config do
server "host1.qa"
user "root"
end
command "echo $USER", ssh_config: cmd_ssh_config
end
You can integrate with Runbook in several different ways. You can create your own project or incorporate Runbook into your existing projects. You can use Runbook via the command line. And you can even create self-executing runbooks.
Runbook can be used to write stand-alone runbook files that can be executed via the command line. Below is a list of examples of how to use Runbook via the command line.
Get Runbook usage instructions
$ runbook help
Render my_runbook.rb
in the default view format (markdown)
$ runbook view my_runbook.rb
Execute my_runbook.rb
using the default executor (ssh_kit)
$ runbook exec my_runbook.rb
Execute my_runbook.rb
in no-op mode, preventing commands from executing.
$ runbook exec --noop my_runbook.rb
Execute my_runbook.rb
in auto mode. Runbooks that are executed in auto mode do not prompt the user for input.
$ runbook exec --auto my_runbook.rb
Execute my_runbook.rb
starting at position 1.2.1. All prior steps in the runbook will be skipped
$ runbook exec --start-at 1.2.1 my_runbook.rb
Execute my_runbook.rb
without confirmation between each step
$ runbook exec --no-paranoid my_runbook.rb
Environment variables can be specified via the command line, modifying the behavior of the runbook at runtime.
$ HOSTS="appbox{01..30}.prod" ENV="production" runbook exec --start-at 1.2.1 my_runbook.rb
Runbooks can be executed using the Runbook::Viewer
and Runbook::Runner
classes. Using these classes, you can invoke runbooks from within your existing codebase. This could be ideal for several reasons. It allows you to maintain a consistent interface with other system tasks such as rake tasks or cap tasks. It allows you to perform setup or sanity check functionality before executing your runbooks. And it allows you to load an environment to be accessed within your runbooks, such as providing access to a canonical list of servers or shared business logic.
Runbook::Viewer
Runbook::Viewer.new(book).generate(view: :markdown)
In this case book is a Runbook::Entities::Book
and :markdown
refers to the specific view type (Runbook::Views::Markdown
).
Runbook::Runner
Runbook::Runner.new(book).run(run: :ssh_kit, noop: false, auto: false, paranoid: true, start_at: "0")
This will execute book
using the Runbook::Runs::SSHKit
run type. It will not run the book in noop
mode. It will not run the book in auto
mode. It will run the book in paranoid
mode. And it will start at the beginning of the book. Noop mode runs the book without side-effects outside of printing what it will execute. Auto mode will skip any prompts in the runbook. If there are any required prompts in the runbook (such as the ask
statement), then the run will fail. Paranoid mode will prompt the user for whether they should continue at every step. Finally start_at
can be used to skip parts of the runbook or to restart at a certain point in the event of failures, stopping and starting the runbook, etc.
Runbooks can be written to be self-executable
#!/usr/bin/env ruby
# my_runbook.rb
require "runbook"
runbook = Runbook.book "Say hello to world" do
section "Address the world" do
step { command "echo 'hello world!'" }
step { confirm "Has the world received your greeting?" }
end
end
if __FILE__ == $0
Runbook::Runner.new(runbook).run
else
runbook
end
This runbook can be executed via the command line or evaluated from within an existing project
$ ./my_runbook.rb
load "my_runbook.rb"
runbook = Runbook.books.last # Runbooks register themselves to Runbook.books when they are defined
# (Or alternatively `runbook = eval(File.read("my_runbook.rb"))`)
Runbook::Runner.new(runbook).run(auto: true)
Runbook is configured using its configuration object. Below is an example of how to configure Runbook.
Runbook.configure do |config|
config.ssh_kit.umask = "077"
config.ssh_kit.default_runner_config = {in: :groups, limit: 5}
config.ssh_kit.default_env = {rails_env: :staging}
config.enable_sudo_prompt = true
config.use_same_sudo_password = true
end
If the ssh_kit
configuration looks familiar, that's because it's an SSHKit Configuration object. Any configuration options set on SSHKit.config
can be set on config.ssh_kit
.
Runbook loads configuration from a number of predefined files. Runbook will attempt to load configuration from the following locations on startup: /etc/runbook.conf
, a Runbookfile
in a parent directory from the current directory, a .runbook.conf
file in the current user's home directory, a file specified with --config
on the command line, any configuration specified in a runbook. Runbook will also load configuration from these files in this order of preference, respectively. That is, configuration values specified at the project level (Runbookfile
) will override configuration values set at the global level (/etc/runbook.conf
), etc.
The following are best practices when developing your own runbooks.
Runbooks allow for a gradual transition from entirely manual operations to full automation. Runbooks can start out as a simple outline of all steps required to carry out an operation. From there, commands and prompts can be added to the runbook, actually carrying out and replacing the manual processes.
Monitoring can transition from a process required by a human into something that can be codified and executed by your runbook. Eventually, the runner's auto
flag can be used to allow the runbook to run uninterrupted without any human intervention. These runbooks can be triggered automatically in response to detected events. This will allow you to do more important things with your time, like eat ice cream.
You will typically want to parameterize your runbooks so they can be run against different hosts or in different environments. Because runbooks are Ruby, you have a multitude of options for parameterizing your runbooks, from config files, to getting host information via shell commands, to using environment variables. Here's an example of a few of these methods:
host = ENV["HOST"] || "<host>"
replication_host = ENV["REPLICATION_HOST"] || "<replication_host>"
env = `facter environment`
rails_env = `facter rails_env`
customer_list = File.read("/tmp/customer_list.txt")
Runbook provides a number of different mechanisms for passing state throughout a runbook. For any data that is known at compile time, local variables can be used because Runbooks are lexically scoped.
home_planet = "Krypton"
Runbook.book "Book Using Local Variables" do
hometown = "Smallville"
section "My Biography" do
step do
note "Home Planet: #{home_planet}"
note "Home Town: #{hometown}"
end
end
end
When looking to pass data generated at runtime, for example data from ruby_command
, ask
, or capture
statements, Runbook persists and synchronizes instance variables for these commands.
Runbook.book "Book Using Instance Variables" do
section "The Transported Man" do
step do
ask "Who's the greatest magician?", into: :greatest, default: "Alfred Borden"
ruby_command { @magician = "Robert Angier" }
end
step do
ruby_command {
note "Magician: #{@magician}"
note "Greatest Magician: #{@greatest}"
}
end
end
end
Instance variables are only passed between statements such as ruby_command
. They should not be set on entities such as steps, sections, or books. Instance variables are persisted using metadata[:repo]
. They are copied to the repo after each statement finishes executing and copied from the repo before each statement starts executing. Because instance variables utilize the repo, they are persisted if the runbook is stopped and restarted at the same step.
Be careful with your naming of instance variables as it is possible to clobber the step's DSL methods because they share the same namespace.
As a best practice, Runbooks should always be nooped before they are run. This will allow you to catch runtime errors such as using the ask statement when running in auto mode, typos in your runbooks, and to visually confirm what will be executed.
Additionally, it can be nice to have a generated view of the runbook you are executing to have a good high-level overview of the steps in the runbook.
Runbook uses SSHKit for remote command execution. When specifying servers
, you are specifying the target host to execute the command. If you want to use a non-standard port or login using a different user than your current user, then you can specify the server
as lucy@host1.prod:2345
. Alternatively, you can use an ssh config file such as ~/.ssh/config
to specify the user and port used to ssh to a given host. See Capistrano's SSH setup instructions for further support on setting up SSH to execute commands on remote hosts.
The user
setter designates the user you will sudo as once sshed to the remote host. Runbook supports password-protected sudo execution. That is, if your server requires a password to execute commands as another user, Runbook will allow you to enter your password when prompted. The enable_sudo_prompt
configuration value controls this behavior. Enabling the sudo password prompt requires that your commands execute using a tty, which can lead to unexpected behavior when executing certain commands. Enabling use_same_sudo_password
will use the same password accross different hosts and users instead of re-prompting for each unique user/host combo.
Runbooks can be composed using the add
keyword. Below is an example of composing a runbook from smaller, reusable components.
restart_services_section = Runbook.section "Restart all services" do
step "Restart nginx"
step "Restart postgres"
end
Runbook.book "Update configuration" do
section "Change config" do
command "sed -i 's/listen 8080;/listen 80;/' /etc/nginx/nginx.conf"
end
add restart_services_section
end
If you want to parameterize these runbook snippets, you can place them in a ruby function that takes arguments and generates the desired entity or statement. If these snippets set information that is used by the runbook, such as with capture
statements, it is a good practice to parameterize where the result is stored. This lets the snippet fit different contexts and makes clear what data is being returned from the snippet.
Because the Runbook DSL is declarative, it is generally discouraged to develop elaborate nested decision trees. For example, it is discouraged to use the ask
statement to gather user feedback, branch on this information in a ruby_command
, and follow completely separate sets of steps. This is because deep nesting eliminates the benefits of the declarative DSL. You can no longer noop the deeply nested structure for example.
If you are looking to make a complex decision tree, it is recommended that you do this by composing separate runbooks or entities and nooping those entities separately to ensure they work as expected. Below is an example of a few different ways to compose nested runbooks
step "Inspect plate" do
ask "What's on the plate?", into: :vegetable
ruby_command do |rb_cmd, metadata|
case (veggie = @vegetable)
when "carrots"
add carrots_book
when "peas"
system("runbook exec examples/print_peas.rb")
else
metadata[:toolbox].warn("Found #{veggie}!")
end
end
end
The first delegation add carrots_book
adds the book to the execution tree of the current runbook. Sections and steps become sub-sections and sub-steps of the current step. The second delegation spins up an entirely new process to run the print_peas
runbook in isolation. Either delegation could be preferred, depending on your needs.
Runbooks can be loaded from files using load
or eval
:
load "my_runbook.rb"
runbook = Runbook.books.last # Runbooks register themselves to Runbook.books when they are defined
runbook = eval(File.read("my_runbook.rb"))
Loading your runbook file is more ideal, but adds slight complexity. This method is prefered because the Ruby mechanism for retrieving source code does not work for code that has been eval
ed. This means that you will not see ruby_command
code blocks in view and noop output when using the eval
method. You will see an "Unable to retrieve source code" message instead.
Runbook provides a number of generators accessible via the command line that can be used to generate code for new runbooks, Runbook projects, and Runbook extensions. Additionally, Runbook provides a generator generator so you can define your own custom generators.
Runbook provides a number of predefined generators. You can see the full list using Runbook's command line help.
$ runbook help generate
Commands:
runbook generate dsl_extension NAME [options] # Generate a dsl_extension for adding custom runbook DSL functionality
runbook generate generator NAME [options] # Generate a runbook generator named NAME, e.x. acme_runbook
runbook generate help [COMMAND] # Describe subcommands or one specific subcommand
runbook generate project NAME [options] # Generate a project for your runbooks
runbook generate runbook NAME [options] # Generate a runbook named NAME, e.x. deploy_nginx
runbook generate statement NAME [options] # Generate a statement named NAME (e.x. ruby_command) that can be used in your runbooks
Base options:
-c, [--config=CONFIG] # Path to runbook config file
[--root=ROOT] # The root directory for your generated code
# Default: .
Runtime options:
-f, [--force] # Overwrite files that already exist
-p, [--pretend], [--no-pretend] # Run but do not make any changes
-q, [--quiet], [--no-quiet] # Suppress status output
-s, [--skip], [--no-skip] # Skip files that already exist
Unless otherwise specified, all NAME
arguments should be specified in a snake case format (e.x. acme_runbook
). The -p/--pretend
flag can be helpful for seeing what files a generator will create before it creates them.
The generator generator is useful for defining your own custom generators. Runbook uses Thor Generators in the background, so any functionality you can do using Thor Generators can also be done using Runbook generators.
Generate your own generator using the generate generator
command
$ runbook generate generator my_new_generator --root lib/runbook/generators
create my_new_generator
create my_new_generator/templates
create my_new_generator/my_new_generator.rb
my_new_generator/my_new_generator.rb
contains all the logic for generating your new code including arguments, options, and new files. ERB-templated files live in my_new_generator/templates
. Remember to require your generator file in a runbook config file such as your Runbookfile
so it can be loaded by the CLI. Generators cannot be required in config files specified at the command line due to the order with which the command line code is loaded. Once loaded, any child classes of Runbook::Generators
will be included in Runbook's generator CLI.
Runbook can be extended to add custom functionality.
In order to add a new statement to your DSL, create a class under Runbook::Statements
that inherits from Runbook::Statement
. This statement will be initialized with all arguments passed to the corresponding keyword in the DSL. Remember to also add a corresponding method to runs and views so your new statement can be interpreted in each context.
module Runbook::Statements
class Diagram < Runbook::Statement
attr_reader :alt_text, :url
def initialize(alt_text, url)
@alt_text = alt_text
@url = url
end
end
end
In the above example a keyword diagram
will be added to the step dsl and its arguments will be used to initialize the Diagram object.
New statements can be generated using the statement generator.
$ runbook generate statement diagram --root lib/runbook/extensions
You can add handlers for new statements and entities to your runs and views by prepending the modules with the new desired functionality.
module MyRunbook::Extensions
module Diagram
def self.runbook__entities__diagram(object, output, metadata)
output << "![#{object.alt_text}](#{object.url})"
end
end
Runbook::Views::Markdown.prepend(Diagram)
end
If you are not modifying existing methods, you can simply re-open the module to add new methods.
You can add arbitrary keywords to your entity DSLs. For example, you could add an alias to Runbook's Book DSL as follows:
module MyRunbook::Extensions
module Aliases
module DSL
def s(title, &block)
section(title, &block)
end
end
end
Runbook::Entities::Book::DSL.prepend(Aliases::DSL)
end
DSL extensions can be generated using the dsl_extension generator.
$ runbook generate dsl_extension aliases --root lib/runbook/extensions
You can add new run and view types by defining modules under Runbook:::Runs
and Runbook::Views
respectively. They will automatically be accessible from the command line or via the Runner
and Viewer
classes. See lib/runbook/runs/ssh_kit.rb
or lib/runbook/views/markdown.rb
for examples of how to implement runs and views.
module Runbook::Views
module Yaml
include Runbook::View
# handler names correspond to the entity or statement class name
# Everything is underscored and "::" is replaced by "__"
def self.runbook__entities__book(object, output, metadata)
output << "---\n"
output << "book:\n"
output << " title: #{object.title}\n"
end
# Add other handlers here
end
end
You can add before
, after
, or around
hooks to any statement or entity by defining a hook on a Run
or View
.
Runbook::Runs::SSHKit.register_hook(
:notify_slack_of_step_run_time,
:around,
Runbook::Entities::Step
) do |object, metadata, block|
start = Time.now
block.call(object, metadata)
duration = Time.now - start
unless metadata[:noop]
message = "Step #{metadata[:position]}: #{object.title} took #{duration} seconds!"
notify_slack(message)
end
end
When registering a hook, you specify the name of the hook, the type, and the statement or entity to add the hook to. before
and after
hooks execute the block before and after executing the entity or statement, respectively. around
hooks take a block which executes the specified entity or statement. When specifying the class that the hook applies to, you can have the hook apply to all entities by specifying Runbook::Entity
, all statements by specifying Runbook::Statement
, or all items by specifying Object
. Additionally, you can specify any specific entity or statement you would like the hook to apply to.
Hooks are defined on the run or view objects themselves. For example, you would register a hook with Runbook::Runs::SSHKit
to have the hook be applied to the SSHKit
run. You would register a hook with the Runbook::Views::Markdown
view to have hooks apply to this view. If you want to apply a hook to all runs or views, you can use the Runbook.runs
method or Runbook.views
method to iterate through the runs or views respectively.
Runbook.runs.each do |run|
run.register_hook(
:give_words_of_encouragement,
:before,
Runbook::Entities::Book
) do |object, metadata|
metadata[:toolbox].output("You've got this!")
end
end
Hooks can be defined anywhere prior to runbook execution. If defining a hook for only a single runbook, it makes sense to define the hook immediately prior to the runbook definition. If you want a hook to apply to all runbooks in your project, it can be defined in a config file such as the Runbookfile
. If you want to selectively apply the hook to certain runbooks, it may make sense to define it in a file that can be required by runbooks when it is needed.
When starting at a certain position in the runbook, hooks for any preceding sections and steps will be skipped. After hooks will be run for a parent when starting at a child entity of a parent.
Every Entity and Statement gets access to a Toolbox in metatada[:toolbox]
. This toolbox is used to provide methods with side effects (such as printing messages) when rendering and running your runbooks. Additional behaviors can be added to the toolbox by prepending Runbook::Toolbox
.
module MyRunbook::Extensions
module Logger
def initialize
super
log_file = ENV["LOG_FILE"] || "my_log_file.log"
@logger = Logger.new(log_file)
end
def log(msg)
@logger.info(msg)
end
end
Runbook::Toolbox.prepend(Logger)
end
Now you can access log
in your handler code using metadata[:toolbox].log("Come on ride the train, train")
.
module MyRunbook::Extensions
module Logging
def self.runbook__entities__book(object, metadata)
super
metadata[:toolbox].log("Executing #{object.title}")
end
end
Runbook::Runs::SSHKit.prepend(Logging)
end
You may want to add additional data to metadata at the time it is initialized so every node can have access to this data. You can add additional metadata to runs by prepending Runbook::Runner
.
module MyRunbook::Extensions
module RunbookNotesMetadata
def additional_metadata
super.merge({
notes: []
})
end
end
Runbook::Runner.prepend(RunbookNotesMetadata)
end
You can add additional configuration to Runbook's configuration by prepending Runbook::Configuration.
module MyRunbook::Extensions
module Configuration
attr_accessor :log_level
def initialize
super
self.log_level = :info
end
end
Runbook::Configuration.prepend(Configuration)
end
This will add a log_level
attribute to Runbook's configuration with a default value of :info
. This configuration value can be accessed via Runbook.config.log_level
.
Runbooks are inherently difficult to test because they are primarily composed of side-effects. That being said, there are a number of strategies you can employ to test your runbooks.
TEST
or DEBUG
environment variables to conditionally disable side-effects during executionSee Runbook's test suite for more ideas on how to test your runbooks. For example, Runbook uses aruba to test Runbook at the CLI level.
Additionally, runbooks should contain their own assertions, sanity checks, monitoring, and alerting to mitigate errors and alert you if intervention is required.
Because ssh_config declarations such as user
, group
, path
, env
, and umask
are implemented as wrappers around your provided commands, you must be aware that issues can arise if your commands contain characters such as single quotes that are not properly escaped.
As of SSHKit 1.16, declaring the above five ssh_config declarations will produce an ssh command similar to the following:
cd /home/root && umask 077 && ( export RAILS_ENV="development" ; sudo -u root RAILS_ENV="development" -- sh -c 'sg root -c \"/usr/bin/env echo I love cheese\"' )
One specific known issue due to improperly escaped characters is when providing a user declaration, any single quotes should be escaped with the following string: '\\''
command "echo '\\''I love cheese'\\''"
Alternatively, if you wish to avoid issues with SSHKit command wrapping, you can specify that your commands be executed in raw form, passed directly as written to the specified host.
tmux_command
wraps the input passed to it in single quotes. Therefore any single quotes passed to the tmux_command
should be escaped using '\\''
. This issue can manifest itself as part of the command not being echoed to the tmux pane.
When specifying the env
for running commands, if you place curly braces {}
around the env values, it is required to enclose the arguments in parenthesis ()
, otherwise the following syntax error will result:
syntax error, unexpected ':', expecting '}' (SyntaxError)
Env should be specified as:
env rails_env: :production
or
env ({rails_env: :production})
not as
env {rails_env: :production}
Yes they are. When you define a runbook, a tree data structure is constructed much like an abstract syntax tree. This is important because you do not have to worry about any side-effects such as executing server commands when this data structure is compiled. Once compiled, choosing either the run or view to execute the runbook object determines what behavior is executed at each node.
Runbook is designed to minimize and mitigate issues that arise when running operations in production enviroments. One way this is accomplished is by compiling all statements in the runbook before execution is started. Validations and assertions can be made to reduce the likelihood that a runbook will encounter an error in the middle of an operation. In other words, Runbook provides some guarantees about proper formatting of a runbook before any commands execute that could affect live systems.
Because runbooks are compiled, statements that set values such as ask
, capture
, and capture_all
statements (using the :into
keyword) only expose their values at runtime. This means any references to these methods or variables (specified with :into
) can only happen within ruby_command
blocks which are evaluated at runtime. If an argument to a statement references the values set by these statements, then the statement must be wrapped in a ruby_command
block. See Passing State for specific examples.
When defining and referencing your own functions in a runbook, functions should be wrapped in a module so they can be referenced globally. For example:
module Adder
def self.add(x, y)
x + y
end
end
Runbook.book "Add Two Numbers" do
step "Add numbers" do
ask "X?", into: :x
ask "Y?", into: :y
ruby_command do |rb_cmd, metadata, run|
metadata[:toolbox].output("Result: #{Adder.add(x, y)}")
end
end
end
There are a number of reasons why a command may work directly on your command line, but not when executed using the command
statement. Some possible things to try include:
PATH
environment variable may not be set.sh
) and your shell (usually bash).After checking out the repo, run bin/setup
to install dependencies. Then, run rake spec
to run the tests. You can also run bin/console
for an interactive prompt that will allow you to experiment.
To install this gem onto your local machine, run bundle exec rake install
.
To execute runbook using this repo, run bundle exec exe/runbook exec examples/layout_runbook.rb
.
To release a new version:
version.rb
.CHANGELOG.rb
.gem signin
to ensure you can push the new version to rubygems.orgbundle exec rake release
, which will create a git tag for the version and push git commits and tags.Bug reports and pull requests are welcome on GitHub at https://github.com/braintree/runbook. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the Contributor Covenant code of conduct.
Any feature requests are always welcome and will be considered in accordance with time and need. Additionally, existing feature requests are tracked in TODO.md. If you choose to contribute, your contributions will be greatly appreciated. Please reach out before creating any substantial pull requests. A bit of discussion can save a lot of time and increase the chances that your pull request will be accepted.
The gem is available as open source under the terms of the MIT License.
Everyone interacting in the Runbook project’s codebases, issue trackers, chat rooms and mailing lists is expected to follow the code of conduct.