search

abcxyz / abc

Apache License 2.0
12 stars 3 forks source link

readme

abc

abc is not an official Google product.

Introduction

abc is a command line interface (CLI) to speed up the process of creating new applications. It achieves this by using a templating system that will allow users to interactively fork existing templates, while providing neccessary context, instructions, or requested inputs.

Using this tool will reduce the cognitive load required to set up GitHub actions properly, or follow development best practices, and avoid copy/pasting from various sources to start a new project.

This doc contains a User Guide and a Template Developer Guide.

Command line usage

The abc command has many subcommands, describes below. In abc versions before 0.9, these commands were called abc templates $SUBCOMMAND, but as of 0.9 they are now also available under the shorter form abc $SUBCOMMAND.

For abc render

Usage: abc render [flags] <template_location>

Example: abc render --prompt github.com/abcxyz/gcp-org-terraform-template@latest

The <template_location> parameter is one of these two things:

Flags

Flags for template developers:

Logging

Use the environment variables ABC_LOG_MODE and ABC_LOG_LEVEL to configure logging.

The valid values for ABC_LOG_MODE are:

The valid values for ABC_LOG_LEVEL are debug, info, notice, warning, error, and emergency. The default is warn.

For abc golden-test

The golden-test feature is essentially unit testing for templates. You provide (1) a set of template input values and (2) the expected output directory contents. The test framework verifies that the actual output matches the expected output, using the verify subcommand. Separately, the record subcommand helps with capturing the current template output and saving it as the "expected" output for future test runs. This concept is similar to "snapshot testing" and "rpc replay testing." In addition, the new-test subcommand creates a new golden test to initialize the needed golden test directory structure and test.yaml.

Each test is configured by placing a file named test.yaml in a subdirectory of the template named testdata/golden/<your-test-name>. See below for details on this file.

Usage:

Note: For new-test, the <location> parameter gives the location of the template. For record and verify, <location> parameter gives the location that include one or more templates and abc cli will recursively search for templates and tests under the given <location>.

Examples:

For record and verify subcommand, the <test_name> parameter gives the test names to record or verify, if not specified, all tests will be run against. This flag may be repeated, like --test-name=test1, --test-name=test2, or --test-name=test1,test2.

For new-test subcommand, the <location> parameter gives the location of the template, defaults to the current directory.

For record and verify subcommand, the <location> parameter gives the location that include one or more templates, defaults to the current directory.

For every test case, it is expected that a testdata/golden/<test_name>/test.yaml exists to define template input params. Each "input" in this file must correspond to a template input defined in the template's spec.yaml. Each required input in the template's spec.yaml must have a corresponding input value defined in the test.yaml. Typically, you will use golden-test new-test subcommand to initialize the needed golden test directory structure and test.yaml, but it's also possible to create the desired goldentest directory structure and test.yaml by hand.

Example test.yaml:

api_version: 'cli.abcxyz.dev/v1alpha1'
kind: 'GoldenTest'

inputs:
  - name: 'my-service-account'
    value: 'platform-ops@abcxyz-my-project.iam.gserviceaccount.com'
  - name: 'my-project-number'
    value: '123456789'

The expected/desired test output for each test is stored in testdata/golden/<test_name>/data. Typically, you'll use the golden-test record subcommand to populate this directory, but it's also possible to create the desired output files by hand.

Builtin vars in golden tests

In spec.yaml, there some built-in variables like _git_tag that are populated automatically based on the environment. For unit testing, we need the ability to populate these variables with fixed values so the test output is the same every time.

To support this, the test.yaml file may have a top-level field builtin_vars that sets the value of built-in variables while running the test. For example:

api_version: 'cli.abcxyz.dev/v1beta2'
kind: 'GoldenTest'

inputs:
  - name: 'some-normal-input'
    value: 'some-value'

# For the purposes of this golden test, provide a fake _git_tag value.
builtin_vars:
  - name: '_git_tag'
    value: 'my-cool-tag'

Technicalities:

For abc describe

The describe command downloads the template and prints out its description, and describes the inputs that it accepts.

Usage:

The <template_location> takes the same value as the render command.

Example:

Command:

abc describe github.com/abcxyz/guardian/abc.templates/default-workflows@v0.1.0-alpha12

Output:

Description:  Generate the Guardian workflows for the Google Cloud organization Terraform intrastructure repo.

Input name:   terraform_directory
Description:  A sub-directory for all Terraform files
Default:      .

Input name:   terraform_version
Description:  The terraform version to use with Guardian
Default:      1.5.4

Input name:   guardian_wif_provider
Description:  The Google Cloud workload identity federation provider for Guardian

Input name:   guardian_service_account
Description:  The Google Cloud service account for Guardian
Rule 0:       gcp_matches_service_account(guardian_service_account)

Input name:   guardian_state_bucket
Description:  The Google Cloud storage bucket for Guardian state

User Guide

Start here if you want to install ("render") a template using this CLI tool. "Rendering" a template is when you use the abc CLI to download some template code, do some substitution to replace parts of it with your own values, and write the result to a local directory.

Installation

There are two ways to install:

  1. The most official way:

  2. Alternatively, if you already have a Go programming environment set up, just run go install github.com/abcxyz/abc/cmd/abc@latest.

Tab Autocompletion

Optionally, for tab autocompletion, run:

COMP_INSTALL=1 COMP_YES=1 abc

This will add a complete command to your .bashrc or corresponding file.

Rendering a template

The full user journey looks as follows. For this example, suppose you want to create a "hello world" Go web service.

  1. Set up a directory that will receive the rendered template output, and cd to it.

    • Option A: for local experimentation, you can just write into any directory, for example: mkdir ~/template_experiment && cd ~/template_experiment
    • Option B: to create a real service that you'll share with others:
      • create a new git repo that will contain your new service
      • clone it onto your machine (git clone ...)
      • cd into the git directory you just cloned into
      • Create a branch (git checkout -b template_render)
    • Option C: if you know what you're doing, you can create a local repo using git init and worry later about connecting it to an upstream repo.

  2. Find the template to install. We assume that you already know the URL of a template that you want to install by reading docs or through word-of-mouth. There is a best-effort list of known templates in template-index.md. For this example, suppose we're installing the "hello jupiter" example from the abc repo.

  3. Run the render command:

    $ abc render \
 github.com/abcxyz/abc/examples/templates/render/hello_jupiter@latest

    This command will output files in your curent directory that are the result of executing the template.

    • (Optional) examine the resulting files and try running the code:

      $ ls
main.go
$ go run main.go
Hello, jupiter!

  4. Git commit, push your branch, create a PR, get it reviewed, and submit it.

    $ git add -A
$ git commit -am 'Initial output of template rendering'
$ git push origin template_render:$USER/template_render

# Assuming you're using GitHub, now go create a PR.

Authentication errors

If abc asks you for a username and password, that probably means that the template you're rendering is in a private git repository, and HTTPS authentication didn't work. You may want to try cloning over SSH instead. To use SSH, you can add --git-protocol=ssh to your command line or set the environment variable ABC_GIT_PROTOCOL=ssh.

Template developer guide

This section explains how you can create a template for others to install (aka "render").

Concepts

A template is installed from a location that you provide. These locations may be either a GitHub repository or a local directory. If you install a template from GitHub, it will be downloaded into a temp directory by abc.

In essence, a template is a directory containing a "spec file", named spec.yaml (example), and other files such as source code and config files.

Model of operation

Template rendering has a few phases:

Normally, the template and scratch directories are deleted when rendering completes. For debugging, you can provide the flag --keep-temp-dirs to retain them for inspection.

The spec file

The spec file, named spec.yaml describes the template, including:

Here is an example spec file. It has a single templated file, main.go, and during template rendering all instances of the word world are replaced by a user-provided string. Thus "hello, world" is transformed into "hello, $whatever" in main.go.

api_version: 'cli.abcxyz.dev/v1alpha1'
kind: 'Template'

desc:
  'An example template that changes a "hello world" program to a "hello whoever"
  program'
inputs:
  - name: 'whomever'
    desc: 'The name of the person or thing to say hello to'
steps:
  - desc: 'Include some files and directories'
    action: 'include'
    params:
      paths: ['main.go']
  - desc: 'Replace "world" with user-provided input'
    action: 'string_replace'
    params:
      paths: ['main.go']
      replacements:
        - to_replace: 'world'
          with: '{{.whomever}}'

List of api_versions

The api_version field controls the interpretation of the YAML file. Some features are only available in more recent versions.

The currently valid versions are:

api_version Supported in abc CLI versions Notes
cli.abcxyz.dev/v1alpha1 0.0.0 and up Initial version
cli.abcxyz.dev/v1beta1 0.2.0 and up Adds support for an if predicate on each step in spec.yaml
cli.abcxyz.dev/v1beta2 0.4.0 and up Adds:
- the top-level ignore field in spec.yaml
- Path globs
cli.abcxyz.dev/v1beta3 0.5.0 Adds:
- _git_* builtin variables
cli.abcxyz.dev/v1beta4 0.6.0 Adds:
- independent rules
cli.abcxyz.dev/v1beta5 0.6.0 Same as v1beta4 for complex reasons
cli.abcxyz.dev/v1beta6 0.7.0 Adds: the _now_ms variable and formatTime function in Go-templates

Template inputs

Typically the CLI user will supply certain values as --input=inputname=value which will be used by the spec file (such as whomever in the preceding example). Alternatively, the user can use --prompt rather than --input to enter values interactively.

A template may not need any inputs, in which case the inputs top-level field in the spec.yaml can be omitted.

Each input in the inputs list has these fields:

The input validation rules may be skipped with the --skip-input-validation flag, documented above.

An example input without a default:

inputs:
  - name: 'output_filename'
    desc: 'The name of the file to create'

An example input with a default:

inputs:
  - name: 'output_filename'
    description: 'The name of the file to create'
    default: 'out.txt'

An example of parsing an input as an integer:

inputs:
  - name: 'disk_size_bytes'
    rules:
      - rule: 'int(disk_size_bytes)' # Will fail if disk_size_bytes (which is a string) can't be parsed as int
        message: 'Must be an integer'

An example input with a validation rule:

inputs:
  - name: 'project_id_to_use'
    rules:
      - rule: 'gcp_matches_project_id(project_id_to_use)'
        message: 'Must be a GCP project ID'

An example of validating multiple inputs together:

inputs:
  - name: 'min_size_bytes'
  - name: 'max_size_bytes'
    rules:
      - rule: 'int(min_size_bytes) <= int(max_size_bytes)'
        message: "the max can't be less than the min"
Top-level rules

Most of the time, validation rules will be part of an inputs declaration as described above. But it's also possible to create validation rules that are independent of any input and go at the topmost scope of the spec file.

To use this feature, your spec.yaml must declare api_version: cli.abcxyz.dev/v1beta4 or greater.

Example:

apiVersion: "cli.abcxyz.dev/v1beta4"
kind: "Template"
desc: "An example of using independent rules"
rules:
  - rule: '_git_sha != ""'
    message: "this template must be installed from a git repo"

Built-in template variables

Besides the template inputs described above, there are built-in template variables that are automatically provided. These can be referenced in a go-template context as {{._my_variable}} and in a CEL context as my_variable. The built-in variables are:

Templating

Most fields in the spec file can use template expressions that reference the input values. In the above example, the replacement value of {{.whomever}} means "the user-provided input value named whomever." This uses the text/template templating language that is part of the Go standard library.

Steps and actions

Each step of the spec file performs a single action. A single step consists of:

Example:

desc: 'An optional human-readable description of what this step is for'
action: 'action-name' # One of 'include', 'print', 'append', 'string_replace', 'regex_replace', `regex_name_lookup`, `go_template`, `for_each`
if: 'bool(my_input) || int(my_other_input) > 42' # Optional CEL expression
params:
  foo: bar # The params differ depending on the action

Action: include

Copies files or directories from the template directory to the scratch directory. It's similar to the COPY command in a Dockerfile.

Params:

Examples:

Action: print

Prints a message to standard output. This can be used to suggest actions to the user.

Params:

Example:

- action: 'print'
  params:
    message:
      'Please go to the GCP console for project {{.project_id}} and click the
      thing'

Action: append

Appends a string on the end of a given file. File must already exist. If no newline at end of with parameter, one will be added unless skip_ensure_newline is set to true.

If you need to remove an existing trailing newline before appending, use regex_replace instead.

Params:

Example:

- action: 'append'
  params:
    paths: ['foo.html', 'web/']
    with: '</html>\n'
    skip_ensure_newline: false

Action: string_replace

Within a given list of files and/or directories, replaces all occurrences of a given string with a given replacement string.

Params:

Example:

- action: 'string_replace'
  params:
    paths: ['main.go']
    replacements:
      - to_replace: 'Alice'
        with: '{{.sender_name}}'
      - to_replace: 'Bob'
        with: '{{.receiver_name}}'

Action: regex_replace

Within a given list of files and/or directories, replace a regular expression (or a subgroup thereof) with a given string.

Params:

Examples:

Action: regex_name_lookup

regex_name_lookup is similar to regex_replace, but simpler to use, at the cost of generality. It matches a regular expression and replaces each named subgroup with the input variable whose name matches the subgroup name.

Params:

Example: replace all appearances of template_me with the input variable named myinput:

- action: 'regex_name_lookup'
  params:
    paths: ['main.go']
    replacements:
      - regex: '(?P<myinput>template_me)'

Action: go_template

Executes a file as a Go template, replacing the file with the template output.

Params:

Example:

Suppose you have a file named hello.html that looks like this, with a {{.foo}} template expression:

<html><body>
{{ if .friendly }}
Hello, {{.person_name}}!
{{ else }}
Go jump in a lake, {{.person_name}}.
{{ end }}
</body></html>

This action will replace {{.person_name}} (and all other template expressions) with the corresponding inputs:

- action: 'go_template'
  params:
    paths: ['hello.html']

Action: for_each

The for_each action lets you execute a sequence of steps repeatedly for each element of a list. For example, you might want your template to create several copies of a given file, one per application environment (e.g. production, staging).

There are two variants of for_each. One variant accepts a hardcoded YAML list of values to iterate over in the values field. The other variant accepts a CEL expression in the values_from field that outputs a list of strings.

Variant 1 example: hardcoded list of YAML values:

- desc: 'Iterate over each (hard-coded) environment'
  action: 'for_each'
  params:
    iterator:
      key: 'environment'
      values: ['production', 'dev']
    steps:
      - desc: 'Do some action for each environment'
        action: 'print'
        params:
          message: 'Now processing environment named {{.environment}}'

Variant 2 example: a CEL expression that produces the list to iterate over:

- desc: 'Iterate over each environment, produced by CEL as a list'
  action: 'for_each'
  params:
    iterator:
      key: 'environment'
      values_from: 'comma_separated_environments.split(",")'
    steps:
      - desc: 'Do some action for each environment'
        action: 'print'
        params:
          message: 'Now processing environment named {{.environment}}'

Params:

Ignore (Optional)

This ignore feature is similiar to skip in include action, the difference here is that ignore is global and it applies to every include action.

We use filepath Match to match the file and directory paths that should be ignored if included/copied to destination directory. In addition, we also match file and directory names using the same accepted patterns.

This section is optional, if not provided, a default ignore list is used: .DS_Store, .bin, and .ssh, meaning all file and directory matching these names will be ignored. To set your custom ignore list, please check accepted patterns here. Note: a leading slash in a pattern here means the source of the included paths.

Example:

ignore:
  # Ignore `.ssh` under root, root is the template dir or destination dir if the
  # included paths are from destination.
  - '/.ssh'
  # Ignore all txt files with name `tmp.txt` recursively (under root and its
  # sub-directories).
  - 'tmp.txt'
  # Ignore all txt files in the sub-directories with folder depth of 2.
  - '*/*.txt'
  # Ignore all cfg files recursively.
  - '*.cfg'
steps:
  - desc: 'Include some files and directories'
    action: 'include'
    params:
      paths: ['.ssh', 'src_dir']
  - desc: 'Include some files and directories from destination'
    action: 'include'
    params:
      paths: ['dest_dir']
      from: 'destination'

Post-rendering validation test (golden test)

We use post-rendering validation tests to record (capture the anticipated outcome akin to expected output in unit test) and subsequently verify template rendering results.

To add golden tests to your template, all you need is to create a testdata/golden folder under your template, and a testdata/golden/<test_name>/test.yaml for each of your tests to define test metadata and input parameters.

The test.yaml for a post-rendering validation test may look like,

api_version: 'cli.abcxyz.dev/v1alpha1'
kind: 'GoldenTest'

inputs:
  - name: 'input_a'
    value: 'a'
  - name: 'input_b'
    value: 'b'

Then you can use abc golden-test to record (capture the anticipated outcome akin to expected output in unit test)or verify the tests.

Using CEL

We use the CEL language to allow template authors to embed scripts in the spec file in certain places. The places you can use CEL are:

CEL, the Common Expression Language), is a non-Turing complete language that's designed to be easily embedded in programs. "Expression" means "a computation that produces a value", like 1+1 or ["shark"+"nado", "croco"+"gator"].

The CEL expressions you write in your spec file will have access to the template inputs, as in this example:

For example:

- desc: 'Iterate over each environment, produced by CEL as a list'
  action: 'for_each'
  params:
    iterator:
      key: 'env'
      values: 'input.comma_separated_environments.split(",")'

The above example also shows the split function, which is not part of the core CEL language. It's a "custom function" that we added to CEL to support a common need for templates (see below).

Custom functions reference

These are the functions that we added that are not normally part of CEL.

Update Checks

abcxyz/abc-updater is run once a day to check for newer versions of abc, results are printed to stderr if an update is available. This check can be disabled by setting the environment variable export ABC_IGNORE_VERSIONS=ALL. Notifications can be disabled for specific versions with a list of versions and constraints export ABC_IGNORE_VERSIONS=<2.0.0,3.5.0.

This check is not done on non-release builds, as they don't have canonical version to check against.

Metrics

We collect non-identifiable usage metrics using abcxyz/abc-updater. You can opt out of these metrics by setting the environment variable ABC_NO_METRICS to TRUE in your shell.

Currently, data is collected on:

Along with each metric, the following metadata is recorded:

Metrics data is retained for 24 months.