Add make integration_test_run task and basic kitchen test

[jeffmccune]

This patch adds a make integration_test_run task reflecting the CI integration-test job. Both the make target and the CI job execute test/ci_integration.sh inside of a Docker container. The Make task and CI Job both share the responsibility of fetching the container, setting up the context for the tests (Service Account credentials, GCP project to run in, etc...), and then executing the container and handing off to ci_integration.sh.

The simple_example kitchen test verifies the default behavior of the startup script. It loads a basic startup-script-custom script and logs a message.

Manage GCP fixture resources like CI does

Without this patch there isn't a clear way to run integration tests locally on a workstation reflecting how the CI system runs integration tests. This is a problem because it creates drift and divergence from how CI operates, creating friction in the development process.

This patch addresses the problem by allowing GCP resources to be created and managed just like they are with the CI system. A project and service account are created and provided as inputs to the integration test suite.

GOOGLE_CREDENTIALS

• Remove config.sh, use GOOGLE_CREDENTIALS instead
• Write credentials to a tempfile and export as a var inside the docker container so inspec can call gcloud from examples without error. This also allows easy cleanup and containment of the credentials file with a trap handler.

Example Run

CI isn't in place yet, but this is the expected output:

$ time make integration_test_run
docker pull gcr.io/cloud-foundation-cicd/cft/kitchen-terraform:0.11.10_216.0.0_1.19.1_0.1.10
0.11.10_216.0.0_1.19.1_0.1.10: Pulling from cloud-foundation-cicd/cft/kitchen-terraform
Digest: sha256:6a24707d0396f7898a24dbef28315401cb8459a68bd4d4d1d8db1688e8c27502
Status: Image is up to date for gcr.io/cloud-foundation-cicd/cft/kitchen-terraform:0.11.10_216.0.0_1.19.1_0.1.10
docker run --rm -it \
        --volume /Users/jmccune/projects/cft/terraform-google-startup-scripts:/terraform-google-startup-scripts \
        --workdir /terraform-google-startup-scripts \
        --env SERVICE_ACCOUNT_JSON \
        --env PROJECT_ID \
        gcr.io/cloud-foundation-cicd/cft/kitchen-terraform:0.11.10_216.0.0_1.19.1_0.1.10 \
        /bin/bash test/ci_integration.sh
+ kitchen create
-----> Starting Kitchen (v1.23.5)
$$$$$$ Running command `terraform version` in directory /terraform-google-startup-scripts
       Terraform v0.11.10

       Your version of Terraform is out of date! The latest version
       is 0.11.11. You can update by downloading from www.terraform.io/downloads.html
$$$$$$ Terraform v0.11.10 is supported
-----> Creating <simple-example-local>...
$$$$$$ Running command `terraform init -input=false -lock=true -lock-timeout=0s  -upgrade -force-copy -backend=true  -get=true -get-plugins=true  -verify-plugins=true` in directory /terraform-google-startup-scripts/test/fixtures/simple_example
       Upgrading modules...
       - module.example
         Updating source "../../../examples/simple_example"
       - module.example.startup-scripts
         Updating source "../../"

       Initializing provider plugins...
       - Checking for available provider plugins on https://releases.hashicorp.com...
       - Downloading plugin for provider "google" (1.20.0)...

       Terraform has been successfully initialized!
$$$$$$ Running command `terraform workspace select kitchen-terraform-simple-example-local` in directory /terraform-google-startup-scripts/test/fixtures/simple_example

       Workspace "kitchen-terraform-simple-example-local" doesn't exist.

       You can create this workspace with the "new" subcommand.
$$$$$$ Running command `terraform workspace new kitchen-terraform-simple-example-local` in directory /terraform-google-startup-scripts/test/fixtures/simple_example
       Created and switched to workspace "kitchen-terraform-simple-example-local"!

       You're now on a new, empty workspace. Workspaces isolate their state,
       so if you run "terraform plan" Terraform will not see any existing state
       for this configuration.
       Finished creating <simple-example-local> (2m3.26s).
-----> Kitchen is finished. (2m5.17s)
+ kitchen converge
-----> Starting Kitchen (v1.23.5)
$$$$$$ Running command `terraform version` in directory /terraform-google-startup-scripts
       Terraform v0.11.10

       Your version of Terraform is out of date! The latest version
       is 0.11.11. You can update by downloading from www.terraform.io/downloads.html
$$$$$$ Terraform v0.11.10 is supported
-----> Converging <simple-example-local>...
$$$$$$ Running command `terraform workspace select kitchen-terraform-simple-example-local` in directory /terraform-google-startup-scripts/test/fixtures/simple_example
$$$$$$ Running command `terraform get -update` in directory /terraform-google-startup-scripts/test/fixtures/simple_example
       - module.example
         Updating source "../../../examples/simple_example"
       - module.example.startup-scripts
         Updating source "../../"
$$$$$$ Running command `terraform validate -check-variables=true   ` in directory /terraform-google-startup-scripts/test/fixtures/simple_example
$$$$$$ Running command `terraform apply -lock=true -lock-timeout=0s -input=false -auto-approve=true  -parallelism=10 -refresh=true  ` in directory /terraform-google-startup-scripts/test/fixtures/simple_example
       data.google_compute_image.os: Refreshing state...
       module.example.google_compute_instance.example: Creating...
         boot_disk.#:                                         "" => "1"
         boot_disk.0.auto_delete:                             "" => "true"
         boot_disk.0.device_name:                             "" => "<computed>"
         boot_disk.0.disk_encryption_key_sha256:              "" => "<computed>"
         boot_disk.0.initialize_params.#:                     "" => "1"
         boot_disk.0.initialize_params.0.image:               "" => "https://www.googleapis.com/compute/v1/projects/centos-cloud/global/images/centos-7-v20190116"
         boot_disk.0.initialize_params.0.size:                "" => "<computed>"
         boot_disk.0.initialize_params.0.type:                "" => "pd-standard"
         can_ip_forward:                                      "" => "false"
         cpu_platform:                                        "" => "<computed>"
         create_timeout:                                      "" => "4"
         deletion_protection:                                 "" => "false"
         description:                                         "" => "Startup Scripts Example"
         guest_accelerator.#:                                 "" => "<computed>"
         instance_id:                                         "" => "<computed>"
         label_fingerprint:                                   "" => "<computed>"
         machine_type:                                        "" => "f1-micro"
         metadata.%:                                          "" => "2"
         metadata.startup-script:                             "" => "#! /bin/bash\n# Copyright 2018 Google LLC\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#      http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\n# Standard library of functions useful for startup scripts.\n\n# Pending UX behaviors\n# TODO(jmccune): call mandatory_argument() before E_MISSING_MANDATORY_ARG\n# TODO(jmccune): add -c <name> for alternate startup-script-custom name\n# Pending initialization functions:\n# TODO(jmccune): gsutil initialization w/ crcmod\n# Pending operational functions:\n# TODO(jmccune): get_from_bucket()\n# TODO(jmccune): setup_init_script()\n# TODO(jmccune): setup_sudoers()\n\n# These are outside init_global_vars so logging functions work with the most\n# basic case of `source startup-script-stdlib.sh`\nreadonly SYSLOG_DEBUG_PRIORITY=\"${SYSLOG_DEBUG_PRIORITY:-syslog.debug}\"\nreadonly SYSLOG_INFO_PRIORITY=\"${SYSLOG_INFO_PRIORITY:-syslog.info}\"\nreadonly SYSLOG_ERROR_PRIORITY=\"${SYSLOG_ERROR_PRIORITY:-syslog.error}\"\n# Global counter of how many times stdlib::init() has been called.\nSTARTUP_SCRIPT_STDLIB_INITIALIZED=0\n\n# Error codes\nreadonly E_RUN_OR_DIE=5\nreadonly E_MISSING_MANDATORY_ARG=9\nreadonly E_UNKNOWN_ARG=10\n\nstdlib::debug() {\n  [[ -z \"${DEBUG:-}\" ]] && return 0\n  local ds msg\n  msg=\"$*\"\n  logger -p \"${SYSLOG_DEBUG_PRIORITY}\" -t \"${PROG}[$$]\" -- \"${msg}\"\n  [[ -n \"${QUIET:-}\" ]] && return 0\n  ds=\"$(date +\"${DATE_FMT}\") \"\n  echo -e \"${BLUE}${ds}Debug [$$]: ${msg}${NC}\" >&2\n}\n\nstdlib::info() {\n  local ds msg\n  msg=\"$*\"\n  logger -p \"${SYSLOG_INFO_PRIORITY}\" -t \"${PROG}[$$]\" -- \"${msg}\"\n  [[ -n \"${QUIET:-}\" ]] && return 0\n  ds=\"$(date +\"${DATE_FMT}\") \"\n  echo -e \"${GREEN}${ds}Info [$$]: ${msg}${NC}\" >&2\n}\n\nstdlib::error() {\n  local ds msg\n  msg=\"$*\"\n  ds=\"$(date +\"${DATE_FMT}\") \"\n  logger -p \"${SYSLOG_ERROR_PRIORITY}\" -t \"${PROG}[$$]\" -- \"${msg}\"\n  echo -e \"${RED}${ds}Error [$$]: ${msg}${NC}\" >&2\n}\n\n# The main initialization function of this library.  This should be kept to the\n# minimum amount of work required for all functions to operate cleanly.\nstdlib::init() {\n  if [[ ${STARTUP_SCRIPT_STDLIB_INITIALIZED} -gt 0 ]]; then\n    stdlib::info 'stdlib::init()'\" already initialized, no action taken.\"\n    return 0\n  fi\n  ((STARTUP_SCRIPT_STDLIB_INITIALIZED++)) || true\n  stdlib::init_global_vars\n  stdlib::init_directories\n  stdlib::debug \"stdlib::init(): startup-script-stdlib.sh initialized and ready\"\n}\n\n# Transfer control to startup-startup-script-custom.  The script is sourced to\n# ensure all functions are exposed and the trap handlers configured here are\n# fired on exit.  A local file path or http URL are both supported.\nstdlib::run_startup_script_custom() {\n  local script_file key\n  local exit_code\n  # shellcheck disable=SC2119\n  script_file=\"$(stdlib::mktemp)\"\n  key=\"instance/attributes/startup-script-custom\"\n\n  if ! stdlib::metadata_get -k \"${key}\" -o \"${script_file}\"; then\n    stdlib::error \"Could not fetch custom startup script.\" \\\n      \"Make sure ${key} exists.\"\n    return 1\n  fi\n\n  stdlib::debug \"=== BEGIN ${key} ===\"\n  # shellcheck source=/dev/null\n  source \"${script_file}\"\n  exit_code=$?\n  stdlib::debug \"=== END ${key} exit_code=${exit_code} ===\"\n  return $exit_code\n}\n\n# Initialize global variables.\nstdlib::init_global_vars() {\n  # The program name, used for logging.\n  readonly PROG=\"${PROG:-startup-script-stdlib}\"\n  # Date format used for stderr logging.  Passed to date + command.\n  readonly DATE_FMT=\"${DATE_FMT:-\"%a %b %d %H:%M:%S %z %Y\"}\"\n  # var directory\n  readonly VARDIR=\"${VARDIR:-/var/lib/startup}\"\n  # Override this with file://localhost/tmp/foo/bar in spec test context\n  readonly METADATA_BASE=\"${METADATA_BASE:-http://metadata.google.internal}\"\n\n  # Color variables\n  if [[ -n \"${COLOR:-}\" ]]; then\n    readonly NC='\\033[0m'        # no color\n    readonly RED='\\033[0;31m'    # error\n    readonly GREEN='\\033[0;32m'  # info\n    readonly BLUE='\\033[0;34m'   # debug\n  else\n    readonly NC=''\n    readonly RED=''\n    readonly GREEN=''\n    readonly BLUE=''\n  fi\n\n  return 0\n}\n\nstdlib::init_directories() {\n  if ! [[ -e \"${VARDIR}\" ]]; then\n    install -d -m 0755 -o 0 -g 0 \"${VARDIR}\"\n  fi\n}\n\n##\n# Get a metadata key.  When used without -o, this function is guaranteed to\n# produce no output on STDOUT other than the retrieved value.  This is intended\n# to support the use case of\n# FOO=\"$(stdlib::metadata_get -k instance/attributes/foo)\"\n#\n# If the requested key does not exist, the error code will be 22 and zero bytes\n# written to STDOUT.\nstdlib::metadata_get() {\n  local OPTIND opt key outfile\n  local metadata=\"${METADATA_BASE%/}/computeMetadata/v1\"\n  local exit_code\n  while getopts \":k:o:\" opt; do\n    case \"${opt}\" in\n    k) key=\"${OPTARG}\" ;;\n    o) outfile=\"${OPTARG}\" ;;\n    :)\n      stdlib::error \"Invalid option: -${OPTARG} requires an argument\"\n      stdlib::metadata_get_usage\n      return \"${E_MISSING_MANDATORY_ARG}\"\n      ;;\n    *)\n      stdlib::error \"Unknown option: -${opt}\"\n      stdlib::metadata_get_usage\n      return \"${E_UNKNOWN_ARG}\"\n      ;;\n    esac\n  done\n  local url=\"${metadata}/${key#/}\"\n\n  stdlib::debug \"Getting metadata resource url=${url}\"\n  if [[ -z \"${outfile:-}\" ]]; then\n    curl --location --silent --connect-timeout 1 --fail \\\n      -H 'Metadata-Flavor: Google' \"$url\" 2>/dev/null\n    exit_code=$?\n  else\n    stdlib::cmd curl --location \\\n      --silent \\\n      --connect-timeout 1 \\\n      --fail \\\n      --output \"${outfile}\" \\\n      -H 'Metadata-Flavor: Google' \\\n      \"$url\"\n    exit_code=$?\n  fi\n  case \"${exit_code}\" in\n    22 | 37)\n      stdlib::debug \"curl exit_code=${exit_code} for url=${url}\" \\\n        \"(Does not exist)\"\n      ;;\n  esac\n  return \"${exit_code}\"\n}\n\nstdlib::metadata_get_usage() {\n  stdlib::info 'Usage: stdlib::metadata_get -k <key>'\n  stdlib::info 'For example: stdlib::metadata_get -k instance/attributes/startup-config'\n}\n\n# Load configuration values in the spirit of /etc/sysconfig defaults, but from\n# metadata instead of the filesystem.\nstdlib::load_config_values() {\n  local config_file\n  local key=\"instance/attributes/startup-script-config\"\n  # shellcheck disable=SC2119\n  config_file=\"$(stdlib::mktemp)\"\n  stdlib::metadata_get -k \"${key}\" -o \"${config_file}\"\n  local status=$?\n  case \"$status\" in\n    0)\n      stdlib::debug \"SUCCESS: Configuration data sourced from $key\"\n      ;;\n    22 | 37)\n      stdlib::debug \"no configuration data loaded from $key\"\n      ;;\n    *)\n      stdlib::error \"metadata_get -k $key returned unknown status=${status}\"\n      ;;\n  esac\n  # shellcheck source=/dev/null\n  source \"${config_file}\"\n}\n\n# Run a command logging the entry and exit.  Intended for system level commands\n# and operational debugging.  Not intended for use with redirection.  This is\n# not named run() because bats uses a run() function.\nstdlib::cmd() {\n  local exit_code argv=(\"$@\")\n  stdlib::debug \"BEGIN: stdlib::cmd() command=[${argv[*]}]\"\n  \"${argv[@]}\"\n  exit_code=$?\n  stdlib::debug \"END: stdlib::cmd() command=[${argv[*]}] exit_code=${exit_code}\"\n  return $exit_code\n}\n\n# Run a command successfully or exit the program with an error.\nstdlib::run_or_die() {\n  if ! stdlib::cmd \"$@\"; then\n    stdlib::error \"stdlib::run_or_die(): exiting with exit code ${E_RUN_OR_DIE}.\"\n    exit \"${E_RUN_OR_DIE}\"\n  fi\n}\n\n# Intended to take advantage of automatic cleanup of startup script library\n# temporary files without exporting a modified TMPDIR to child processes, which\n# would cause the children to have their TMPDIR deleted out from under them.\n# shellcheck disable=SC2120\nstdlib::mktemp() {\n  TMPDIR=\"${DELETE_AT_EXIT:-${TMPDIR}}\" mktemp \"$@\"\n}\n\nstdlib::main() {\n  DELETE_AT_EXIT=\"$(mktemp -d)\"\n  readonly DELETE_AT_EXIT\n\n  # Initialize state required by other functions, e.g. debug()\n  stdlib::init\n  stdlib::debug \"Loaded startup-script-stdlib as an executable.\"\n\n  stdlib::load_config_values\n\n  stdlib::run_startup_script_custom\n}\n\n# if script is being executed and not sourced.\nif [[ \"${BASH_SOURCE[0]}\" == \"${0}\" ]]; then\n  stdlib::finish() {\n    [[ -d \"${DELETE_AT_EXIT:-}\" ]] && rm -rf \"${DELETE_AT_EXIT}\"\n  }\n  trap stdlib::finish EXIT\n\n  stdlib::main \"$@\"\nfi\n"
         metadata.startup-script-custom:                      "" => "#! /bin/bash\nURL=${URL:-\"http://ifconfig.co/json\"}\nstdlib::info \"Fetching ${URL}\"\nstdlib::cmd curl --silent \"${URL}\"\necho\n"
         metadata_fingerprint:                                "" => "<computed>"
         name:                                                "" => "startup-scripts-example1"
         network_interface.#:                                 "" => "1"
         network_interface.0.access_config.#:                 "" => "1"
         network_interface.0.access_config.0.assigned_nat_ip: "" => "<computed>"
         network_interface.0.access_config.0.nat_ip:          "" => "<computed>"
         network_interface.0.access_config.0.network_tier:    "" => "<computed>"
         network_interface.0.address:                         "" => "<computed>"
         network_interface.0.name:                            "" => "<computed>"
         network_interface.0.network:                         "" => "default"
         network_interface.0.network_ip:                      "" => "<computed>"
         network_interface.0.subnetwork_project:              "" => "<computed>"
         project:                                             "" => "<computed>"
         scheduling.#:                                        "" => "1"
         scheduling.0.automatic_restart:                      "" => "true"
         scheduling.0.on_host_maintenance:                    "" => "MIGRATE"
         scheduling.0.preemptible:                            "" => "false"
         self_link:                                           "" => "<computed>"
         tags_fingerprint:                                    "" => "<computed>"
         zone:                                                "" => "<computed>"
       module.example.google_compute_instance.example: Still creating... (10s elapsed)
       module.example.google_compute_instance.example: Creation complete after 13s (ID: startup-scripts-example1)

       Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

       Outputs:

       project_id = startup-scripts
       region = us-east4
       Finished converging <simple-example-local> (0m15.59s).
-----> Kitchen is finished. (0m17.07s)
+ kitchen verify
-----> Starting Kitchen (v1.23.5)
$$$$$$ Running command `terraform version` in directory /terraform-google-startup-scripts
       Terraform v0.11.10

       Your version of Terraform is out of date! The latest version
       is 0.11.11. You can update by downloading from www.terraform.io/downloads.html
$$$$$$ Terraform v0.11.10 is supported
-----> Setting up <simple-example-local>...
       Finished setting up <simple-example-local> (0m0.00s).
-----> Verifying <simple-example-local>...
$$$$$$ Running command `terraform output -json` in directory /terraform-google-startup-scripts/test/fixtures/simple_example
       {
           "project_id": {
        "sensitive": false,
        "type": "string",
        "value": "startup-scripts"
           },
           "region": {
        "sensitive": false,
        "type": "string",
        "value": "us-east4"
           }
       }
Verifying system

Profile: simple_example
Version: (not specified)
Target:  local://

  ✔  simple startup-script-custom: With the simple example of startup-script-custom calling stdlib::info and stdlib::cmd
     ✔  Command: `gcloud compute instances list --project startup-scripts` exit_status should equal 0
     ✔  Command: `gcloud compute instances list --project startup-scripts` stderr should eq ""
     ✔  Command: `gcloud compute instances list --project startup-scripts` stdout should match /startup-scripts-example.*RUNNING/

Profile Summary: 1 successful control, 0 control failures, 0 controls skipped
Test Summary: 3 successful, 0 failures, 0 skipped
       Finished verifying <simple-example-local> (0m2.15s).
-----> Kitchen is finished. (0m3.67s)
+ finish
+ echo 'BEGIN: finish() trap handler'
BEGIN: finish() trap handler
+ kitchen destroy
-----> Starting Kitchen (v1.23.5)
$$$$$$ Running command `terraform version` in directory /terraform-google-startup-scripts
       Terraform v0.11.10

       Your version of Terraform is out of date! The latest version
       is 0.11.11. You can update by downloading from www.terraform.io/downloads.html
$$$$$$ Terraform v0.11.10 is supported
-----> Destroying <simple-example-local>...
$$$$$$ Running command `terraform init -input=false -lock=true -lock-timeout=0s  -force-copy -backend=true  -get=true -get-plugins=true  -verify-plugins=true` in directory /terraform-google-startup-scripts/test/fixtures/simple_example
       Initializing modules...
       - module.example
       - module.example.startup-scripts

       Initializing provider plugins...

       Terraform has been successfully initialized!
$$$$$$ Running command `terraform workspace select kitchen-terraform-simple-example-local` in directory /terraform-google-startup-scripts/test/fixtures/simple_example
$$$$$$ Running command `terraform destroy -auto-approve -lock=true -lock-timeout=0s -input=false  -parallelism=10 -refresh=true  ` in directory /terraform-google-startup-scripts/test/fixtures/simple_example
       data.google_compute_image.os: Refreshing state...
       google_compute_instance.example: Refreshing state... (ID: startup-scripts-example1)
       module.example.google_compute_instance.example: Destroying... (ID: startup-scripts-example1)
       module.example.google_compute_instance.example: Still destroying... (ID: startup-scripts-example1, 10s elapsed)
       module.example.google_compute_instance.example: Still destroying... (ID: startup-scripts-example1, 20s elapsed)
       module.example.google_compute_instance.example: Destruction complete after 26s

       Destroy complete! Resources: 1 destroyed.
$$$$$$ Running command `terraform workspace select default` in directory /terraform-google-startup-scripts/test/fixtures/simple_example
       Switched to workspace "default".
$$$$$$ Running command `terraform workspace delete kitchen-terraform-simple-example-local` in directory /terraform-google-startup-scripts/test/fixtures/simple_example
       Deleted workspace "kitchen-terraform-simple-example-local"!
       Finished destroying <simple-example-local> (0m28.63s).
-----> Kitchen is finished. (0m30.18s)
+ [[ -d /tmp/tmp.iaKPlg ]]
+ rm -rf /tmp/tmp.iaKPlg
+ echo 'END: finish() trap handler'
END: finish() trap handler

[jeffmccune]

I am genuinely surprised CI passed on the first try.

[jeffmccune]

@adrienthebo Assuming CI passes, this should be good to go for another review.