Common developer-tools container with credentials handling (DevEx 2.0)

[jeffmccune]

This ticket captures discussions and decisions regarding credential handling when running automated tests for CFT Terraform modules. The scope of this design is local test development on a developer's workstation and also in CI.

Containers

For both local development and CI, a single developer-tools container is used. This container includes all necessary dependencies to run the lint, unit and integration tests for a CFT terraform module.

Credentials

CFT terraform modules commonly use the following tools which need credentials to be initialized within the context of the running container. Each tool has a slightly different way of handling credentials:

1. terraform
2. inspec
3. gcloud
4. gsutil

If additional tools a required as dependencies:

1. Install the tool into the developer-tools image via the Dockerfile
2. If the new tool does not respect one of the existing credentials mechanisms (e.g. GOOGLE_CREDENTIALS), then update the init_credentials function inside of /usr/local/bin/task_helper_functions.sh to initialize the credentials from SERVICE_ACCOUNT_JSON for the new tool.
3. Build and release a new version of the developer-tools image
4. Update CI pipelines to use the new version as necessary.

Design Decision

Credentials are to be passed into the container using a single environment variable, SERVICE_ACCOUNT_JSON. This variable name is chosen for backwards compatibility with existing CI jobs and to avoid a naming conflict with GOOGLE_CREDENTIALS. The value of SERVICE_ACCOUNT_JSON must be the actual JSON string content of a service account JSON key. For example SERVICE_ACCOUNT_JSON="$(< ~/.credentials/jmccune-devtools-0abcde01234.json The value must not be a filesystem path.

Filesystem Paths

Generally, speaking filesystems paths should be avoided when loading SA credentials. The intent is to rely on automatic loading of credentials when possible so that jobs may be deployed with service accounts managed by the platform. For example, consider a VM running in GCE bound to a service account. This VM may be used to run the integration tests without transferring a private key, increasing security.

Credentials Initialization

The developer tools image initializes credentials from the SERVICE_ACCOUNT_JSON input. This initialization is automatically handled via ~root/.bashrc for interactive shells. For non-interactive shells, e.g. when running a command, the initialization does not happen automatically. The caller is responsible for credentials initialization in this case, e.g. the Make task or the CI pipeline definition.

The expected workflow for a developer working on their local workstation is:

1. make docker_run
2. Presented with an interactive bash shell with credentials initialized already.

Credentials initialization may require writing the key to the filesystem. In this situation, a secure temporary file should be used and should be automatically cleaned up when the running container reaches end of life, or sooner depending on the use case.

Note

Example Implementation

An example implementation is provided in #250 and the local development workflow looks like:

Enter the container:

docker run --rm -e SERVICE_ACCOUNT_JSON -it \
  -v `pwd`:/workspace \
  cft/developer-tools:0.0.1

Load the helper functions:

source /usr/local/bin/task_helper_functions.sh

Initialize credentials:

init_credentials

At this point, gcloud, gsutil, inspec and terraform are all expected to work without having to specify a credentials file to load from disk. See the init_credentials() function for the distinct initialization steps.

Credential Handling Responsibility

Note, credential handling should be handled solely by init_credentials when possible. Other tasks, e.g. make test_integration should not unconditionally call init_credentials to allow customization of behavior and separation of concerns.

Per-module customization

A module may need to customize the behavior of one of the task helper functions, e.g. run_integration_tests. The module should create a test/task_helper_functions.sh file which defines the function to be overridden, e.g. run_integration_tests() { ... }.

/usr/local/bin/task_helper_functions.sh loads /workspace/test/task_helper_functions.sh after defining the common function. This provides the mechanism to override these common functions.

Filesystem location

The git repository of the CFT terraform module should be mounted at /workspace within the container. This path is chosen fro compatibility with Cloud Build artifacts, which are automatically passed from job to job in a Cloud Build pipeline.

If a particular job needs to persist data for a downstream job within a pipeline, the job should write the persistent data into a sub-directory of /workspace. For example, /workspace/build may be used to capture build artifacts from an early pipeline job.

Make Tasks

Considerable time has been spent on the discussion of make tasks and their behavior. Here are the decisions. Note, make tasks are considered a public API because both the developer and the CI system calls them. Breaking the API would require updates to both developer workflows and more impactful, CI pipelines across most CFT terraform modules.

Enter the container

Developer's are expected to enter the container using make docker_run. This make tasks should be kept as simple as possible (-Morgante).

Note, credentials are expected to be setup by either the developer or by the CI system. Individual make tasks should not initialize credentials themselves to avoid entanglement and to separate credential management from task execution.

Make task specification

A task outside the container will enter the container and run the corresponding make task inside the container. If the task is started within the container, e.g. interactively via make docker_run, then call the corresponding task directly, e.g. make test_integration.

The decision on the name make docker_test_integration entering and calling make test_integration was made by Morgante and Aaron in consultation with Gary and Jeff in the CFT chat room. The other names follow from this scheme.

Additional tasks:

Outside container	Inside Container	Aliases
make docker_run	Not applicable
make docker_test_integration	make test_integration
make docker_test_lint	make test_lint	make check
make docker_test_unit	make test_unit
make docker_generate_docs	make generate_docs

[jeffmccune]

Once #250 is merged, I suggest resolving this ticket. The ticket is intended to capture design decisions from recent conversations about the developer experience.

[morgante]

One clarification in the expected workflow: the end developer should not have to load the helper functions or run init_containers. When they enter the container, those functions should already have been run for them and they get a working authenticated bash shell.

[jeffmccune]

When they enter the container, those functions should already have been run for them and they get a working authenticated bash shell.

Agree, this is nice to have. What mechanism should provide this behavior? make docker_run?

In order to avoid issues it's important to also have a mechnism to enter the container without credentials being initialized. For example, the use cases of running lint checks and documentation generation should not require credentials to be initialized. It's also helpful to not execute init_credentials upon entry when troubleshooting or modifying the init_credentials function itself.

Using ENTRYPOINT to initialize credentials caused issues because it executed unconditionally and too soon to be skipped.

[jeffmccune]

@morgante FYI, I updated the Credentials Initialization section to state interactive shells automatically have credentials initialized, while non-interactive shells (e.g. docker run ... bash -c make foo) do not have credentials initialized.

[morgante]

This is now implemented and available as our preferred CI tool.

Documentation: https://docs.google.com/document/d/1eiHpZlhEDMDG8gi4LHyv2aZs1TJmeFlF8sqCE57_j4A/edit