Closed jeffmccune closed 5 years ago
Once #250 is merged, I suggest resolving this ticket. The ticket is intended to capture design decisions from recent conversations about the developer experience.
One clarification in the expected workflow: the end developer should not have to load the helper functions or run init_containers. When they enter the container, those functions should already have been run for them and they get a working authenticated bash shell.
When they enter the container, those functions should already have been run for them and they get a working authenticated bash shell.
Agree, this is nice to have. What mechanism should provide this behavior? make docker_run
?
In order to avoid issues it's important to also have a mechnism to enter the container without credentials being initialized. For example, the use cases of running lint checks and documentation generation should not require credentials to be initialized. It's also helpful to not execute init_credentials
upon entry when troubleshooting or modifying the init_credentials
function itself.
Using ENTRYPOINT to initialize credentials caused issues because it executed unconditionally and too soon to be skipped.
@morgante FYI, I updated the Credentials Initialization section to state interactive shells automatically have credentials initialized, while non-interactive shells (e.g. docker run ... bash -c make foo
) do not have credentials initialized.
This is now implemented and available as our preferred CI tool.
Documentation: https://docs.google.com/document/d/1eiHpZlhEDMDG8gi4LHyv2aZs1TJmeFlF8sqCE57_j4A/edit
This ticket captures discussions and decisions regarding credential handling when running automated tests for CFT Terraform modules. The scope of this design is local test development on a developer's workstation and also in CI.
Containers
For both local development and CI, a single
developer-tools
container is used. This container includes all necessary dependencies to run the lint, unit and integration tests for a CFT terraform module.Credentials
CFT terraform modules commonly use the following tools which need credentials to be initialized within the context of the running container. Each tool has a slightly different way of handling credentials:
If additional tools a required as dependencies:
init_credentials
function inside of/usr/local/bin/task_helper_functions.sh
to initialize the credentials from SERVICE_ACCOUNT_JSON for the new tool.Design Decision
Credentials are to be passed into the container using a single environment variable,
SERVICE_ACCOUNT_JSON
. This variable name is chosen for backwards compatibility with existing CI jobs and to avoid a naming conflict withGOOGLE_CREDENTIALS
. The value ofSERVICE_ACCOUNT_JSON
must be the actual JSON string content of a service account JSON key. For exampleSERVICE_ACCOUNT_JSON="$(< ~/.credentials/jmccune-devtools-0abcde01234.json
The value must not be a filesystem path.Filesystem Paths
Generally, speaking filesystems paths should be avoided when loading SA credentials. The intent is to rely on automatic loading of credentials when possible so that jobs may be deployed with service accounts managed by the platform. For example, consider a VM running in GCE bound to a service account. This VM may be used to run the integration tests without transferring a private key, increasing security.
Credentials Initialization
The developer tools image initializes credentials from the
SERVICE_ACCOUNT_JSON
input. This initialization is automatically handled via~root/.bashrc
for interactive shells. For non-interactive shells, e.g. when running a command, the initialization does not happen automatically. The caller is responsible for credentials initialization in this case, e.g. the Make task or the CI pipeline definition.The expected workflow for a developer working on their local workstation is:
make docker_run
Credentials initialization may require writing the key to the filesystem. In this situation, a secure temporary file should be used and should be automatically cleaned up when the running container reaches end of life, or sooner depending on the use case.
Note
Example Implementation
An example implementation is provided in #250 and the local development workflow looks like:
Enter the container:
Load the helper functions:
Initialize credentials:
At this point, gcloud, gsutil, inspec and terraform are all expected to work without having to specify a credentials file to load from disk. See the
init_credentials()
function for the distinct initialization steps.Credential Handling Responsibility
Note, credential handling should be handled solely by
init_credentials
when possible. Other tasks, e.g.make test_integration
should not unconditionally callinit_credentials
to allow customization of behavior and separation of concerns.Per-module customization
A module may need to customize the behavior of one of the task helper functions, e.g.
run_integration_tests
. The module should create atest/task_helper_functions.sh
file which defines the function to be overridden, e.g.run_integration_tests() { ... }
./usr/local/bin/task_helper_functions.sh
loads/workspace/test/task_helper_functions.sh
after defining the common function. This provides the mechanism to override these common functions.Filesystem location
The git repository of the CFT terraform module should be mounted at
/workspace
within the container. This path is chosen fro compatibility with Cloud Build artifacts, which are automatically passed from job to job in a Cloud Build pipeline.If a particular job needs to persist data for a downstream job within a pipeline, the job should write the persistent data into a sub-directory of
/workspace
. For example,/workspace/build
may be used to capture build artifacts from an early pipeline job.Make Tasks
Considerable time has been spent on the discussion of make tasks and their behavior. Here are the decisions. Note, make tasks are considered a public API because both the developer and the CI system calls them. Breaking the API would require updates to both developer workflows and more impactful, CI pipelines across most CFT terraform modules.
Enter the container
Developer's are expected to enter the container using
make docker_run
. This make tasks should be kept as simple as possible (-Morgante).Note, credentials are expected to be setup by either the developer or by the CI system. Individual make tasks should not initialize credentials themselves to avoid entanglement and to separate credential management from task execution.
Make task specification
A task outside the container will enter the container and run the corresponding make task inside the container. If the task is started within the container, e.g. interactively via
make docker_run
, then call the corresponding task directly, e.g.make test_integration
.The decision on the name
make docker_test_integration
entering and callingmake test_integration
was made by Morgante and Aaron in consultation with Gary and Jeff in the CFT chat room. The other names follow from this scheme.Additional tasks: