CI validation of Profile Registries

[chrnorm]

We can support some kind of CI-based testing for Profile Registries so that pull requests to profile registry repos can be validated prior to being merged.

[chrnorm]

Some additional context from Eric in the Common Fate Community Slack:

I'm setting up some CI for my granted registry (since I am concerned a breaking change could break all users in an unexpected way, and am curious if there are any ideas around how that could work. Will also post some of my ideas.

A lot can be done with just basic linting and checking... yamllint, check EOF, syntax checking to avoid duplications. Using pre-commit for a decent amount of that static checking... including checking for aws-credentials, which seems prudent given the subject matter.

The other idea I had was iterating over all the repos and adding them from the local directory with --prefix-all-profiles to automatically disambiguate. That seems like good, basic smoke test. Something I do note is that it seems difficult to programmatically remove profiles. Makes sense for a human CLI tool, but with the registry, feels like a gray area to me.

I didn't set up the AWS CLI — and it would be very limited anyway, because this CI doesn't have much in the way of credentials, but I was thinking about other possibilities... perhaps just checking expected profiles against output from aws configure list-profiles would be a sensible test? If nothing else, that ought to ensure the INI renders in a way AWS SDKs can understand.

Another thought I had was around validation of human configuration changes... maybe a JSON Schema or something for the configuration files?

Or maybe even further, and a granted registry validate command that checks + lints through all of the source files, including templates (with placeholders)

I'm not using variables yet, but at the point when I am, I wonder how I could incorporate and mock those in my CI. I'm sure there's a config file somewhere that I could poke at to set those non-interactively (or even just prepare the file) but I had a hard time finding docs on that.

[sosheskaz]

JSON schema would be nice in general — VS Code supports arbitrary JSON schemas when validating YAML files, so it would enable me to pretty easily configure my local editor to validate registries.

For reference, here's a quick look at the core of the shell script I use for validating currently:

#!/bin/bash

die() {
  echo "fatal:" "$@"
  exit 1
}

ROOT_DIR=$(git -C "$(dirname "$0")" rev-parse --show-toplevel) || die "failed to determine root dir"
cd "${ROOT_DIR}" || die "failed to cd to '${ROOT_DIR}'"
mkdir -p ~/.aws || die "failed to create aws dir"

for filename in *.yml
do
  name="$(basename "${filename}")" || die "failed to get basename of file '${filename}'"
  granted registry add --name="${name}" --url="${ROOT_DIR}" --filename="$(basename "${name}")" --prefix-all-profiles || die "failed to add registry ${name}"
done

Crude, and makes assumptions that may not always hold, but it at least checks that granted exits successfully, and supports multiple files.