server/cli: improve/redesign template validation

[colindean]

Description

I have a repository of templates used in my repositories. I want to validate these templates; to have some kind of CI running a smoke test on them before allowing them to be merged into the default branch.

Normally, I could run vela validate pipeline as a pre-commit hook to validate that file locally to prevent a broken config from getting committed.

If I wanted to simply validate the YAML of a template, I could run it through just about any YAML parser. However, because YAML templates have gotemplates in them, I can’t use just any YAML parser.

Obviously, using Starlark would obviate this problem to a degree — I assume that Starlark has some kind of syntax checking validation mode — but I’d still want some validation for that but I suspect it already exists in a more generic way.

I tried using the built-in template validation, but it seems that the templates are only loaded and checked when referenced. I don't want to actually consume the templates this the templates repo config.

See this example .vela.yml for the templates repo:

version: "1"

templates:
  - name: docker
    source: git.example.com/myorg/vela-templates/docker.yml
    type: github
  - name: alerts
    source: git.example.com/vela-templates/alerts.yml
    type: github
  - name: build
    source: git.example.com/vela-templates/poetry-setup-unittest-staticchecks.yml
    type: github

steps:
  - name: validate templates
    image: alpine:3.15
    commands:
      - apk add curl tar gzip
      - curl -L https://github.com/go-vela/cli/releases/latest/download/vela_linux_amd64.tar.gz | tar zx
      - cp vela /usr/local/bin
      - |
        vela validate pipeline --template \
          --template-file docker:docker.yml \
          --template-file alerts:alerts.yml \
          --template-file build:poetry-setup-unittest-staticchecks.yml

The above passes event when docker.yml is modified in a way that isn't even valid YAML.

I also tried using gomplate but it lacks at least the default function that Vela exposes as a part of integrating the sprig library extensions for gotemplates.

Value

Validating at least the syntax of templates would prevent bad templates from breaking downstream pipelines.

Definition of Done

The vela CLI tool can validate template files OR a Vela plugin exists that can validate all templates in a file glob or referenced in the config without invoking them.

[JordanSussman]

Obviously, using Starlark would obviate this problem to a degree — I assume that Starlark has some kind of syntax checking validation mode — but I’d still want some validation for that but I suspect it already exists in a more generic way.

Starlark validation will get you further - but it will not let you know if the returned output is valid for vela or not.

I tried using the built-in template validation, but it seems that the templates are only loaded and checked when referenced. I don't want to actually consume the templates this the templates repo config.

The Vela compiler automatically expands any templates within the file so doing something like what you described is not possible.

The vela CLI tool can validate template files OR a Vela plugin exists that can validate all templates in a file glob or referenced in the config without invoking them.

A couple of ways to accomplish this today:

First option - create multiple .vela.yml files

Have a minimal .vela.yml like the following:

version: "1"

steps:
  - name: validate templates
    image: alpine:3.15
    commands:
      - apk add curl tar gzip
      - curl -L https://github.com/go-vela/cli/releases/latest/download/vela_linux_amd64.tar.gz | tar zx
      - cp vela /usr/local/bin
      - |
        vela validate pipeline --template \
          --template-file docker:docker.yml \
          --template-file alerts:alerts.yml \
          --template-file build:poetry-setup-unittest-staticchecks.yml \
          --file vela-test.yml

Create a vela-test.yml file with the template(s) you want to test:

version: "1"

templates:
  - name: docker
    source: git.example.com/myorg/vela-templates/docker.yml
    type: github
  - name: alerts
    source: git.example.com/vela-templates/alerts.yml
    type: github
  - name: build
    source: git.example.com/vela-templates/poetry-setup-unittest-staticchecks.yml
    type: github

steps:
  - name: sample
    template:
      name: sample

  - name: alerts
    template:
      name: alerts

  - name: build
    template:
      name: build

Second option - utilize Vela compiler functions within Golang code

Building something like this example would enable you to have thorough input and output validation that vela validate pipeline --template will not be able to accomplish without some additional wrapper scripts. You could even take this to another level by validating the output matches what you expect by doing something like what the Vela compiler tests do.

[colindean]

This is kicking me in the right direction, but the validator output isn't very helpful:

$ vela validate pipeline --template \\n  --template-file docker:docker.yml \\n  --template-file alerts:alerts.yml \\n  --template-file build:poetry-setup-unittest-staticchecks.yml\n  --file vela-test.yml\n

time="2022-03-23T18:17:51Z" level=fatal msg="templates block not properly configured in pipeline"

It tells me there's an error but not where.

[colindean]

Nevermind, missed a \ when I put the --file option in the .vela.yml so it was still trying to read .vela.yml instead of vela-test.yml. I've now got a passing validation!

[colindean]

Ya know, I think this pattern could be refactored into a plugin. The plugin could take the list of files to validate and build the vela-test.yml dynamically. There might be some limitations to it, such as requiring all of the templates to use default filters throughout, or perhaps the validation plugin could allow passing testing vars to the test config.