agent support for jsonnet

[jmendiara]

When developing multiple complex pipelines, a repetition problem may arise. You can end having the same steps defined in different pipelines, and maintaining them implies modifying lots of files

One approach can be "yaml templates" + "concatenation"

# steps.yml
deploy: &deploy
  label: "Deploy"
  command: deploy.sh

undeploy: &undeploy
  label: "Undeploy"
  command: undeploy.sh

test: &test
  label: "Test"
  command: test.sh

# pipelineA.yml
steps: 
  - *deploy
  - wait
  - *undeploy

# pipelineB.yml
steps: 
  - *deploy
  - wait
  - *test
  - wait
  - *undeploy

And upload the pipeline this way, as yml does not have an "import" directive cat steps.yml pipelineA.yml | buildkite-agent pipeline upload cat steps.yml pipelineB.yml | buildkite-agent pipeline upload

This solves a single use case making a workaround: both pipelineA.yml and pipelineB.yml are not valid yaml.

Supporting jsonnet can solve that use case and lots more. One i'm thinking about are "preprocessor", as it could be coded directly in our pipelines without any external tool

There is a go official implementation and will allow us to code dynamic pipelines.

// steps.libsonnet
{
  deploy: {
    label: 'Deploy',
    command: 'deploy.sh',
  },
  undeploy: {
    label: 'Undeploy',
    command: 'undeploy.sh',
  },
  test: {
    label: 'Test',
    command: 'test.sh',
  },
}

// pipelineA.jsonnet
local steps = import 'steps.libsonnet';
{ 
  steps: [
    steps.deploy,
    'wait',
    steps.undeploy,
  ],
}

buildkite-agent pipeline upload pipelineA.jsonnet

[petemounce]

👎

I would really like to avoid BK supporting a pre-processor for the really-quite-simple YAML, because I think

• making something general enough will be quite difficult to onboard people into
• allowing imperative logic into a format that's currently a clean and declarative format will make it really easy for pipeline-maintainers to shoot themselves in the foot with respect to prediction and external dependencies
• I'd rather, if I need to do complex things to a pipeline, home-grow a tool that meets my specific needs rather than understand a really general one that might or might not hit my use-case(s) that cannot be met by the combination of simple YAML syntax, named anchors and dynamic-steps-upload

TeamCity allows Kotlin, Jenkins allows Groovy - in my experience using, supporting, maintaining and educating non-build/release engineers with both of those, it's entirely too easy to end up with something completely inscrutable. In my experience, engineers that use CI treat CI-configuration a bit like shell scripts rather than a delivery-pipeline without which their code cannot get in front of customers.

Generate a pipeline from a template - sure. But make the actual steps that are invoked be from a static declarative file rather than have the cognitive load of figuring out what the steps should be after the pre-processor is done.

[keithpitt]

@jmendiara thanks for really good suggestion! I don't think this is something we'd want to introduce into the official syntax. I can't think of many use-cases where it would be useful aside from the current use case your facing (really complex pipelines).

One of my favourite things about buildkite-agent pipeline upload is that you can write your pipelines how ever you want and transpose them into our pipeline.yml format. For example, this tool here: https://github.com/saymedia/jobsworth supports a custom pipeline.yml format that is turned into a something that buildkite-agent pipeline upload understands after it's applied a bunch of it's own rules.

In your case, you could have a cli tool that parses jsonnet for you, i.e.:

jsonnet my-pipeline.yml | buildkite-agent pipeline upload

As long as what ever you pipe into buildkite-agent pipeline upload is a valid Buildkite pipeline.yml, everything should just work :)

I'll close this for now - and thanks again for the suggestion! Keep 'em coming!