Support flattening lists in step plugins list

[KevinGreen]

Is your feature request related to a problem? Please describe. In an effort to keep our pipeline steps more concise we're using YAML anchors and aliases to define the plugins for our steps at the top level and we reuse the lists in multiple places. This works well but we have some steps that need to fetch artifacts or add additional plugins. Since YAML doesn't support merging lists with anchors like it does for dictionaries, we need to redefine the list of plugins for these other use cases. Because of how our caches are layered, our lists of plugins can be 9 elements or longer and we have 3 redefinitions of the same list with only one plugin change each. The motivation to DRY this up is that our pipeline is already pretty long and over time when new cache layers are added, I don't want it to be so easy to forget to add it to the other plugin lists used by other steps.

Describe the solution you'd like If the command plugins option also supported lists of lists of plugins as elements, we could do something like this simplified example:

docker-plugins: &docker-plugins
  - docker-compose#v3.9.0: ~
  - cache#v1.0.0:
    keys: "v1-gems"
        paths: "vendor/bundle"

asset-plugins: &asset-plugins
  - cache#v1.0.0:
    keys: "v1-assets"
        paths: "public/assets"

steps:
  - label: Do something with assets
    command: ".buildkite/steps/asset_stuff"
    plugins: *asset-plugins

  - label: Do something with docker
    command: ".buildkite/steps/docker_stuff"
    plugins: *docker-plugins

  - label: Do something with assets and docker
    command: ".buildkite/steps/rspec"
    plugins: [*asset-plugins, *docker-plugins]

The statement [*asset-plugins, *docker-plugins] produces a list with 2 elements both being lists of plugins. If the agent supported running a flatten operation on the plugins list, this would work.

Describe alternatives you've considered I don't see any downsides to this solution, the plugin order is still clear as long as the flattening happens in order of list definition. It should also be a pretty small change, though might require a schema change in the backend to support validating the pipeline.

Additional context Add any other context or screenshots about the feature request here.

[moskyb]

Hey kevin! i've raised this with the team internally, and we'll discuss it over the next little while - i see your use case here, having heaps of plugin defs gets hard to read.

one thing that you could do in this case is generate your pipeline.yml using some other tool (possibly Starlark/Jsonnet and friends), but in that case you'd probably lose the readability of it, which would be a bummer.