Enumerate meta-data keys

[moensch]

Originally logged as https://github.com/buildkite/agent/issues/822

I'd love to be able to enumerate meta-data keys. I am building pipelines for terraform, and I want to make it easy for my devs to set meta-data keys in an early step which later can be consumed by the terraform steps.

For example, an early step would have this in a build step:

steps:
  - label: build docker
    command:
     - docker build -t someimage:${BUILDKITE_COMMIT} .
     - docker push someimage:${BUILDKITE_COMMIT}
     - buildkite-agent meta-data set terraform/variables/docker_image someimage:${BUILDKITE_COMMIT}

Then, my terraform stuffs (for which I have a plugin), could have a pre-command hook to do something like this:

for var in $(buildkite-agent meta-data list 'terraform/variables/*'
do
  # TF_VAR_the_last_bit_after_the_final_slash
  tf_var_name="TF_VAR_$(echo $var | sed -e 's#.*/##')"
  printf -v $tf_var_name $(buildkite-agent meta-data get $var)
  export $tf_var_name
done

I currently have a work-around in place where I provide a bash function for folks to set those meta-data keys and every time it is called, I keep track of all the variables as a comma separate list in a separate meta-data key.

[lox]

This sounds sensible, we'll see what we can do!

[moensch]

I am just checking in to see if this is something that can be done ;-)

I currently have some home-grown implementation where I track a comma separated list of meta-data keys in a separate key. Today is the first time (surprisingly) where I hit issues with parallel steps overwriting the "tracking key" and would now have to implement some mutex. And that's a bit more effort than I wanted to put into this ;-)

[BEvgeniyS]

I'm also quite keen on this feature. I have pipeline-wide meta-data variable that I'd like to append to, and use aggregated value in the finalizing step of the pipeline. But sometimes parallel steps start at the same time and the value gets lost/overwritten.

I'd like to write separate meta-data keys with prefix, and then merge them in final stage by enumerating/filtering them. Right now I'd have to use something external to achieve this sort of consistency.

[lox]

I believe we implemented this a while back, buildkite-agent meta-data keys

[BEvgeniyS]

This is pure magic. I wish it and it's instantly there! Thanks a lot!

P.S. Please update the docs =) https://buildkite.com/docs/pipelines/build-meta-data

[lox]

Will do! Sorry it wasn’t there when you looked!

[moensch]

Uh oh - I just thought I'd check on this issue just to find it is resolved. Tested and found to be working fantastically.

I also saw that this got documented here: https://buildkite.com/docs/agent/v3/cli-meta-data#listing-keys (the pipelines/build-meta-data docs page should probably be removed).