Agent should update to the latest Plugin version in some cases

[lox]

Currently the agent will download a plugin for something like blah-blah-plugin from the master branch the first time it needs it, but it won't update it ever again.

This means that go get your changes as you are iterating, you end up updating your pipeline to reference specific commits like blah-blah-plugin#{commit-hash}.

The meta-effect of this is that it's very hard to get your plugin users hotfixes on master. They need to delete the plugin or change their pipeline.

It would be great to fetch the latest version on each job if the plugin isn't a ref or a symbolic ref /cc @sj26.

[lox]

For now the conclusion in discussion with @keithpitt is that we should encourage people to use tags vs branches!

[vektah]

I agree!

For public plugins tags are :ok_hand:, you don't want unexpected changes to external repos breaking builds.

For internal plugins you really want this instant cascade, maybe following master, maybe following a go style branch for breaking changes (v1, v2). In this case it should always pull the latest ref for that branch.

[mipearson]

Assuming that this runs on every step, how much time will checking that the tag/brach is up to date over multiple plugins take?

Github can be .. slow, sometimes.

[lox]

Good question @mipearson! That was why we didn't implement it this way initially.

[mipearson]

... if it's done inside agent (ie, in Go), could be easily done in parallel.

(and I wonder if a similar thing can be done with the s3 secrets plugin which is another slows-down-every-step point)

[lox]

Yeah, most likely. Will experiment.

In terms of the s3 secrets plugin, I just think that s3 is not a good spot for secrets. It's very hard to make efficient. SSM or Secrets Manager is the future there IMO.

Edit: Check out https://github.com/buildkite/aws-secrets-manager-agent-hooks

[tigris]

FWIW you can hack this yourself by doing a git fetch v1 (or whatever "stable" branch you're using, perhaps you can grok the branch from vars) as early as possible in the hook chain. E.g. the environment hook.

The downside is you now can't use the environment hook for anything else.

Plus the obvious existing downsides of using a mutable branch, including but not limited to non-repeatable builds, slower build steps, etc.

But the upside of instant release. Great for things you deem internally to be sufficient for the risks, e.g. vulnerability scanning. If you trust the source code chain of access, then I can see this being quite beneficial.

[evandam]

:+1: for this, not sure if there have been any recent developments.

I'm looking to manage a plugin internally that other teams in my org will use, and I'd love for them to be able to pull the latest version of the plugin without needing to update their pipelines, make sure each feature branch being built references the correct version, etc.

I agree for public-facing plugins that referencing tags/releases is the way to go, but I'd like to have the option to pull the latest when possible rather than let plugins go stale.

Alternatively, is there a recommendation for the meantime that we might be able to accomplish with an agent hook perhaps that goes into $BUILDKITE_PLUGINS_PATH, loops over $BUILDKITE_PLUGINS checking for unpinned versions and does a git pull?

[KaanOzkan]

@lox +1. Is this option on the roadmap? It has been a year since you commented on this.

[pda]

Our current thinking is that:

• we encourage pinning to a specific stable ref (a tag or a commit)
  • but we understand that doesn't suit all cases, so…
• the agent should update plugins before every step/job
  • unless they've already been updated in the past hour

Any feedback on whether a one hour cache time would be sensible?

[sj26]

unless they've already been updated in the past hour

This sounds sensible to me.

[mipearson]

This makes sense.

Would it still check for an update if pinned to a tag (or other ref that isn't a commit hash)?

Not a use case we have right now, but it'd be nice to tell users to use #v1 of a plugin, then work on #v2 but still have bugfixes to the #v1 tag/branch come through automatically.

[sj26]

I think convention is that tags shouldn't change, so I'd expect a tag to continue working as it does.

A common convention to provide version updates is to create a "1-stable" branch or similar which can be tracked.

We've also spoken about semver-style tag negotiation, but that seems much more complicated without proportional benefit, at least for the moment.

[mipearson]

Yep - so, same question as above, but they've specified a branch id - the refresh once-per-hour behaviour would apply?

[sj26]

That's my understanding! 🙌

@pda?

[pda]

@mipearson @sj26 I think ideally we'd treat tags as immutable (even though git doesn't), but how would we differentiate between a tag and a branch? Unless there's a good answer to that, I'd say the update after an hour would apply to anything that isn't definitely a commit hash.

[pda]

• the agent should update plugins before every step/job
  • unless they've already been updated in the past hour

Oh, and I left the reasoning for that implicit, I should have been explicit.

Part of the concern is performance — each build step fetching multiple plugins every single time could slow things down substantially.

The other concern is rate limiting — if a plugin is used org-wide in a large organisation, it could be triggered thousands of times per minute. Apart from being inefficient, fetching it each time could incur rate limiting from the plugin repo provider (👋🏼 GitHub).

[sj26]

... how would we differentiate between a tag and a branch?

[[ "$(git rev-parse --symbolic-full-name "master")" == refs/tags/* ]] && echo yes || echo no
no

[[ "$(git rev-parse --symbolic-full-name "v1.0")" == refs/tags/* ]] && echo yes || echo no
yes

[mipearson]

I'd err on not treating tags as immutable, personally, in large part because git doesn't. You've then got a solid split between "refs that are immutable" (commit hashes) and "refs that might change" (everything else).

[sj26]

Hmm, I think I disagree.

Git tags are designed to be immutable. You have to try really hard to mutate them:

$ git tag test 
fatal: tag 'test' already exists

$ git tag --force test
Updated tag 'test' (was a862e5c)

$ git push origin test
To https://github.com/sj26/test.git
 ! [rejected]        test -> test (already exists)
error: failed to push some refs to 'https://github.com/sj26/test.git'
hint: Updates were rejected because the tag already exists in the remote.

$ git push --force origin test
Enumerating objects: 4, done.
Counting objects: 100% (4/4), done.
Delta compression using up to 8 threads
Compressing objects: 100% (2/2), done.
Writing objects: 100% (4/4), 1.04 KiB | 1.04 MiB/s, done.
Total 4 (delta 0), reused 0 (delta 0), pack-reused 0
To https://github.com/sj26/test.git
 + a862e5c...8b625d6 test -> test (forced update)

I'd say we should respect that design and treat tag mutations as exceptional, the same way that force pushing a commit makes it inaccessible is considered exceptional.

Tags are a good, conventional way for us to make common use cases like pinning to a version much, much faster, and more reliable. I think this outweighs the risk of a forced tag mutation.

[mipearson]

TIL. I had a mental model in my head that tags were easily mutable which was not correct, as you've shown. :)

[pda]

Sounds good 👌🏼

[staticfloat]

We are working around this issue by putting the following in our agent environment hook; it has a few problems:

• It updates all plugins, so if you use many across multiple pipelines in your agent, the environment hook may start taking up an appreciable amount of time
• If you have git tags that are 40 hexadecimal characters, it will not update it. ;)

# Update all our git repositories, in parallel
for d in /buildkite/plugins/*-plugin*; do
    TAG="${d#*-plugin}"
    TAG="${TAG#-}"

    # If the tag is a git literal, don't bother to update it
    if [[ ${TAG} =~ [0-9a-z]{40} ]]; then
        continue
    fi

    # If we didn't properly determine the tag name, skip
    if ! git -C "${d}" rev-parse "${TAG}" >/dev/null 2>&1; then
        continue
    fi

    # Do git update/checkout in parallel
    (
        echo "Updating ${d} to latest tag ${TAG}";
        git -C "${d}" fetch --tags --force &&
        git -C "${d}" checkout "${TAG}"
    ) &
done

# Wait for all `git` operations to finish
wait

We use a github action to create major-only tags such that when we release vX.Y the vX tag is updated to point to the latest vX.Y tag, so that users can depend on the latest API-compatible release by specifying vX, or locking to a specific release by specifying vX.Y.

[tinder-tremaineeto]

I'd like to +1 this ask -- we're hoping to use the #latest tag in multiple pipelines, but the fact that the agent won't always update the tag once checked out means that the plugin could be out of date with what we have tagged as #latest. Having a way around this would be awesome.