Ability to restrict runs-on

[TWiStErRob]

Intro

Hey, following recent changes, essentially any value of runs-on: is accepted. Since one of the outcomes of this action is to reduce the possibility of broken workflows, I think it would be very beneficial to validate runs-on somehow.

Proposal

• By default validate against freely available GitHub Hosted runners (no XL, no self-hosted)
  This way the validator brings immediate value to hopefully most users,
• With a CLI parameter (--allowed-runners) allow setting valid (incl. custom) values.
  but also gives the ability to override if needed.

Syntax

• Comma separated list
• Prefix @ for built-in (in the validator) groups
• Prefix - for exclusion
• No - prefix = positive listing

Built-in groups

(might be more):

• github-free: ubuntu-latest, ubuntu-22.04, etc.
• github-latest: ubuntu-latest, windows-latest, etc.
• github-xl: -xl and -cores suffixed ones
• self-hosted: special value for current 0.5.4 behaviour

Custom groups

If there's a known list of self hosted runners, it's possible to just list them all and use no groups. That is, to define a custom group, users need to list all runners explicitly.

For reusability the custom groups could be extracted into environment variables / organization level variables, like ACTION_VALIDATOR_ALLOWED_RUNNERS, and then that used in CLI with shell expansion: --allowed-runners=${ACTION_VALIDATOR_ALLOWED_RUNNERS},-ubuntu-latest.

Calculation

• Start with empty set
• Add all positive groups, and positive runners
• Remove all negative groups and runners

i.e. order of flags doesn't matter.

Examples

0. the default when no argument provided

   --allowed-runners=!github-free

1. a GitHub Enterprise user might validate like this, if they allow 2 self-hosted runners, and want to be explicit about versions:

--allowed-runners=!github-free,@github-large,-@github-latest,my-special-runner1,my-special-runner2

2. an open-source community member GitHub user, who wants to use latest without automagic updates:

   --allowed-runners=ubuntu-22.04,windows-2022,macos-12

[mpalmer]

Hi Robert, thanks for starting this convo.

I'm definitely in favour of introducing a way to better validate runs-on values. Not being a custom runner user myself, I'm probably not in a strong position to define a specification for how to do it. Thus, I can at best kibbitz on the work of others.

The only feedback I've got on your proposal so far is that I'm wary of using ! in a command-line argument, as shells have a habit of treating that character specially. I know of programs that use @ as a prefix for group-like behaviour; would that be an acceptable prefix in this case?

[TWiStErRob]

Thanks. I agree with @, changed above and reformatted a bit.

I'm not a self-hosted user either, so let's get opinions from those who do: @MPV (#59), @mathew-fleisch (#5), @deviantintegral (#51)

[deviantintegral]

👏 This all looks good to me. It gives us a way in CI to make sure runners are valid, catching typos in custom runner names.