Support for self-hosted runner labels or bypass rules

[mathew-fleisch]

I run self-hosted github action runners, and action-validator doesn't like the syntax when the runner name is not self-hosted/ubuntu-latest/macos-latest etc. Self-hosted runners support runs-on triggers by labels. Is there maybe a way to bypass some rules or add an exception configuration file?

name: Self-Hosted Ci Job

on:
  workflow_dispatch:
    types:
      - trigger-job
jobs:
  sync:
    runs-on: [custom-runner-name]

The type of error I see looks like this:

Running pre-commit hook for GitHub Actions: https://github.com/mpalmer/action-validator
gitValidation failed: ValidationState {
    errors: [
        OneOf {
            path: "/jobs/sync",
            states: [
                ValidationState {
                    errors: [
                        OneOf {
                            path: "/jobs/sync/runs-on",
                            states: [
                                ValidationState {
                                    errors: [
                                        Enum {
                                            path: "/jobs/sync/runs-on",
                                        },
                                    ],
                                    missing: [],
                                    replacement: None,
                                },
                                ValidationState {
                                    errors: [
                                        WrongType {
                                            path: "/jobs/sync/runs-on",
                                            detail: "The value must be array",
                                        },
                                    ],
                                    missing: [],
                                    replacement: None,
                                },
                                ValidationState {
                                    errors: [
                                        Pattern {
                                            path: "/jobs/sync/runs-on",
                                        },
                                    ],
                                    missing: [],
                                    replacement: None,
                                },
                            ],
                        },
                    ],
                    missing: [],
                    replacement: None,
                },
                ValidationState {
                    errors: [
                        Properties {
                            path: "/jobs/sync",
                            detail: "Additional property 'runs-on' is not allowed",
                        },
                        Properties {
                            path: "/jobs/sync",
                            detail: "Additional property 'steps' is not allowed",
                        },
                        Required {
                            path: "/jobs/sync/uses",
                        },
                    ],
                    missing: [],
                    replacement: None,
                },
            ],
        },
    ],
    missing: [],
    replacement: None,
}
❌ .github/workflows/sync.yaml
Fatal error validating .github/workflows/sync.yaml: validation failed

[mpalmer]

There's no way to bypass or disable particular checks, but I'm not sure there needs to be here. If GitHub accepts that same Workflow, then action-validator is at fault, pretty much by definition. It uses a third-party JSON schema to define what is acceptable, so presumably that JSON schema just needs to be tweaked a little.

That being said, looking at the fine manual, that suggests that self-hosted runners need to use runs-on like this:

runs-on: [self-hosted, something, something]

... and looking at the schema, it's at least making it look like it's trying to conform to that, although I'm far from being a JSON schema expert.

I'm loathe to move the JSON schema too far away from what the GitHub documentation describes, because (at least in theory) GitHub could change the implementation behind the scenes so that it more closely resembles the docs, at which point action-validator would be giving the :+1: to workflows that GitHub was rejecting, which would be... bad. On the other hand, there's the annoyance of having action-validator give the :middle_finger: to workflows that GitHub is happy to accept, so... there's no good solution here. I think it'll have to be decided on a case-by-case basis, comparing the breakage risk to the annoyance risk.

If you can give me a minimal working self-hosted workflow that GitHub accepts and action-validator rejects (or at least confirm that the example you gave in your initial message does, in fact, do the right thing), then I'll be able to gauge the exact level of deviation we're talking about needing.

[mathew-fleisch]

Yea... this is a tricky one. I was thinking something like how shellcheck has exceptions you can place at the top of files as a comment or some such. 🤷 Or maybe like a warning, but not an outright failure?

I actually wrote a blog post about setting up self-hosted runners and how to add these arbitrary labels: https://github.com/mathew-fleisch/blog/blob/main/blog/2021-05-17/self-hosted-github-action-runners-on-self-hosted-kubernetes-cluster.md

I can give you a working action that runs-on one of these arbitrary labels i've set up for a self-hosted runner that fails action-validator: https://github.com/mathew-fleisch/docker-dev-env/blob/main/.github/workflows/build-container.yaml

I also use the self-hosted runners to build new versions of themselves the same way: https://github.com/mathew-fleisch/github-actions-runner/blob/main/.github/workflows/build-container.yaml

[mpalmer]

Hmm, well that seems fairly conclusive: real-world workflows that successfully use a runs-on that would appear to be completely at odds with the published documentation. I wonder how responsive the GitHub people would be to requests to fix their documentation? That'd be the ideal outcome, because then we'd have a published spec that I could adjust the schema to match. In the absence of that, though, I think I'll probably do it anyway, because if nothing else, if GitHub go and quietly change the behaviour in the future, you and (I assume) a whole lot of other people will be Extremely Inconvenienced.

[mpalmer]

OK, I've submitted an issue against the GitHub docs to see if they'll be willing to clarify the situation. Having gone through the docs a couple more times now with a fine-toothed comb, and comparing it to the schema, I'm pretty sure there's at least one place where the schema is unnecessarily restrictive; I plan on whipping up some dummy test cases to verify my supposition, and then getting my hands dirty in the schema.

[mathew-fleisch]

Following up on this issue, the syntax highlighted by the github action folks works for me and the action-validator still passes. However, the problem now is that there isn't an arm version (to run this tool on a raspberry pi). I'll close this issue and open another one for that: https://github.com/mpalmer/action-validator/issues/10