finer grained controls for what triggers a job

[jclulow]

Today, jobs are triggered whenever at least one of these actions occurs:

• a push is made to the repository (any branch name)
• a pull request is opened against the repository (from some other fork on GitHub)

Note that jobs are run for unique GitHub Check Suites, and there is generally a 1:1 relationship between a specific git commit (based on the commit hash) and a Check Suite; if a job would be triggered by both a pull request and a push to a branch, we still only run it at one time.

Occasionally a job will represent the combination of:

• expensive to build and creates large output objects
• is only needed for final integrated changes on the default branch, not for personal branches or pull requests

The per-job configuration should allow finer control over the events that will trigger a job to run, allowing for a list of patterns to match branch names, and the ability to ignore pull requests.

[pfmooney]

As we're looking to build a more robust test suite in propolis, it might be nice to be able to explicitly trigger a buildomat run, rather than automatically based on pushes to any of the covered git refs. Runs of the "expensive" suite could be limited to situations when the consumer wishes to opt-in.

[jclulow]

How configurable are you hoping this will be? Would it be enough to have a button to click to mark a particular pull request as one that includes the optional jobs?

[pfmooney]

That, or perhaps an API endpoint to request that optional job(s) be run? For example, we're building Falcon binaries for propolis on every commit when really that could be deferred to when Ry and co. explicitly want a new version. Similarly, if someone is iterating in a PR, we probably don't want the "run PHD tests, including some expensive Windows work" job to run until they're reasonably please with the state of things.

Maybe @gjcolombo has more thoughts as well?

[jclulow]

An API endpoint is a challenge from an authentication and authorisation perspective, but we can certainly look at something.

[gjcolombo]

For Propolis integration testing I think having scheduled runs would go a long way. With scheduled runs, per-pull-request CI would run inexpensive smoke tests with just one kind of guest--enough to make sure nothing is grievously broken. Then, every once in a while (every 24-48 hours or so?) a scheduled run would kick off that runs a more extensive test suite with more kinds of guests. The hope is that if something fails in the big test suite, it will be relatively clear from inspection (or easily determined by bisection) which commit actually broke things.

I know GHA supports periodic scheduled actions, but don't know enough about how the GHA/Buildomat webhooks work to know if they'd be suitable here or if we'd have to do something else (e.g. checking a schedule into the Buildomat config in the repo).

Also, since this model doesn't tie scheduled runs to specific PRs, we'd need some other way to discover their statuses and be notified when tests are failing; perhaps this is something else that GHA can give us "for free" if we use it as the task scheduling mechanism. (But, again, I don't really understand any of the tradeoffs here yet.)