Switch to syncing tracks via push rather than pull config

[SaschaMann]

GitHub recently introduced a new feature called Environments that let you create a »container« of sorts to limit access to secrets in a more fine-grained way than giving everyone with the commit bit permission to access and use them. GHA workflows can require access to an environment, which will prevent them from running until a select group of people has »reviewed« the run and clicked a button.

This feature would allow us to switch to trigger events from problem-specifications to the track repos, rather than the current configuration where track repos must pull new info.

What would (need to) change?

• Configure a machine user with write access to all track repos.

• Configure an environment in problem-specifications with an access token of that bot user.

• Determine a trusted group of (currently up to 6) people who can approve workflow runs in that environment.

• Configure a workflow in problem-specifications that sends out repository_dispatch to each track repo. repository_dispatch events are kinda like webhooks. They are web requests that may contain an arbitrary payload that are sent to the track repos and trigger a workflow run there. There are two viable triggers, both with their own up- and downsides:

  • push:
    • Send an event to tracks on every commit to the main branch in problem-specifications.
    • 👍 Changes to problem-specifications map 1-to-1 to repository_dispatch.
    • 👍 Changes will make their way to the track repos ASAP.
    • 👎 This may cause a lot of work for the workflow approvers, as they’ll have to approve a run for every PR.
    • 👎 It is not clear when a change will hit the track repos as it depends on the availability of the approvers.
  • schedule:
    • Once a day (or more/less often), send out an event that contains a »summary« of recent changes (if there were any) to the track repos.
    • 👍 The work for workflow approvers is limited.
    • 👍 It’s fairly predictable how long it will take for a change to trigger an event in the track repos.
    • 👎 It may take more time for a change to make its way into a track repo.
    • 👎 The receiving workflows will be more complex, as they’ll have to handle several changes at once.

• Each track can then decide if and how they want to act on these events. For example, they could…

  • … automatically run their test generators on the new tests.
  • … open an issue that contains a summary of the change.

  Some of these could be standardized in actions, so that tracks don’t have to write their own workflows from scratch.

Advantages

• Less resources are wasted on scheduled runs looking for changes in problem-specs.

• We can create an event payload ourselves to make it easier for tracks to handle updates (semi-) automatically.

• It would be easy to setup each track with a default behaviour, like creating an issue, so that even tracks that lack the peoplepower to setup elaborate automated workflows would benefit from update notifications.

• All of this would also be possible with webhooks and an external (serverless) service where secrets are stored. However, that would require additional infrastructure and would make it harder to change or fix the setup compared to a purely GHA-based setup.

[ErikSchierboom]

I like the idea here. It would be really helpful for maintainers to chip in and indicate if they think this is useful for their tracks.

This may cause a lot of work for the workflow approvers, as they’ll have to approve a run for every PR.

At the current rate of PR merging, this would roughly equate to one approval per day. So I don't think it is that bad. But the schedule option could work equally well and is fully automated.

[SaschaMann]

The schedule would still require approval but it has a cap for how often it would need it.

[wolf99]

I like the idea here. It would be really helpful for maintainers to chip in and indicate if they think this is useful for their tracks.

For the C track we have low contributor time and no test generator. Creating an issue for each change may be good for the track as it would give a place for potential new contributors to easily pick up from, i.e. being an on-ramp of sorts.

However, I would be concerned that when issues don't get picked up but later ones continue to come in, it may result in multiple historic issues for one exercise that contributors will then need to navigate. Perhaps this could be solved in automation on the track end in some way?

[SaschaMann]

However, I would be concerned that when issues don't get picked up but later ones continue to come in, it may result in multiple historic issues for one exercise that contributors will then need to navigate. Perhaps this could be solved in automation on the track end in some way?

I think having one issue per exercise and then updating it unless it's been closed would be a good default to start with.

[SleeplessByte]

https://github.com/exercism/problem-specifications/issues/1750#issuecomment-750261417 would be perfect for me. Have the same concerns as @wolf99 .

[SaschaMann]

I've started working on a proof of concept here: https://github.com/SaschaMann/probby

[iHiD]

I've unsubscribed from PS (and this issue) as I no longer think it needs my "oversight". If you need me for this, please tag me again :)

[ErikSchierboom]

Let me know if anyone wants a review.