batches: automatically configure webhooks wherever possible

[LawnGnome]

One of our biggest issues in Batch Changes is getting site admins to enable and configure webhooks. It's painful to do, since it has to be per-repo or (at best) per-org depending on the code host, and yet it's basically the difference between Batch Changes being usable and not much of the time. Anecdotally, it feels like we've seen an uptick in webhook-related support issues of late.

Our longer term plans for dealing with this have tended to be either eventually writing a GitHub App to fix this, or hoping that site-wide webhooks will do the job (which we've recently discovered they likely won't, because they don't get down to this level of granularity).

Meanwhile, it turns out that we got a rudimentary background worker to configure webhooks automatically in #39992 and #39833. (Who knew? Certainly not me. I stumbled across this while starting to build the same thing on a rainy Friday afternoon.)

This would be extremely useful for Batch Changes!

Unfortunately, it has a few drawbacks right now:

1. It only supports GitHub.
2. It only adds the push event type, which is useful for repo updates, but not for Batch Changes.
3. It attempts to write the webhook configuration back to the external service configuration, which will fail if an overlay file is in use.
4. There are some things hard coded in github.V3Client that probably shouldn't be, like the event types and the string to search for to consider a particular webhook a "Sourcegraph webhook".
5. It's off by default, and turning it on is hidden behind an experimentalFeatures site configuration flag.

So, let's make it better so we can turn it on by default and our users can enjoy the benefits of having webhooks Just Work.

Rough plan of attack for GitHub:

• [ ] Move the worker bits out of internal/repos and into cmd/worker/internal. (This may include giving the job a slightly more descriptive name, since it's currently "webhook build" and that doesn't really describe what it actually does.) 1d
• [ ] Move secret storage for automatically configured webhooks out of the external service configuration and into something linked to repo. (This needs to be encrypted if encryption is enabled.) 2d
• [ ] Refactor the webhook methods in github.V3Client to be less opinionated. 1d, assuming the VCR tests aren't too bad
• [ ] Have the worker handler also add the Batch Changes event types if Batch Changes is enabled. (Or just do it in general.) 0.5d
• [ ] Stop hiding this as an experimental feature in the site configuration, and make it something that can be opted into within the external service configuration, preferably either on a per-org/repo or global basis according to the site admin's desires. (This needs to include documenting that the external service token needs the right scopes to be able to configure webhooks for this to work.) 1d

Future incremental work:

• [ ] Add support for other code hosts. (As part of this, the GitHub specific bits of the worker should probably go somewhere else — I'm thinking a new internal/webhooks/inbound/{code host name} package structure, since I'm planning internal/webhooks/outbound for #38278.) dunno; probably something like 2d per code host kind

Nice to haves:

• We could — and probably should, in the longer term — configure webhooks at the organisation level if possible, rather than the repo level. This is hard to do today, though, because we stream repos rather than syncing an external service as a whole where we could more easily aggregate the repos.

[chrispine]

@malomarrec Thoughts on the priority of webhook related work? Feels like it comes after strategic readiness, so backlog, but wanted to hear what you thought.

[courier-new]

There's some active conversation around webhooks that Ryan Slade's driving here that may influence who/how/when we pick this up. I'd actually argue adequate automation around webhooks is incredibly important to strategic customers, for 2 reasons:

• We all agree setting up webhooks is one of the MOST important things for Batch Changes customers to do, because without them their changesets don't sync reliably and they lose trust in the product.
• Without some sort of automation, it's up to an admin to manually go through and create individual webhooks for every single repo/org they care about using with Batch Changes, which quickly becomes unreasonable.

[courier-new]

This is also effectively tackling the same problem and serving as a direct complement to https://github.com/sourcegraph/sourcegraph/issues/24312, which has been a long-standing fave item on our board.

[LawnGnome]

Yeah, I'm going to take this off the board for now because Ryan and Alex are basically working on this already. This may come back later as a more tightly scoped ticket if I end up pitching in to help after outbound webhooks.