x/build/cmd/relui: create proof of concept for release automation

toothrot commented 4 years ago

As part of an effort to improve the reliability and reduce the complexity of Go releases, we should automate more of the release process.

A possible solution is a hosted release management tool (relui) that is responsible for the scheduling and operating of the release, much as releasebot handles this process on the CLI. We can improve the observability of the process by using a persistent web UI for coordinating releases.

As a proof of concept, relui acts as the coordinator (and currently the task executor) for release tasks.

Rough Architecture

For the proof of concept, there will be a single application for both starting and running workflows. However, careful API boundaries should be maintained between the scheduler process and the runner processes to allow for separation in the future.

In general, the runner process should only know how to execute specific tasks without broader context, and report back to the scheduler. Tasks should have an associated type, and the runner should decide whether or not it has an implementation for a given task type.

Tasks, or BuildableTasks, have a shared configuration. All tasks have a type, a name, an optional task name to depend on, and an artifact URL. If a task depends on another named task, it will be provided with that task's artifact URL upon starting.

Finally, workflows should be described through a configuration, in order to allow us to share steps between workflows, and separate concerns between the implementation and the definition of all steps that must be completed for a release. This is especially important for steps that may need to run on different platforms (outside of our tooling on GCP, such as the signing process).

Rough initial workflow for a local workstation release:

Clone repo @ ref In:

Repo
Ref

Out:

Tarball of Go src
Tarball URI

Run make.bash In:

Tarball URI of Go Src Out:
Tarball URI of Go src after make

Build Race Detector In:

Tarball URI of Go src after make Out:
Tarball URI of Go src after race build

Clean (version.cache, pkg/bootstrap, race for other GOOS/GOARCH, pkg/GOOS_GOOARCH/cmd) In:

Tarball URI of Go src after race build Out:
Tarball URI of Go src after cleanup

Run all.bash In:

Tarball URI of Go Src after cleanup Out:
Tarball URI of Go src after all.bash

Finalize In:

Tarball URI of Go Src after cleanup (not all.bash) Out:
Tarball URI of binary release

Tasks (remaining as of 2020-10-05)

[ ] Subscribe to a PubSub Topic
[ ] Publish status upon accepting a task
[ ] Run FetchGitSource task
[ ] Create Status API
[ ] Datastore Integration

relworker

(bootstrap worker) Subscribe to a PubSub Topic

A worker can connect to PubSub and subscribe to the configured topic. Messages should be subscribed to using the Receive API, which handles spawning goroutines for handling the message, as well as auto-extending the Ack deadline while a message is being processed.

For now, we can just log that we got the message,can handle it, and Ack it.

See: https://pkg.go.dev/cloud.google.com/go/pubsub#Subscription.Receive

Publish status on accepting a task

When the worker picks up a task, it should update the status of the task in relui as started.

Run FetchGitSource task

The FetchGitSource task should fetch the specified Git repo at the configured ref. The source should be tarred, record the artifact URL to relui, and mark the task as complete.

On gitiles, there is an +archive URL for this task:

Web: https://go.googlesource.com/go/+archive/refs/heads/master

Archive: https://go.googlesource.com/go/+archive/refs/heads/master.tar.gz

Handle non-transient errors on FetchGitSource

If a permanent error occurs when executing a task, the message should be ACK’d to prevent retries, and a terminal status for the task should be reported back to relui.

relui

Create Status API (gRPC server)

A worker can communicate the status of a task back to the coordinator as it progresses on a task. The initial API should at least be able to mark a task as started.

Host gRPC and HTTPS on the same port (with some caveats)
1. See caveats here: https://github.com/grpc/grpc-go/issues/2662#issuecomment-468497692
Host gRPC on same service but different port as HTTPS relui web
Use separate instances of relui for gRPC

Datastore integration

Currently, relui commits state to disk. It should have a persistent database for handling multiple instances, like Datastore.

/cc @dmitshur @cagedmantis @andybons

gopherbot commented 4 years ago

Change https://golang.org/cl/243340 mentions this issue: cmd/relui: add proto definition for workflows

gopherbot commented 4 years ago

Change https://golang.org/cl/243339 mentions this issue: cmd/relui: enable creation of mock workflow

gopherbot commented 4 years ago

Change https://golang.org/cl/243338 mentions this issue: cmd/relui: add relui webserver for releases

gopherbot commented 4 years ago

Change https://golang.org/cl/245277 mentions this issue: maintner/maintnerd/maintapi/version: support beta and RC release tags

gopherbot commented 4 years ago

Change https://golang.org/cl/245645 mentions this issue: cmd/relui: add package comment

gopherbot commented 4 years ago

Change https://golang.org/cl/246298 mentions this issue: cmd/relui: add persistence to development database

gopherbot commented 4 years ago

Change https://golang.org/cl/250917 mentions this issue: cmd/relui: load development state on boot

gopherbot commented 4 years ago

Change https://golang.org/cl/257237 mentions this issue: cmd/relui: create pubsub topic on start

gopherbot commented 4 years ago

Change https://golang.org/cl/257239 mentions this issue: cmd/relui: publish task start message to pubsub

gopherbot commented 4 years ago

Change https://golang.org/cl/257238 mentions this issue: cmd/relui: generate uuid for workflow and task instances

toothrot commented 4 years ago

The next step before moving forward needs to be guarding routes for specific HTTP methods. Capturing this from a comment in https://golang.org/cl/257239.