syz-ci: umbrella for v2

[dvyukov]

While syz-ci served us well for several years now, the overall architecture is quite simplistic and ad-hoc. At some point we probably need to do major rework of syz-ci implementation. This is umbrella bug to collect things we want to improve.

• better transparency and control, i.e. some web interface with current state of things, logs and maybe some actions (rebuild kernel, force rebuild kernel, update syzkaller, etc)
• single beefy instance managing a dozen of syz-manager's is somewhat painful (restarts take too long; if target build for one OS is broke, can't update other OSes), it would be nice to mostly detach all managers from each other (restart/update just one)
• proper multi-OS support (support multiple SOURCEDIR's for different OSes and build target using the new kernel's toolchain for OSes that use kernel toolchain (netbsd, fuchsia, akaros))
• don't file a dozen of bugs if syzkaller is build/test broken
• patch testing should take stronger precedence over bisection (some bisections take days); however, we try to not run more than 1 kernel build at a time and we may not have that many additional VMs for both bisection and patch testing; so we need to somehow "pause" bisection (?)
• better isolation/recreation/deployment, probably containers should be part of the story (however, unclear what we do for host machines that don't run linux)
• move more configs to dashboard; it would be nice if syz-ci would not need a separate config that needs to be updates separately; namely, syz-ci connects to dashboard and says "I am instance X", dashboard instructs it to run a set of instances in particular configuration; then we will only have (and need to update) only single config; however, do we want to keep the ability to run a standalone instance?
• we don't preserve kernel source for the "latest" build, get_maintainers and coverage reports just use whatever is in the kernel dir (which is a wrong version); this can lead either to inconsistencies, or even errors (e.g. coverage report tries to read file that does not exist anymore). Maybe we could create a "snapshot" of the kernel sources with git worktree command.

Also existing syz-ci issues:

• syz-ci: stop old manager before uploaded new build (#1329)
• syz-ci: use a single check out per OS (#1235)
• syz-ci: do regression testing (#884)
• syz-ci: don't rebuild broken trees (#576)
• syz-ci: exit instead of re-exec for update (#2985)

[a-nogikh]

patch testing should take stronger precedence over bisection (some bisections take days); however, we try to not run more than 1 kernel build at a time and we may not have that many additional VMs for both bisection and patch testing; so we need to somehow "pause" bisection (?)

Should be covered by https://github.com/google/syzkaller/pull/3607