update to Cockroach 22.2.0

[davepacheco]

Cockroach have just released v22.2.0 with Go 1.19.1. It'd be good to update to this as soon as convenient.

• [ ] update garbage-compactor to build v22.2.0
  • [ ] need Go 1.19.1 IPS package
  • [ ] need newer Node IPS package (>=16.10)
• [ ] update omicron to use the v22.2.0 build

[ahl]

Do we want to consider running CRDB within a Linux VM?

[jclulow]

No I don't think so. From my perspective the original reasons still apply:

• there would be no DTrace, and even if there was, there would be a virtualisation barrier between the workload and the host
• we have no nested virtualisation so there would be no way to test the control plane under Propolis as we can do today
• we'd have to select and maintain a Linux distribution and do the release engineering for that in addition to everything else
• we'd have to come up with a different way to configure it; we're doing self assembling zones configured through injected SMF profiles now, so we'd have to look at doing something with hypervisor metadata and something like cloud-init, and we'd need some other way to reach into the guest to determine health of services and inject updated configuration etc

[jclulow]

NB: wrt. the dependencies, Node and Go are already maintained by other people (e.g., pkgsrc and OmniOS both ship up to date versions, and we pull updates from OmniOS into Helios) it's "just" the actual Cockroach build that needs the bigger work.

[davepacheco]

I don't think so. I don't think much about the original calculus has changed.

There's a bunch of stuff that we get "for free" by running this as a Helios zone that we'd have to re-implement for guest VMs:

• Building the image itself. Best case is that we could take some existing distro image as-is. It seems likely to me we'd need to customize it. How would we do that?
• Building our own software to run in that image. (Example: we have Rust programs to start up Cockroach and configure it the way we want, which currently involves querying DNS but will probably eventually involve querying Nexus, too.) Right now, we build everything for production on Helios and generate Helios binaries. We'd presumably need a separate build process that generates some Linux binaries (and runs in a separate build environment? maybe not if we cross-compile?).
• Distributing and deploying the VM: there's a bunch of stuff in sled agent (and there will be a bunch of stuff in the upgrade system) for deploying an image (and distributing it in the first place). We'll need to build parallel stuff for deploying a VM instead.
• Operating a Linux guest: operating a database, we'll definitely wind up with nasty issues to debug; we'll need to maintain expertise and ensure we have the tools we need to debug these.
• Also part of operating a guest: facilities for support -- like being able to view the CockroachDB logs or the logs for our own start method.

To be clear, we need all of this for whatever environment we deploy our software in. It's just that we're already doing all of this for Helios zones. It'd be quite a lot of work to build parallel infrastructure for Linux VMs.

What have been / will be the costs to running on Helios?

1. Some amount of work to keep CockroachDB building on Helios. Modulo Bazel (see below), I think this is fairly small (certainly much smaller than the above). To get a sense of our local delta, you can see our local patches here.
2. The cost of debugging illumos#15254 and friends. This was considerable. But I'm not sure what conclusion to draw from it. First, it was a general OS problem, and we still need the OS to work, so it's investment in more than just CockroachDB. Also, Linux had the same kind of problem; it's just that it was found and fixed by others already. That's a potential advantage in itself. Of course, if we had a similarly nasty problem again, on Linux I think we've got a lot less expertise and tooling to debug it effectively.
3. The Bazel thing. I expect this to be largely a one-time cost (though there may be some ongoing maintenance too). We don't know how big yet. I would be surprised if it were even close to as large as the above. There would be ongoing maintenance either way (for our parallel infrastructure or for the illumos support in Bazel). Again, my guess is Bazel will be easier here because it's pretty separate from everything else. Just the sled agent support for deploying zones alone has been a tricky area with lots of churn; I can't imagine also maintaining a version of that for VMs as we keep iterating on it.

Basically, I think switching would be a ton of up-front work and a bunch of ongoing work that the whole team would keep having to deal with (i.e., everyone will have to deal with the VM and the infrastructure around it); whereas I think (but don't know!) that doing the Bazel work would be much less work up front, less ongoing, and only the people keeping it up would have to deal with it.

[citrus-it]

An illumos port of Bazel recently landed in OmniOS. I've opened a PR to pull this into Helios at https://github.com/oxidecomputer/helios-omnios-extra/pull/3 More details over there.

[iliana]

Moving to the 8 milestone aspirationally; it's likely if we want to eventually get to Cockroach v23.2 we'll want to go one release at a time, and it'd be best to do that sooner rather than later.

Fortunately it seems to build now, some notes at https://github.com/oxidecomputer/garbage-compactor/pull/7#issuecomment-2038998312. I've uploaded my build to /staff/iliana/cockroach-v22.2.19.illumos.tar.gz.

However there are a significant number of test failures after introducing v22.2, caused by trying to set the first blueprint as the target blueprint. The blueprints InsertTargetQuery CTE is returning the no-such-blueprint sentinel for reasons I can't figure out quite yet. (cargo nextest run -p nexus-db-queries -- db::datastore::deployment::tests::test_representative_blueprint is the most trivial way I've found to reproduce this.)

thread 'db::datastore::deployment::tests::test_representative_blueprint' panicked at nexus/db-queries/src/db/datastore/deployment.rs:1457:14:
called `Result::unwrap()` on an `Err` value: ObjectNotFound { type_name: Blueprint, lookup_type: ById(5234919e-1005-441a-9575-614305173dbe) }

[iliana]

5434 fixes the issue I mention above. All tests pass on v22.2.19 with that change.