Sled Agent must manage durable storage for configs, zones, explicitly

[smklein]

See also: RFD 118

Sled Agent currently configures a few pieces of data outside datasets explicitly allocated in pools:

• /var/oxide contains a variety of configuration information, including...
  • RSS setup information
  • The "Sled Request" used to launch the sled-agent (including underlay information)
  • A list of "All services which should be launched on this sled"
  • A list of "All services which should be launched on u.2 zpools"
• /zone contains the "filesystem for all zones"
• /opt/oxide contains all the "latest installed system images", and is used to update control plane software which exists outside the ramdisk.

Q: So, why is this bad? A: All those paths are currently backed by a ramdisk -- specifically rpool -- on gimlets.

This means that when we reboot, a significant portion of the necessary configuration information to launch the sled will be lost. Furthermore, for the zonepath filesystems, a significant portion of user RAM will be dedicated to zone-based filesystems, which we'd prefer to distribute to disk-backed file storage.

Here's a list of some of the work we need to accomplish to mitigate this in a production environment:

• [ ] Sled agent should be responsible for formatting U.2s and M.2s with necessary datasets
  • [ ] See: https://github.com/oxidecomputer/omicron/issues/2688
• [x] #2998
• [x] #2970
• [x] #2969
  • [ ] This would be made easier with https://github.com/oxidecomputer/omicron/issues/1898 , as we'd only need to store the zone information, and not the corresponding config for "how to launch the zone".
• [ ] Sled agent should allocate zonepaths from U.2s
  • [ ] This means sled agent needs to handle the allocation / de-allocation of zonepaths much more explicitly.
  • [ ] Related: https://github.com/oxidecomputer/omicron/issues/1119 , https://github.com/oxidecomputer/omicron/issues/1313
• [x] #2971

[andrewjstone]

I just remembered that https://github.com/oxidecomputer/omicron/pull/3007 adds some support for persisting zone filesystems under crypt/zone.

This code has been ready to go for a few weeks now and has been tested numerous times. I've done cold boot testing on paris with it, but since there is no persistent install of software on the M.2s like dogfood, I have to re-run omicron-package install after reboot to rediscover and mount the U.2 encrypted datasets. The PR has the details about this. While I'd like to test with a proper install on dogfood I"m at about the point where I'd like to just merge this and see what happens.

[smklein]

FYI: I've been punting this a bit, because we clear these zones out on reboot, and the system is functional with self-managed storage of these zones. It's possible to put Nexus more explicitly in control of this "zone filesystem management", but not urgent.