Closed mkeeter closed 4 days ago
Is it possible to have the SP require some kind of asymmetric key based authentication for control plane agent messages? Presumably we're keen to do that anyway.
Is it possible to have the SP require some kind of asymmetric key based authentication for control plane agent messages? Presumably we're keen to do that anyway.
That (or something like it) would be https://github.com/oxidecomputer/hubris/issues/723 / https://github.com/oxidecomputer/management-gateway-service/issues/152. Definitely something we want, not something we've designed or are actively working on at the moment.
(This step would only be an enhancement to a proper solution.) Only part of a solution since it limits or impedes some access, but TP connected to only a Sidecar sounds good. The rack is not going anywhere without Sidecar and though tedious, sleds can be swapped into a scrimlet position if it comes to that.
I guess the benefit of only being able to access the sidecar is that, at least in theory, we'd notice if someone then tried to access anything else? (Because they would have had to replace the software in the sidecar to do so)
TP connected only to the sidecar would only give us some recovery possibilities. If there's a bug in sled-agent that prevents the switch zone from coming up (as we had a week ago), access to the sidecar can't help us; we need the console proxy to the scrimlet's SP to recover, and swapping scrimlets might or might not help. I guess we could ourselves do the unfortunate thing in this case and replace the sidecar SP image with one that does let us access the scrimlet SP?
It sounds like we need a "rack provisioning mode" and/or a "rack debug mode". The provisioning mode would be the default delivery mode. Once provisioned sufficiently in the DC, it would switch to production mode with a highly restricted if not disabled TP. Transition out of production mode would have to be through the normal UI with proper authorization and may remove some access to production data. TP in production mode might still have access to basic monitoring and identification if it can be secured.
So, some very complex flows are possible, but what is the MVP for having a secure TP when the customer has production workloads running?
TP connected only to the sidecar would only give us some recovery possibilities. If there's a bug in sled-agent that prevents the switch zone from coming up (as we had a week ago), access to the sidecar can't help us; we need the console proxy to the scrimlet's SP to recover, and swapping scrimlets might or might not help. I guess we could ourselves do the unfortunate thing in this case and replace the sidecar SP image with one that does let us access the scrimlet SP?
Yeah, I guess to be clear, when I was asking about the benefit, I meant: what is the benefit of making any change at all right now, if we're basically leaving the access mechanism totally open to someone who can get and apply new sidecar SP software (which, it must be said, is anyone, as it's open source!). If the benefit is "well, at least someone would have to replace the software" rather than "well, someone could get in without even having to do that", I suppose that's basically fine -- it's at least a potential change that the control plane or an operator could notice is occurring.
Done in #1859
Right now, the technician port (TP) is on a VLAN that grants it access to every SP (see
configure_vlan_semistrict
).This is intentional: without this access, the only way to an SP is TP → Monorail → Tofino → Scrimlet → Tofino → Monorail → SP. If we don't have confidence that the Tofino + Scrimlet will come up with 100% reliability (and we don't!), a TP → SP path is necessary for system recovery.
However, it means that many of our security properties don't hold against an attacker with access to the TP:
Of course, anyone with physical access to the rack could – instead of plugging into the TP – just remove a Gimlet and reflash the SP firmware with a programmer. Still, it's not great that TP → SP security is notably weaker than our TP → Scrimlet requirements, which use a special support Yubikey to authenticate the user.
So, what's the right policy here?
sled-agent
bugs to make this path inaccessible, which happened literally last week