Allow admin to disable BMC SSH, IPMI, and HTTP access

[joseph-reynolds]

Expected Delivery Dates

• [ ] UX: xx/xx/xx
• [ ] Backend: 7/1/2020 estimate
• [ ] FED: xx/xx/xx

Stakeholders

SME: Joseph Reynolds Design Researcher: @ParishrutB @priyanka-pillai97 UX Designer: @ParishrutB @priyanka-pillai97 FED: @dixsie

Use Case

The BMC admin should have an option to disable BMC shell access as a way to ensure the system is managed only by its intended interfaces (like Redfish REST APIs). Security conscious users will want to disable shell access when build the OpenBMC image or when provision their BMC. They require that, for example, to better control and log use of the BMC's management functions, and to pass audits. Use cases are (1) large-scale data centers where uniform access is desired, and (2) systems with sensitive (personal, financial, etc.) data where shell access constitutes a back door into the system.

Specifically, when disabled, secure shell (ssh) access to the BMC (ssh -p 22) will fail. Note that ssh access to the host console (via ssh -p 2200) is not affected by this design.

The admin will be able to re-enable access, allow the BMC shell to be used for some function, debugging, or whatever, and then disable access again. Presumably use of the shell will be a rare event and closely watched to ensure no back doors into the BMC are created.

The BMC admin should be able to log the fact that BMC shell access was disabled or re-enabled. For example, if the design implements the Redfish ManagerNetworkProtocol SSH property (reference below), then Redfish REST API logging would suffice. The BMC admin should also be able to log ssh connection attempts, for example, log files written by the ssh server, PAM, etc.

Requirements

Design

We don't want the GUI to turn this function on or off by accident. My crude GUI design sketch: I envision a new status field on the admin page that shows if "BMC shell access is enabled" (and clearly indicate this feature is separate from the "host console ssh" feature). Maybe have a way to change its state, indicating one of:

• You are about to disable BMC shell access via SSH. No SSH shell connections can be made, but existing connections will still work.
• You are about to enable BMC shell access via SSH. Users will be able to connect to the BMC shell via ssh and attempt to authenticate.

Development

Shell access will remain enabled by default in the current OpenBMC releases.

InVision Prototype

---

Design Issue (phosphor-webui)

---

Development Issue

---

References/Resources

• The Redfish ManagerNetworkProtocol SSH property (https://redfish.dmtf.org/schemas/ManagerNetworkProtocol.v1_4_0.json).

[jk-ozlabs]

We'll need a way to perform debugging in the field, and the functionality there won't be all available over REST. Will there be a way to re-enable ssh for this use-case?

[jk-ozlabs]

Also:

Access to the BMC's shell should be disabled because it offers too much power to users.

This seems to indicate an element of contempt for our users. Should we not give them the facility to perform any sort of automation/interaction method that suits their workflow?

If it's a security issue, then let's look at securing it. If it's warranty issue, I'm sure we can find ways to put safeguards in place. But otherwise, this seems like removing flexibility that some users may rely on.

[joseph-reynolds]

The OpenBMC project direction is to provide a complete set of function via Redfish, including diagnostic functions. My understanding is Redfish provides a forward compatible abstraction layer which frees us to modify the implementation while maintaining compatibility, and that providing shell access to D-Bus and kernel interfaces does not support that direction.

The use case for shell access will remain in the OpenBMC project. This story is for the use case of someone who wants to block shell access to the BMC. We'll have both use cases.

The primary use case for being able to lock out shell access to the BMC is for system owners to ensure the system is managed only by its intended interfaces (like Redfish). They require that, for example, to better control and log use of the BMC's management functions, and to pass audits. Use cases are (1) large-scale data centers where uniform access is desired, and (2) systems with sensitive (personal, financial, etc.) data where shell access constitutes a back door into their data.

Debugging in the field should be a rare event, with the escalation path being:

1. Send in logs.
2. Send in more logs.
3. Try to recreate the problem in the development lab.
4. Create "test fixes" (IBM's old PTF terminology) designed to capture more data.
5. Hop onto the customer system via ssh.

We can implement the Redfish ManagerNetworkProtocol SSH property (https://redfish.dmtf.org/schemas/ManagerNetworkProtocol.v1_4_0.json) to control access to the BMC's shell via ssh -p 22. I feel this would be compatible with the use cases described above because:

1. The fact that the BMC's configuration is changing to allow shell access could be logged (example: sent to an external server) in a way that the shell user could not hide their tracks.
2. When done debugging, removing shell access provides an additional layer of assurance that shell access is disabled. (Example: in addition to changing LDAP permissions, passwords, or ssh certificates.)

[joseph-reynolds]

It didn't come across in the template above, but I meant for this epic to identify all of the functions that need to be in place to support the use cases for disabling BMC shell access. I should edit the description. Thanks for pointing that out.

[jk-ozlabs]

I think it's an admirable goal to have all possible debug functionality exposed over standardised interfaces, but we're never going to be able to anticipate the full spectrum of what we need for all debug scenarios.

As an example, here's what I during the my last debug session (last week), on a system that needed to be in field configuration:

1. Check which versions of crypto libraries were installed on the system
2. Inspect the state of the network stack (using ip)
3. Upload a python script to allow me to capture network packets
4. Run that script several times (while triggering various management actions via standard interfaces), outputting debug data to local files
5. Perform initial analysis on the captured packet data, locally to the BMC
6. Run an editor to implement various packet filtering functions within that script
7. Retrieve captured packet data from the BMC

It's possible that some of these would have been possible through other methods, but I don't think we want facilities for all of those over the standard interfaces.

Additionally, we know that particular issue I have been working here is sensitive to code versions and external network environment, so the option to recreate this in our development environment is not always possible.

This is just one example, which happens to be my most recent.

With that in mind, we're absolutely going to need shell access at some points during field-based debugging, otherwise we're completely reliant on being able to reproduce specific problems in our development environment and/or specific BMC code loads.

[joseph-reynolds]

So we need interfaces to:

1. get the BMC's version,
2. query the BMC's network status, and
3. capture network packets.

That's a good start, thanks!

[mikey]

@joseph-reynolds the point @jk-ozlabs is making is that it's infeasible to add all the functionality that shell access provides for developers. Taking his one use case example and adding that functionality is just the start of playing an endless game of whack-a-mole.

[joseph-reynolds]

Yes I got that, thank you. My previous post was an attempt at very dry humor.

Given that we will have customers who want/need to disable BMC shell access over ssh, I am exploring the options. It seems like:

1. Permanently disabling shell access will limit our ability to service the machine.
2. The customer will want the ability to lock out shell access while the system is operational, re-enable it only in exceptional circumstances, and be able to audit the fact that shell access was re-enabled.
3. Therefore, the Redfish ManagerNetworkProtocol schema seems like the way to go.
4. Over the next 20 years we can add more functions to reduce the need for shell access.
5. (Item 4 was my humor again.)

[jk-ozlabs]

OK, phew. As @mikey says, I don't think the whack-a-mole approach would work; there's just too many corner-cases to cover.

I think your new plan is solid. Having the ability to disable shell access in regular scenarios would be good, as long as we can re-enable without too much disruption to the state of the BMC.

[joseph-reynolds]

I updated the initial comment of this issue to reflect the new design. Thanks for you input!

[amboar]

You are about to disable BMC shell access via SSH. No SSH shell connections can be made, but existing connections will still work. This does not affect "host console ssh". Are you sure? Ok/Cancel. You are about to enable BMC shell access via SSH. Users will be able to connect to the BMC shell via ssh and attempt to authenticate. Are you sure? Ok/Cancel.

I don't know what our GUI design guidelines are like, but Ok/Cancel buttons are sub-par UX IMO. Much better to have the text of the action on the button, e.g. "Disable SSH"/"Leave SSH Enabled" and "Enable SSH"/"Leave SSH Disabled" respectively

[joseph-reynolds]

Sorry. My Web UI design experience was all from the 1980s. Change it however you'd like it. I updated the GUI design to remove the worst parts, and indicate it is merely a design sketch.

[derick-montague]

@joseph-reynolds - The Design Label is meant to indicate that this element is ready for a GUI Designer and GUI is so we know the issue is a GUI issue. Since this is an Epic, I am wondering if this is more than just a GUI design issue.

[joseph-reynolds]

Long lead time item... If we are going to implement the Redfish ManagerNetworkProtocol (https://redfish.dmtf.org/schemas/ManagerNetworkProtocol.v1_4_0.json) to be able to work with "SSH" properties, does that mean we should also implement properties for KVM, VirtualMedia, etc? If so, should we add a field to the Redfish spec for the host serial console (SOL, port 2200)? It doesn't feel like that should be wedged into the Oem field.

[derick-montague]

@nicoleconser @jandraa - FYI - We may want to ask SUs if they see value on this feature also.

[cjengel]

FYI, I concur with Jeremy and Mikey about the need to allow the admin to enable SSH access at any point.

[joseph-reynolds]

There is another reason to disable SSH access. SSH gives an easy way to brick the machine. If the agent who owns the machine is different from the agent who services the machine, then restricting SSH access to the BMC becomes important to the agent who provides service. In this use case, the service agent locks out everyone else from BMC SSH access.

[jk-ozlabs]

Not sure I understand - why would the owner give SSH credentials to an untrusted user (who then bricks the machine)?

[joseph-reynolds]

Thanks for asking. One such scenario is the system owner trusts the BMC admin but has failed to set up adequate training or monitoring programs. (That happens more than you might think.) The BMC admin uses SSH with intentions of improving their workflow, but accidentally causes havoc.

[amboar]

Thanks for asking. One such scenario is the system owner trusts the BMC admin but has failed to set up adequate training or monitoring programs. (That happens more than you might think.) The BMC admin uses SSH with intentions of improving their workflow, but accidentally causes havoc.

That sounds a lot like a technical solution to an education problem :( Isn't the solution better training?

[joseph-reynolds]

My current thinking is two stories for this.

1. One story under this epic would implement the Redfish ManagerNetworkProtocol properties for SSH, IPMI, and HTTPS. There would be no way to control SOL (port 2200). The implementation would tie-in with the underlying systemd configuration, for example, reading the values would query systemd, and setting the values would perform operations like systemctl disable/stop/enable/start ssh.
2. The Web GUI would be another story. Here's a fun consequence: if you used the GUI to disable HTTPS access, your GUI ought to stop working, and the GUI user would have no way to re-enable access. (But a ssh user could reenable access with systemctl commands.)

[joseph-reynolds]

Updated the title to reflect the current design scope.

[joseph-reynolds]

Idea for the GUI: Once the Redfish ManagerNetworkProtocol is implemented, enhance the WEB Gui to complain loudly if IPMI is enabled. See email: https://lists.ozlabs.org/pipermail/openbmc/2019-September/018373.html

[joseph-reynolds]

Similar work is here: per rthomaiy123 IRC discussion:

The srvcfg-mgr will provide D-Bus objects(basically services which we are interested in) and corresponding object properties can be used to enable / disable / change the port id if needed. (Any interface can use the same). Today we control disabling/ enabling phosphor-ipmi-net@, phosphor-ipmi-kcs@, bmcweb, ipkvm, obmc-console. It can be extended as per the need It doesn't require entity manager, as this is independent service, but if service list must be changed based on platform, etc, then can use entity-manager or any other mechanism as desired. Current source is available at https://github.com/Intel-BMC/provingground/tree/master/srvcfg-manager

[derick-montague]

@joseph-reynolds - Dean Sanner brought up the need to disable USB. Wondering if we should track this in this Epic as well. @mzipse - are you aware of this reqeust?

Another eBMC GUI request -- recently for 940 there was requirement where ASMI implemented a USB disable for both FSP and Host USB. Request is to make sure that USB disable is present... and then have a separate disable for Host / BMC.

[joseph-reynolds]

Disabling USB-over-IP could probably be added to the Redfish ManagerNetworkProtocol (if it isn't there already). Disabling the physical USB ports would not be covered by that protocol.

[joseph-reynolds]

The ability to disable (or reenable!) a network interface belongs in the threat model. See more information about OpenBMC’s network interface threat model here: https://github.com/openbmc/docs/blob/master/security/network-security-considerations.md

[joseph-reynolds]

If you bring up the JSON for https://redfish.dmtf.org/schemas/ManagerNetworkProtocol.v1_4_0.json you can see the possible BMC network interfaces under: definitions > ManagerNetworkProtocol > properties. The ones that apply to OpenBMC include: HTTP, HTTPS, IPMI, KVMIP, SSH, etc.

[joseph-reynolds]

This ought to be able to disable the BMC's Avahi service so a security conscious BMC admin can control what information about their server gets advertised. I don't see Avahi in the Redfish ManagerNetworkProtocol spec v1.4.0. Can it be added?

[joseph-reynolds]

To clarify, I can foresee a customer who wants to block SoL (via SSH port 2200) access. I think we should pursue adding that to the Redfish ManagerNetworkProtocol spec. I believe this may be a long lead time item (as in, we want the spec updated and published before we implement).

[rfrandse]

refresh

[rfrandse]

refresh

[rfrandse]

refresh

[rfrandse]

refresh

[rfrandse]

refresh

[rfrandse]

refresh

[joseph-reynolds]

Discussion on email list: https://lists.ozlabs.org/pipermail/openbmc/2019-December/019864.html

[joseph-reynolds]

I pushed an gerrit review for an overview of the BMC's interfaces. It include the interfaces we are talking about disabling. -- https://gerrit.openbmc-project.xyz/c/openbmc/docs/+/27969

[rfrandse]

refresh

[joseph-reynolds]

Here is a reference to an auditing design which supports the use case to disable the ssh acccess. The design doc (in review) states (in part):

This design assumes that an end user is never given a direct access to the system shell. The shell allows for direct manipulation of user database (add/remove users, change passwords) and system configuration (network scripts, etc.), and it doesn't seem feasible to track such user actions taken within the shell. This design assumes that all user interaction with OpenBMC is limited to controlled interfaces served by other Phosphor OpenBMC components interacting via D-Bus.

https://gerrit.openbmc-project.xyz/c/openbmc/docs/+/23870/11/designs/phosphor-audit.md#40

[rfrandse]

refresh

[rfrandse]

refresh

[rfrandse]

refresh

[rfrandse]

refresh

[joseph-reynolds]

The CSIS paper > BMC section (https://github.com/opencomputeproject/Security/blob/master/SecureFirmwareDevelopmentBestPractices.md) recommends being able to disable these interfaces:

• Telnet
• SSH
• Insecure Web (80)
• Media Management Services used to attach and detach host media (platform specific)
• IPMI

OpenBMC does not offer TELNET or Insecure Web (port 80).

The list I propose for OpenBMC is below. It matches with the CSIS paper pretty well.

• SSH
• network IPMI
• mDNS (avahi)
• Remote USB / Virtual media
• KVMIP

[rfrandse]

refresh

[joseph-reynolds]

I asked the Redfish forum if we can add mDNS (Avahi) alongside SSDP. As far as I know, OpenBMC prefers to use mDNS/Avahi. Do we go to Redfish with that?

[rfrandse]

refresh

[rfrandse]

refresh

[joseph-reynolds]

Adding another interface we can enable and disable:

• BMC-attached USB