NVMe drive temp sensor fan control loop integration support

[derekhoward55]

NVMe drive temps. On nvme backplane, 0-3 in system. Drives are hot-pluggable. The drives are behind mux on optional JBOF pcie slot cable card (Flett, 0-3). Entity manager should support this (presence, going through mux to get readings, showing up when hot-plugged, etc).

Quanta (https://github.com/openbmc/docs/blob/master/designs/nvmemi_over_SMbus.md) is writing nvme interface device driver (using direct i2c) and the BMC code that puts the nvme drive temps and info on dbus (phosphor_nvme). Ensure temps get from device/dd into dbus, where they will be read by fan control app. It does not differentiate not present from i2c failures, this will need to be updated as the system fan control needs to be different for these two scenarios.

Alternatively, Intel recently put up "nvmesensor" app that hopefully can be used to read the temps via the full nvme interface (https://github.com/openbmc/dbus-sensors/tree/b669b6b54b1bfa499380e518bf508b672dd9a125/src). However some of the needed support has not been upstreamed (libmctp), so that would need to be upstreamed or written ourselves.

If can't use Intel/Quanta support, could get with "register" common support. This is supported in FSP systems today via straight i2c register read. Spec at https://www.nvmexpress.org/wp-content/uploads/NVMe_Management_-_Technical_Note_on_Basic_Management_Command.pdf. Then later switch over to Intel/Quanta support depending on quality/timing.

If BMC can't determine presence: PHYP sends device/vendor ids to bmc with new pldm cmd or in pldm PDRs/fru records (since neither phyp nor bmc plans to read nvme drive vpd/ccin, nvme drive vpd not on sb power(?), unsure if ccin is equivalent to device ids, etc). BMC has table which associates device id and slot (and ccin for future usage) with i2c address to read, tcontrols, fan floor setting etc. Then do direct (non-dd) i2c read of the register (or get via Intel/Quanta app) if present (pgood good mapped into presence status in hw) and use in fan control algorithm.

Help add temp limits, floor, other info to json, figure out i2c bus/etc.

[amboar]

@derekhoward55 the nvmemi-over-smbus link is broken - it lives here: https://github.com/openbmc/docs/blob/master/designs/nvmemi-over-smbus.md

Also, for the record, "phosphor_nvme" is https://github.com/openbmc/phosphor-nvme

is writing nvme interface device driver (using direct i2c) and the BMC code that puts the nvme drive temps and info on dbus (phosphor_nvme). Ensure temps get from device/dd into dbus, where they will be read by fan control app.

To me "device driver" implies kernel work, however there's no mention of this in Quanta's design document; it looks like they just planned to send smbus commands from userspace.

However some of the needed support has not been upstreamed (libmctp), so that would need to be upstreamed or written ourselves.

As the libmctp maintainer, can you elaborate on this? What needs to be done?

If can't use Intel/Quanta support, could get with "register" common support. This is supported in FSP systems today via straight i2c register read. Spec at https://www.nvmexpress.org/wp-content/uploads/NVMe_Management_-_Technical_Note_on_Basic_Management_Command.pdf. Then later switch over to Intel/Quanta support depending on quality/timing.

How is this approach different from what Quanta proposed?

[amboar]

However some of the needed support has not been upstreamed (libmctp), so that would need to be upstreamed or written ourselves.

As the libmctp maintainer, can you elaborate on this? What needs to be done?

Okay I looked, and now I wish I hadn't.

Quanta's plan looks pretty well formed and avoids the downstream-only maze of MCTP support required by nvmesensor.

[derekhoward55]

@derekhoward55 the nvmemi-over-smbus link is broken - it lives here: https://github.com/openbmc/docs/blob/master/designs/nvmemi-over-smbus.md

Also, for the record, "phosphor_nvme" is https://github.com/openbmc/phosphor-nvme

is writing nvme interface device driver (using direct i2c) and the BMC code that puts the nvme drive temps and info on dbus (phosphor_nvme). Ensure temps get from device/dd into dbus, where they will be read by fan control app.

To me "device driver" implies kernel work, however there's no mention of this in Quanta's design document; it looks like they just planned to send smbus commands from userspace.

Sounds good. This story was written before Quantas support was added and the exact details of their support was known.

However some of the needed support has not been upstreamed (libmctp), so that would need to be upstreamed or written ourselves.

As the libmctp maintainer, can you elaborate on this? What needs to be done?

Looks like you dug into this a little, but this is based on @bradbishop very helpful investigation of their support.

If can't use Intel/Quanta support, could get with "register" common support. This is supported in FSP systems today via straight i2c register read. Spec at https://www.nvmexpress.org/wp-content/uploads/NVMe_Management_-_Technical_Note_on_Basic_Management_Command.pdf. Then later switch over to Intel/Quanta support depending on quality/timing.

How is this approach different from what Quanta proposed?

When this story was written the exact details of Quanta's support was not known; it was assumed they would use the nvme interface and not just direct i2c. Perhaps at this point there is not a difference, and utilizing Quantas support is basically the same as doing ourselves.

Thanks for digging into it.

[amboar]

I've done a bunch of work on this and we are at least exposing drive temperatures to be consumed by fan control. There are a bunch of problems that still need to be solved, but we're making progress.

[msbarth]

@amboar Where do we stand on getting these last "bunch of problems" resolved? Fan control is utilizing the temperatures but with early delivery of full function fan control, are there areas that need to still be worked so fan can react appropriately?