Power Control Service

[nieuwsma]

Abstract

CAPMC is going to be replaced with the Power Control Service (PCS). PCS will be a RESTful microservice (CAPMC is not very restful), PCS will have a smaller, better aligned portfolio of responsibilities.

Problem Statement

CAPMC is a low level hardware abstraction service. It offers fire-and-forget style control for power control and many times leaves the caller 'hanging' wondering why or if their power control command worked (because hardware is inherently unstable).

There are several core reasons why we are re-working our approach to power control.

architectural alignment (microservice)

Cray Systems Management has a much more solidified Shasta architecture and we have much more mature set of capabilities for developing RESTful microservices. CAPMC is very monolithic and has a very large portfolio. As the Shasta product has many capabilities already provided across the platform (e.g. telemetry via SMF, or p-state control on node) it makes sense to leverage those capabilities where they are most mature.

core functionality (resolving power states)

Perhaps the most common complaint of CAPMC is that when someone uses it to power on/off a node that CAPMC 'tries' and gives up; often times without returning useful information or cryptic information at best. CAPMC is a hardware abstraction but purposefully does not cope with the reliability levels of the hardware. A try-once-and-die model will not scale as we approach exascale and wont help us meet our SLOs (system level objectives) around booting. CAPMC cannot be modified to better cope with this reality of hardware as this was the intentional paradigm of the API.

CAPMC is approximately (with regards to powering on and off of devices) about one step more advanced and abstracted than just using redfish. Meaning that it 'abstracts' most of the uniquenesses of the individual devices and can cope to some degree with their irregularity. A higher level of abstraction will further separate the caller from redfish and the 'base' idea of just turning on or off a device, but working towards guaranteeing that a device experiences its intended power transition. An even higher level of abstraction would introduce the idea of coordination across devices (orchestration).

towards 'ramp rate limiting'

A feature that Shasta CAPMC does not yet have, is some notion of 'ramp rate limiting'. Im layman's terms, controlling the amount of devices you power on (or off) to not exceed a rate of change. This is important for customers in power sensitive markets and can many times be tied to system acceptance. PCS will need to share state across instances to know how many devices are being powered up at a time to stay within an envelop. This feature set is not trivial and the programming necessary is not very doable given the level of abstraction that CAPMC offers from Redfish.

maintainability

Shasta CAPMC does not have a well layered design. The design is very vertically siloed. This makes resolving power states and working towards 'ramp rate limiting' very difficult. The code has fairly mature unit testing, but because of the lack of horizontal break points in the design any testing requires mocking a full http stack for every call (more or less). This means that unit tests aren't testing 'units' of code but more 'features'.

Internal References

• PCS Docs
• swagger.yaml
• Jira EPIC

External References

This section left empty on purpose

Proposed Solution(s)

1. Create PCS (API and CLI)
2. Deprecate CAPMC (following the policy). CAPMC will be converted to call PCS under the hood until its death
3. update cray internal services to use PCS: BOS, FAS, SAT, etc.

Shasta CAPMC had a lot of features that PCS will not recreate or carry on. They will be discontinued from the 'power control' portfolio because there are other more appropriate ways to engage with the resources.

The features of the power control service are limited to (As of 2021Q3 plan):

• powering of devices (transitions)
• reporting power status of devices (power-status)
• power capping (power-cap)

The features that will not be part of PCS that were originally part of Shasta CAPMC:

• telemetry
• p-state / c-state control
• Emergency Power Off

Impact of Action/Inaction

We will not be able to address the customer requirement of ramp rate limiting and will not be able to solve the internal RFE of 'guarantee'ing that a power transition occurs. CAPMC is fire-and-forget. PCS is much more robust by design. CAPMC is also very monolithic and not very extensible or testable given its code line organization.

Further Information

CAPMC Paper 2014

Suggested Reviewers

- [ ] @mpkelly-hpe - [ ] @anieuwsma-hpe _- [ ] @jsollom-hpe

Comment Period

Comment period for this proposal shall close on [[June 25, 2021]]. (well really at the end of the tech leads meeting on the 24th)

[jsollom-hpe]

That guarantee of quality of service is what BOS is after. BOS wants standardized, usable error codes. BOS wants to be agnostic about the hardware, as agnostic as it can be.

BOS doesn’t care about ramp rate limiting. However, as an electrical engineer and the son of an electrical engineer who worked for a power company, I understand its importance.

You could list explicitly what other services supply the areas that PCS is not taking over from CAPMC. What is handling emergency power off? I don't even know what that is or what would trigger it.

[nieuwsma]

You could list explicitly what other services supply the areas that PCS is not taking over from CAPMC. What is handling emergency power off? I don't even know what that is or what would trigger it.

With regards to 'whats replacing what' read:

https://stash.us.cray.com/projects/HMS/repos/hms-power-control/browse/docs/design/CAPMC_vs_PCS?at=EPO_NodeRules

With regards to EPO please read:
https://stash.us.cray.com/projects/HMS/repos/hms-power-control/browse/docs/design/CAPMC_vs_PCS/EmergencyPowerOff.md?at=refs%2Fheads%2FEPO_NodeRules

[SeanWallace]

Overall PCS sounds good to me. I wonder about this however:

CAPMC will be converted to call PCS under the hood until its death

Is this going to be fully possible? You spent a good amount of the problem statement describing the inefficiencies and fire-and-forget nature of the CAPMC API to me suggesting (I haven't poured over the Swagger) the PCS API will function much differently. So is it reasonably possible to make CAPMC behave the same while using PCS? Perhaps said more simply, how do you feel like the user experience of CAPMC will compare if it is to be converted to use PCS? Close enough to not matter?

Along these same lines, normally I feel like it's kind of a silly argument to worry about what happens if CAPMC and PCS try to do competing activities but in this case I really do wonder if that will be a problem. Until PCS is available CAPMC will remain the only higher-level utility for power control, and it's certainly widely used directly by administrators today. Would you say the threat of these two stepping on one another is worth worrying about, or will there be mitigations in place sufficient enough to prevent "very bad things"?

[nieuwsma]

Overall PCS sounds good to me. I wonder about this however:

CAPMC will be converted to call PCS under the hood until its death

Is this going to be fully possible? You spent a good amount of the problem statement describing the inefficiencies and fire-and-forget nature of the CAPMC API to me suggesting (I haven't poured over the Swagger) the PCS API will function much differently. So is it reasonably possible to make CAPMC behave the same while using PCS? Perhaps said more simply, how do you feel like the user experience of CAPMC will compare if it is to be converted to use PCS? Close enough to not matter?

I think so. CAPMC is blocking, so it would be converted to just check on PCS. We only have to convert for the calls that do something (namely transitions, maybe power capping, not power status).

Along these same lines, normally I feel like it's kind of a silly argument to worry about what happens if CAPMC and PCS try to do competing activities but in this case I really do wonder if that will be a problem. Until PCS is available CAPMC will remain the only higher-level utility for power control, and it's certainly widely used directly by administrators today. Would you say the threat of these two stepping on one another is worth worrying about, or will there be mitigations in place sufficient enough to prevent "very bad things"?

Yes it is a big threat. The key to success is PCS uses the reservation api under the hood and accepts it as an input. With CAPMC as a wrapper it would rely on PCS to do reservation internally and would just occasionally get freezed out. I think that the hollowing out will be sufficient to mitigate any concerns as far as we can see now.

[rkleinman-hpe]

A few general comments about the proposal language and not directly related to PCS/CAPMC:

1. There is some use of technical jargon here that I think would make it more easily understandable to folks who don't know much about power/hardware control. Specifically "p-state" and "c-state" control and power capping. As someone who is unacquainted with these terms, I am not sure what importance they have.
2. There is also some use of software design jargon like "vertically siloed". I don't really know what that means, but context tells me it is bad.

The spirit of EPs that @alexlovelltroy hoped for was that they would be understandable by a wide audience, with links for deeper information for those who wanted to jump in further.

This comment is applicable to most of the EPs that have come through, and not specific to this one. I just happened to find time to actually say something about it right now. I'll make sure to reiterate this in a future tech leads meeting.

[rkleinman-hpe]

Deprecate CAPMC (following the policy). CAPMC will be converted to call PCS under the hood until its death

Do you expect that the scope of changes to CAPMC will be large enough that a phased transition out of CAPMC will be more disruptive to getting PCS out to customers than a forklift upgrade to PCS? There is a lot of discussion here about the CAPMC code being monolithic, difficult to test, and not extensible. That makes me worry that we'll spend a lot of time making changes to CAPMC that is throwaway.

Have you considered other deprecation strategies that are less flexible, but still acceptable to customers (especially if we can deliver PCS faster)?

[nieuwsma]

Deprecate CAPMC (following the policy). CAPMC will be converted to call PCS under the hood until its death

Do you expect that the scope of changes to CAPMC will be large enough that a phased transition out of CAPMC will be more disruptive to getting PCS out to customers than a forklift upgrade to PCS?

I guess I’m not sure what you mean.

There is a lot of discussion here about the CAPMC code being monolithic, difficult to test, and not extensible. That makes me worry that we'll spend a lot of time making changes to CAPMC that is throwaway.

That is true it is all throw away.

Have you considered other deprecation strategies that are less flexible, but still acceptable to customers (especially if we can deliver PCS faster)?

we haven’t found a strategy that’s better. Do you have any suggestions.

[nieuwsma]

A few general comments about the proposal language and not directly related to PCS/CAPMC:

1. There is some use of technical jargon here that I think would make it more easily understandable to folks who don't know much about power/hardware control. Specifically "p-state" and "c-state" control and power capping. As someone who is unacquainted with these terms, I am not sure what importance they have.
2. There is also some use of software design jargon like "vertically siloed". I don't really know what that means, but context tells me it is bad.

The spirit of EPs that @alexlovelltroy hoped for was that they would be understandable by a wide audience, with links for deeper information for those who wanted to jump in further.

This comment is applicable to most of the EPs that have come through, and not specific to this one. I just happened to find time to actually say something about it right now. I'll make sure to reiterate this in a future tech leads meeting.

Randy I agree with you; the curse of jargon is you dont know you are using it unless someone tells you. :)

Specifically in these cases p-state and c-state are settings for frequency and sleep limits for the processor; since we arent doing them I wont go into long detail about them; this provides PCS context. also -> Here is an intel blog about p-states if you are curious.

Regarding vertical siloing... Im not sure how to best describe this; but ideally software should have layers with interfaces between them so layers can be changed out or worked on independently. CAPMC has 'levels of misdirection'; it has many nested function calls; and function re-use, but no hard layer boundaries which means that trying to change something in the business logic is really hard because it is spread across many parts of the code. Not sure if there is a good phrase to describe this.

[mjendrysik-hpe]

Both CAPMC and the PCS service use component reservations. Unless PCS is going to determine power state by inference and not by querying the hardware, I don't see there being a big concern with CAPMC and PCS being able to control power at the same time.

With respect to having CAPMC call PCS, there is some throwaway code there yes, but not much. And the logic used can be ported over to the CLI. We could always replace CAPMC with a python script that is basically a HTTP server wrapper around the CLI.

[SeanWallace]

Review period has ended. Closing.