xcause save privilege modification bits

[billhuffman]

As the spec stands, if MPRV, SUM, or MXR is used, instructions need to be added to handle them. I suggest that we add fields in xcause to save these bits and clear them.

In particular, I suggest than when an M-mode interrupt occurs: mcause[26] gets set to MPRV and MPRV gets cleared mcause[25:24] gets set to MPP before MPP is set to indicate the privilege we're coming from

When an S-mode interrupt occurs: scause[25] gets set to MXR and MXR gets cleared scause[24] gets set to SUM and SUM gets cleared

We'd need some way to copy the values back on returning. MRET and SRET are the obvious possibilities, but they return from exceptions as well. On the other hand, MRET already sets MPP to U-mode so we can't set them before that. Maybe xRET does the copy if xcause[31] is set.

[kasanovic]

MPRV is used for emulation of missing load/store instructions, and is usually run with interrupts disabled. If disabling interrupts is not acceptable, one approach is to delegate illegal instruction exceptions to lower privilege mode using medeleg.

[kasanovic]

SUM is only for systems with supervisor mode. SUM only affects M-mode if using MPRV. Currently only affects page-based translation though there is a proposal to add S-mode PMPs, which would be possibly affected by SUM. Original usage model for SUM assumed that supervisor-mode interrupts would be off while SUM was turned on. If interrupts can be enabled while SUM is being used, then handlers have to save SUM, set/clear SUM as needed for handler, then restore SUM.

If don't want to use SUM and have page table, then can have replicated page table entries (one with supervisor access, one with user access). If don't want to use SUM and have S-mode PMPs without translation, then don't have same ability to prevent user from causing supervisor to touch that memory.

So, in case that have SUM, supervisor-mode interrupts need to save/restore SUM in handlers while interrupts are disabled.

[kasanovic]

MXR is only for systems with supervisor mode. Like SUM, MXR only affects M-mode if using MPRV and only affects page-based translation currently though might also impact S-mode PMP systems. Like SUM, supervisor-mode interrupts need to save/restore MXR in handlers while interrupts are disabled.

[Kevin-Andes]

From: Allen Baum

Just a note: MPRV is used for far more than emulation of missing load/store - anything that needs emulation via an illegal op trap can use it, and many services provided by Mmode via eCalls will use it, and debuggers entered through eBreak will use it

[billhuffman]

I agree that MPRV can be used with interrupts disabled. But in an embedded system, turning interrupts off for even a few cycles is often not good. Same for SUM and MXR in supervisor mode. So, for environments with 1000 cycle interrupt latency, this may not matter. But then the purpose of CLIC is biased toward embedded systems.

In addition, there are cases like memcpy where the code doesn't a priori know how long it is. Setting up a translation is a lot of overhead for a short memcpy and the interrupt latency is a problem for a long memcpy. Of course, the length can be tested. But...

I'd like to be able to save MPRV in mcause (just like MPP and MIE) and SUM/MXR in scause (just like SPP and SIE). Specifically:

• mcause[26] for MPRV
• scause[26] for SUM
• scause[29] for MXR

[kasanovic]

From TG meeting 8/17/2021. SUM and MXR do not seem to get in way of time-critical interrupt handlers, as handler can save/restore state without knowing their value, and in most cases can ignore their setting completely.

The MPRV problem is really a problem with MPP, which is assumed to always have correct value when using MPRV. This causes a problem if we want to reduce worst-case interrupt latency by not disabling interrupts for duration of handler that uses MPRV.

An exception/trap handler has to save/restore MPP before enabling interrupts, but this is not enough to allow an horizontal M-mode trap handler to ignore MPP, as it will wipe out the state of MPP.

This is a problem with the MPRV model which relies on MPP being correct at all times (cannot just save/restore MPP at beginning/end of routine). One workaround is for the routine using MPRV to always first save MPP before enabling interrupts, then whenever an MPRV-modulated access is required to do the sequence: disable interrupts/restore MPP/perform access (using MPRV)/enable interrupts. This reduces interrupt latency at cost of Increased code size/reduced performance in MPRV-using routine.

[dansmathers]

Adding Bill Huffman email: We talked on Tuesday about issues with using MPRV, MXR, and SUM in an interrupt latency sensitive environment. For each, there are potential issues both for the interrupt handler because they aren’t being saved and cleared at interrupt time and for the operation of the sequence without damage from interrupts. I’ll go through each set of issues. There are bold questions in two places.

MPRV interrupt handler issues:

MPRV won’t be on unless we’re in M-mode. A horizontal interrupt can be taken but MPP will then be set to M-mode. MPRV on is the same as MPRV off when MPP is M-mode. So it doesn’t directly affect the operation. If the handler changes MPP, it should also change MPRV. It will probably do that naturally.

MPRV sequence execution issues:

When setting MPRV, we are also using MPP, which is lost on an interrupt. We have a few choices for what to do. The ones I’ve thought of are:

Make additional register stack fields so that MPP is preserved across interrupts – perhaps the additional stack would be in mcause Disable interrupts while using MPRV – Linux does this now and it’s is fine, but it may be an issue in an interrupt latency sensitive environment Don’t use MPRV at all – use memory regions shared between M and S/U for all communication. Since M-mode accesses by physical address and S/U access using the MMU, addresses will need careful tracking. Let the interrupt happen and fix up the results – When the interrupt returns, MPP will be U-mode. If the access then does not succeed the exception handler can note that the PC was in a region that was using MPRV and return to a different location that fixes up MPP and jumps back into the original sequence.

The reason for discussing this in the context of CLIC is to determine whether we should do #1 to make MPRV easy to use. #2 is fine in interrupt latency insensitive environments. Do we believe #3 is always acceptable in interrupt latency sensitive environments? Or are there issues there? Is #4 realistic?

MXR and SUM interrupt handler issues:

Under the current Privilege and CLIC specs, these are not saved/cleared or restored as part of some interrupt handling step.

M-mode is not affected by MXR or SUM unless MPRV is used and MXR/SUM can be cleared, if necessary, when MPRV is set. S-mode interrupt handlers will need to be careful not to do anything that could matter before saving and replacing sstatus or after restoring it. Time critical interrupt handlers are believed to be able to do their work without being affected by MXR/SUM. Do we believe that’s always true? U-mode interrupt handlers, if we have them, can only have come from U-mode and MXR and SUM should already be clear.

MXR and SUM sequence execution issues:

MXR and SUM should be restored before returning and should not have any issues in the execution sequence using them.

[billhuffman]

The hypervisor spec is including explicit load and store instructions to access with lower privileges rights. This direction may well replace the use of the state bits considered here. Given that, it's probably best just to let the restrictions described above remain for the time being.