M-mode interrupt registers for systems supporting multiple supervisor domains

[NicholasWoodIMG]

For systems supporting multiple supervisor domains, add AIA extension along the lines of:

• mseip and mseie CSR
• mip.msdeip = |(mseip[] & mseie[])

allowing M-mode to arbitrate interrupts across multiple supervisor domains, analogous to corresponding registers allowing a hypervisor/OS to arbitrate interrupts across multiple guests.

Supporting an interrupt model along the following lines:

• (Logically) separate APLIC and IMSIC for each supervisor domain
• Each follow current AIA specification (no change to AIA spec; no change to programming model for S-mode software)
• M-mode manages access control to each (logical) instance of APLIC/IMSIC - for example PMP/IOPMP
• M-mode can arbitrate across S-mode interrupts and domains (context switch, mute, whatever)
• M-mode can delegate interrupts for the currently active supervisor domain

Refer also to discussion with Ved.

In a virtualised system the same mechanism can be used for multiple domains at HV level.

[ved-rivos]

Here are some thoughts on this topic

[gokhankaplayan]

@ved-rivos your document is self-explanatory, thank you very much. I put a comment on it about using a different wording for APLIC.

In addition to everything said in this thread, I think we need to protect writes to IMSICs by IOPMP. Since MSIs are sent by external devices (PCIe devices), Non-secure device should not be able to send MSIs to Secure IR-file. IOPMP need is a more generic problem for systems having external masters, but it would be good to have common understanding for IOPMP need here.

[ved-rivos]

@gokhankaplayan - I updated the wording for APLIC. The protection of writes to IMSIC can also be accomplished using either a) MTT or b) PMP (and its IO counterpart). There is a proposal for MTT based IO protection in PR #4 . The IOPMP or PMP may also be used.

[rsahita]

From 9-19 discussion - question regarding active SD being interrupted: It is M-mode responsibility that the bit corresponding to supervisor domains not active are set to 0 in the msdeia (register previously called msdeip)

[gagachang]

I have a question about applying this extension to APLIC without IMSIC.

AIA specification section 4.2 says:

When a RISC-V hart’s external interrupt controller is an APLIC, not an IMSIC, the hart can be
within only one interrupt domain of this APLIC at each privilege level.

The google slide says: each domain associated with its dedicated APLIC domain.

Does this imply that each domain should run on different harts? e.g., SD - 0 should run on hart 0 and associate with APLIC domain 0. SD - 1 should run on hart 1 and associate with APLIC domain 1.

SD - 0 and SD - 1 can not run on identical hart by time-slice manner, i.e., context switch by RDSM ?

[NicholasWoodIMG]

@gagachang The idea is that each supervisor domain only "sees" a single (logical) IMSIC/APLIC instance.

For firmware in each supervisor domain the AIA specifications stand without change. So a system can be built around IMSIC, or APLIC, or both, as defined by AIA.

M-mode can manage which IMSIC/APLIC instance is mapped into each supervisor domain using PMP, or MTT, for example. Supervisor domains can share harts - M-mode manages context switching between domains, including PMP/MTT configurations.

[gagachang]

So if the system supports N supervisor domains, it also needs N APLIC instances? Each domain has its own APLIC instance, am I correct?

Let me take two supervisor domains, one for REE and another for TEE for example. If SD - 0 and SD - 1 share hart 0, the architecture looks like this ?

[NicholasWoodIMG]

At least logically two APLIC instances - separate interfaces for each supervisor domain. M-mode firmware essentially can see all interrupts as per current AIA spec. The new part is the ability of M-mode firmware to manage interrupts across multiple S-mode contexts, and decide policy - delegate, mute, context switch, etc. as required per S-mode domain.

Note - the AIA spec uses the term "domain" with a different meaning than "supervisor domain". Going forward we should probably make an effort to use only the terms "supervisor domain" or "interrupt domain" respectively to avoid confusion.

[rsahita]

I opened a PR for the outline of this chapter - PTAL : https://github.com/riscv/riscv-smmtt/pull/9

[gagachang]

Hello,

I am wondering if Smsdia can "let current running Supervisor domain know that there are any other Supervisor domain pending interrupts".

For ARM, they can route all of the foreign interrupts, which are owned by other supervisor domains, as FIQ signal injected into current running domain. If there is a pending interrupt owned by another supervisor domain (e.g., REE), the current supervisor domain (e.g., TEE) can handle FIQ and do managed exit [1], finally the current supervisor domain can yield the control to target supervisor domain (i.e., REE) to handle that pending interrupt. This allows current supervisor domain to manage its internal state before yielding control to the supervisor domain where the foreign interrupt is handled. This is how ARM handles Linux task migration between different PEs (Processing Element) during TEE call.

I am not sure whether it is suitable for RISC-V. But it might be useful if we can support it.

[1]: https://developer.arm.com/documentation/den0077/latest/

[gokhankaplayan]

Instead of the ARM approach, M-mode Software can let the interrupted supervisor domain to do managed exit. Do you think approach works for Smsdia, @gagachang ?

[gagachang]

Instead of the ARM approach, M-mode Software can let the interrupted supervisor domain to do managed exit. Do you think approach works for Smsdia, @gagachang ?

Hi @gokhankaplayan I think "current" Smsdia can only let M-mode software know there are pending interrupts owned by current supervisor domain (via mip.SEIP) or other supervisor domains (via new defined mip.MSDEIP).

There are no new bit fields for CSR sip, which can be accessed by interrupted supervisor domain.

[gokhankaplayan]

Foreign interrupts are reflected in mip.MSDEIP and cause M-mode Software to take the control. M-mode software can give the control to the interrupted supervisor domain by mret?

[ved-rivos]

Whether the Domain Security Manager performs the interruption autonomously or uses methods like managed exit may be a software policy. When using methods like managed exit the manager may either inject a virtual interrupt into the SD (pre-determined or negotiated/configured) or may use method like SSE.

[gagachang]

@ved-rivos Thank you! I didn't notice SBI SSE. After brief checking SBI SSE spec, it should need some spec expanding, since we are not restoring context to the interrupted domain, but to the target domain which owns that pending interrupt. While SBI SSE restores context to the same supervisor domain.

Inject a virtual interrupt by M-mode software is another solution, such as a fake PLIC/APLIC interrupt source which indicates foreign interrupts from other supervisor domains. But injection via PLIC/APLIC leads to some additional bus transactions. A new bit field for CSR sip might have best performance. The optional delegation can also be added so that pending interrupts from other supervisor domains can be directly taken to the current supervisor domain. Like ARM directly route the non-secure interrupts to secure world via FIQ signal.

Update: virtual interrupt can be injected via new AIA CSR mvien & mvip, so there might not be additional bus transactions.

[gagachang]

Hi @ved-rivos In addition to injecting virtual interrupts into the SD or using method like SSE by M-mode software, could we also add support for direct interrupt routing without entering M-mode software first?

Please refer to ARM's GIC specifications:

• GICv3 and GICv4 Software Overview [1] - Figure 11 Alternative routing model
• Arm Generic Interrupt Controller Architecture Specification [2] - section 4.6.2 Interrupt assignment to IRQ and FIQ signals

Those specifications say that non-secure interrupts can be taken as FIQ signal to Trusted OS (S.EL1), without the need to first enter EL3 secure monitor, so it can reduce overhead.

I can imagine that Smsdia is a right place to add such similar support. since ARM's FIQ is akin to interrupts from other supervisor domains. Please let me know your thoughts, thanks!

[1]: https://developer.arm.com/documentation/dai0492/latest [2]: https://developer.arm.com/documentation/ihi0069/h/?lang=en

[gagachang]

Foreign interrupts are reflected in mip.MSDEIP and cause M-mode Software to take the control. M-mode software can give the control to the interrupted supervisor domain by mret?

Hi @gokhankaplayan

I finally realized that foreign interrupts can be reflected as

if (mip.MSDEIP == 1 && mip.SEIP == 0)

since the value of mip.MSDEIP is or_all_bits(msdeia & msdeie) and the bit of current SDID should be excluded. Am I correct?

[ved-rivos]

The proposed architecture allows the following:

1. Direct delivery of a SDs interrupt to the SD without needing M-mode intervention
2. Mechanism to notify M-mode about pending interrupts for SDs other than the one that is presently scheduled

The #2 allows the M-mode to preempt the executing SD if required. That preemption may be accomplished by the M-mode autonomously context switching the SD or by doing a managed exit through a M-mode injected interrupt into the SD.

Sending a interrupts of other SDs into an SD is likely not desirable. For example, in case of ARM there are only two SDs - Secure and non-secure. And secure is more trusted than non-secure. The secure may be trusted enough to actually do managed exit on the FIQ. However, when we have mutually distrusting SDs, sending equivalent of a FIQ into a SD asking it to kindly yield to another SD may not be desired. The SD receiving this "FIQ" may choose to ignore it - not this SD is no more trusted or "secure" than any other SD.

We avoid all these by providing the notification to the M-mode. If the M-mode decides to do managed exit, it can also start a timer with for an upper bound on how long the SD may take to yield. Some M-mode may decide not to do a managed exit to avoid the side channel of occurrence of an unrelated interrupt to an SD.

All of these are however software policies that the domain security manager can implement. As you rightly noted the proposed ISA supports all of these models.

[gagachang]

Thanks you @ved-rivos , it is so clear. I agree with you that RISC-V has mutually distrusting SDs, and the proposed ISA is flexible for software to support different policies. Now my problem is addressed, thanks!

[rsahita]

will capture this discussion in the informative section (and will then close this issue).

[rsahita]

published a new release with the recently updated PR for Smsdia - closing this issue.