Concerns related to the requirement of inquiry of DC for PRI enabling (EN_PRI)

[zetalog]

PCIe page request (PRI) requires high speed responses to refill faulted pages. However, IOMMU specification makes it impossible to achieve higher performance responding PRI by putting EN_PRI in DC. Requiring PRI to be sent to the configuration control hardware and more bus transactions / configuration cache fetches to be done for non-leaf DDTs and leaf DDT entry (DC) before allowing such a PRI request.

Should we add a capability bit allowing an implementation to choose not to do so but directly sending PRI to the software. Say, software should be able to determine whether this PRI should be serviced and make such a determination faster w/ CPU frequency.

[ved-rivos]

Page Requests compared to MRd/MWr (Translated or Translation) form a small proportion of the total number of transactions and the rate at which Page Request need to lookup device context is not more than what Mrd/Mwr need. Implementations typically implement a device context cache to accelerate this lookup and an implementation should be able to look up the device context for PR at least with the same performance as it can lookup device context for Mrd/Mwr requests. Typically a Page Request follows in close proximity to a Translation Request that returned insufficient permissions which had to lookup (and fill) the device context cache.

[zetalog]

I can understand what you mean.

However, this check means a differentiation between RISCV and other IOMMU implementation (ARM, etc.) to me. AFAIK, ARM won't check such an enabling at device context basis, but will leave it to the software driver to determine.

When a differentiation between implementations is observed, I will always consider it more in details, asking why the other implementation chooses a different usage model. So shall we give an implementation a chance to mark such a variation using a capability bit?

[ved-rivos]

The Intel IOMMU has a similar check - see Page Request Enable (PRE) in the context entry. The ARM IOMMU does not have this check but does have fields - PPAR - it needs to access in the context entry when it needs to auto-generate a response. The RISC-V IOMMU design is to not allow bad inputs into the system. Adding this optionality and its software impact is is not desired or justified.

[zetalog]

OK, thanks for the detailed explanation.

[zetalog]

According to my information source, ARM IOMMU does contain product variations where PPAR is not implemented (probably there is no PPAR checking real product in the market).

By omitting PPAR checks, PRI request need not be combined with the translation request. If an implementation chooses to do PPAR check with device tables, lots of logics need to be added to modify the format of the request from ATS request, buffered ATS request, queued ATS request, ... and configuration faults.

So why doesn't RISCV make this an implementation specific capability to allow a design to choose a simpler micro-architecture? As long as there is no usage model issue as software can easily determine if the page request can be granted for the give device.

[ved-rivos]

The PPAR parameter is needed to support PCIe 4.0 and higher (latest is PCIe 6.0). The RISC-V IOMMU design is to not allow bad inputs into the system and to support PCIe 6.0 specification. An IOMMU compliant with PCIe 4.0 and beyond has to support this capability. The reason a request will fault is if the device is generating a page request when it is configured not to - either its malicious or buggy. The logic needed to lookup the device context for the PR transaction and the associated faults is no different than any other transaction that the IOMMU needs to support and from a microarchitecture perspective its a simplification to not have special case for this transaction. There is no optionality defined in the RISC-V IOMMU architecture in this aspect; adding the optionality and its software impact is is not desired or justified.

[zetalog]

OK, thanks for the decision. Anyway, doing or not doing just needs to be determined.