search

riscv-non-isa / riscv-iommu

RISC-V IOMMU Specification
https://jira.riscv.org/browse/RVG-55
Creative Commons Attribution 4.0 International
82 stars 16 forks source link

Why the status is set to Success,when no other faults were encountered but the "Page Request" could not be queued due to the page-request queue being full (pqt == pqh - 1) or had a overflow (pqcsr.pqof == 1)? #262

Closed wangyongzhen0322 closed 10 months ago

wangyongzhen0322 commented 10 months ago

As explained below, when an overflow occurs, the page request message is discarded, that is, the PRG is not fully received by the software. In my opinion, the status should also be set to Response failure at this time. I cannot understand why status is set to Success.

While either error bit is set in pqcsr, the IOMMU discards all subsequent "Page Request" messages, including the message that caused the error bits to be set.

ved-rivos commented 10 months ago

Good question. This behavior is specified to allow the OS to be able to recover from a temporary overflow of the Page Queue.

Lets first recap the page fault protocol. The device makes a ATS translation request and receives insufficient permissions in response. This leads the device to send a Page Request message to the operating system asking for the desired permissions.

The Page Request may be responded three ways a) Success b) Invalid Request c) Response failure. The Invalid Request response is for cases where request cannot be granted and the ATS request will never succeed even if retried. The Response Failure indicates that a catastrophic error has occurred and the device should disable its PRI and not issue any more Page Request till it is re-enabled. The Success response indicates that the pages have been made resident and the device should retry its ATS request.

A page queue overflow is not a catastrophic even that requires disabling of the PRI. The request that triggered the page fault may be recoverable - we just are not able to hand it to the OS. In this case the IOMMU sends a Success response. This tells the device it should retry the ATS translation request - which will obviously fail again - but on this failure the device will resend the Page Request. Hopefully the OS has fixed the overflow condition by now and this resent Page Request will reach the OS.

wangyongzhen0322 commented 10 months ago

I got it. Thank you for your answer. And I have another question, why not hold those requests until overflow is resolved. If the overflow is fixed quickly, this won't waste much performance. If it takes some time for the overflow to be fixed, then directly returning success may cause the device to repeatedly send unnecessary ATS requests.

ved-rivos commented 10 months ago

It is usually not a good idea to require implementations to "buffer" or "hold on" to packets in an IOMMU in hardware. Thats why the page request queue - a buffer - is allocated in system memory. There is no amount of buffer that can be reasonably built in hardware such that it would never overflow. And if it does overflow then this sequence needs to be implemented anyway. The OS is expected to add up the maximum number of page requests the devices in scope of an IOMMU can generate (this is reported in the Page Request Capability) to size the page queue. There is also an option to limit the number of page requests a device can generate through the page request capability. So while a page queue overflow can occur it is not expected to be frequent.

wangyongzhen0322 commented 10 months ago

I see. Thank you for your answer.