search

riscv-non-isa / riscv-iommu

RISC-V IOMMU Specification
https://jira.riscv.org/browse/RVG-55
Creative Commons Attribution 4.0 International
72 stars 14 forks source link

Question about the behavior of IOMMU when iommu_mode is changed to off #361

Closed viktoryou closed 1 week ago

viktoryou commented 2 weeks ago

From the spec,

When the iommu_mode field value is changed to Off the IOMMU guarantees that in-flight transactions from devices connected to the IOMMU will be processed with the configurations applicable to the old value of the iommu_mode field and that all transactions and previous requests from devices that have already been processed by the IOMMU be committed to a global ordering point such that they can be observed by all RISC-V harts, devices, and IOMMUs in the platform.

What about the case when the iommu_mode field value is changed to Bare? Is IOMMU also required to guarantee such outcome?

Furthurmore, I do not understand well about the mechanism here. Since a CSR configuration and a translation request are sent by different masters, I guess it is not easy for IOMMU to know the global ordering point of them. What would be the proper flow for IOMMU to update the iommu_mode without applying the wrong configuration to the translation request? Would the software need to assist to some extent?

ved-rivos commented 1 week ago

What about the case when the iommu_mode field value is changed to Bare? Is IOMMU also required to guarantee such outcome?

As stated it is only the transition to Off. The IOMMU is expected to use the same machinery it uses for IOFENCE with PR=PW=1 to provide this capability.

viktoryou commented 1 week ago

About the machinery of IOFENCE, I would like to confirm its detailed flow. A IOFENCE command may be used by software when the iommu_mode field value is changed to Off. Then the following request from device would be stalled until this IOFENCE command is commited and gets observed. Is it right?

ved-rivos commented 1 week ago

Once the IOMMU has seen the register write to make mode=Off, it starts rejecting/aborting/dropping new packets - because IOMMU is now Off. Packets that had already been processed need to be made visible before the busy bit in the ddtp gets cleared. To make the packets already processed - previous reads and writes - are implicitly fenced.

viktoryou commented 1 week ago

More specifically, as Figure 6 from the spec (IOMMU interfaces) shows, once the IOMMU has seen the register write to make mode=Off, whether the inbound traffic from IO device or device translation request is rejected/aborted/dropped?

ved-rivos commented 1 week ago

Correct. When mode is Off, all inbound memory transactions are disallowed.

viktoryou commented 1 week ago

Some question occurred to me and I would like to make all things clear:

  1. in such case, request from IO device would still be sent normally, right?

  2. whether in-flight transactions from devices connected to the IOMMU contains the remaining device translation request in IO bridge that has not been processed by IOMMU, after iommu_mode=off being observed by IOMMU?

    (i). if it contains, to reject all inbound memory transactions, there exists several hardware delays from signaling IO bridge that ddtp busy and new inbound memory transactions may get allowed. Then how could we aviod this problem?

    (ii). if it dosen't contains, it would be a compliant implementation that IOMMU immediately rejects to accept the device translation request from IO bridge, right?

ved-rivos commented 1 week ago

The host bridge makes translation requests to the IOMMU. It may get a translation response if the IOMMU mode is not Off but once the IOMMU has observe the mode as Off it will always get Failure response. The situation may be as follows:

                                          <----------time----------------------  
        NR NW NR....NW...   {IOMMU mode is observed as Off} PR PW....PR PW  -> Memory

So in this example, all the new read (NR) and new write (NW) will get aborted. All previous read (PR) and previous write (PW) will get committed to a global observation point before the ddtp.busy goes from 1-> 0. The IOMMU does not need to wait for PR/PW to drain before starting to abort NR/NW. The net of this is that once mode has been set to Off and the busy goes 1->0 there will be no more memory accesses from that IO port.

See also https://github.com/riscv-non-isa/riscv-iommu/pull/243/commits/2358c137a5efa62a288c3eaade98ed29025940c1