Why is it necessary to invalidate the page table when DC.fsc.MODE == Bare？

[baimengwei]

the iommu spec have the following sentences:

** If `DC.tc.PDTV==1 || DC.tc.PDTV == 0 && DC.fsc.MODE == Bare`
*** `IOTINVAL.VMA` with `GV=AV=PSCV=0`
** else
*** `IOTINVAL.VMA` with `GV=AV=0` and `PSCV=1`, and `PSCID=DC.ta.PSCID`

https://github.com/riscv-non-isa/riscv-iommu/blob/main/iommu_sw_guidelines.adoc?plain=1#L153-L154

if DC.tc.PDTV == 0 && DC.fsc.MODE == Bare, I would think that after the DC has been invalidate at line 146 , the iosatp in the DC.fsc has been updated from the IOATC. The value in the iosatp does not need to point to a page table for the first-level translation.

https://github.com/riscv-non-isa/riscv-iommu/blob/main/iommu_sw_guidelines.adoc?plain=1#L146

I want to know if IOTINVAL.VMA with GV=AV=PSCV=0 is required under DC.tc.PDTV == 0 && DC.fsc.MODE == Bare . thanks.

[ved-rivos]

The sequence is still recommended as implementations may have Bare mode translations in the IOATC.

[baimengwei]

IOATC has data about Bare mode seems meaningless abort page table walk. Is there any purpose of this kind of implementations ?

[ved-rivos]

Sorry I missed you had a follow up question. Caching Bare mode translation is commonly done in microarchitectures as that is how the pipelines are usually built - hitting the ATC as a fast path and missing being a slower path. However, you are right that we can strike out the DC.tc.PDTV == 0 && DC.fsc.MODE == Bare as Bare mode translations do not need to be invalidated either as they are always correct. I will strike that term out in the errata PR.

[baimengwei]

What I want to express is that storing bare mode address mapping in ATC does not seem plausible.

If there is a mapping table entry for bare mode address on the ATC, it seems to cause a performance degradation in design. When ATC performs tag comparison in bare mode, it will take a long time in the pipeline. However, bare mode address could have returned the result directly after being recognized by IOMMU for getting faster translation results back, rather than search the ATC.

If it is to simplify the design, it is also possible to store bare mode address mapping in ATC and bare mode address will search the ATC, same as other mode address translation. However, such IOMMU performance seems have a negative impact.

The IOTINVAL.VMA command is still necessary if bare mode address mapping stores in ATC. Because when switching from non-bare mode to bare mode, using bare mode still accesses ATC, without the IOTINVAL.VMA command, the iommu will return the old translation result of non-Bare mode.

Please correct me if my understanding is wrong. Thanks.

Sorry I missed you had a follow up question. Caching Bare mode translation is commonly done in microarchitectures as that is how the pipelines are usually built - hitting the ATC as a fast path and missing being a slower path. However, you are right that we can strike out the DC.tc.PDTV == 0 && DC.fsc.MODE == Bare as Bare mode translations do not need to be invalidated either as they are always correct. I will strike that term out in the errata PR.

[ved-rivos]

What I want to express is that storing bare mode address mapping in ATC does not seem plausible.

I am unable to speak definitely about the your hardware implementation. It may certainly not be plausible.

If there is a mapping table entry for bare mode address on the ATC, it seems to cause a performance degradation in design. When ATC performs tag comparison in bare mode, it will take a long time in the pipeline. However, bare mode address could have returned the result directly after being recognized by IOMMU for getting faster translation results back, rather than search the ATC.

It certainly seems like your implementation does not perform a O(1) lookup in the ATC and so seems like your tradeoffs are different. It is possible that looking up ATC is a slow operation in your implementation.

The IOTINVAL.VMA command is still necessary if bare mode address mapping stores in ATC. Because when switching from non-bare mode to bare mode, using bare mode still accesses ATC, without the IOTINVAL.VMA command, the iommu will return the old translation result of non-Bare mode.

When a Bare mode translation is cached it would be cached with the GSCID and/or PSCID as not valid. When the regime switches to non-Bare more the GSCID and/or PSCID will be valid and so Bare more translations are not eligible for a match. Bare translations being identity-mappings do not need explicit invalidation.

[baimengwei]

Thank you very much, what you said makes sense.

When a Bare mode translation is cached it would be cached with the GSCID and/or PSCID as not valid. When the regime switches to non-Bare more the GSCID and/or PSCID will be valid and so Bare more translations are not eligible for a match. Bare translations being identity-mappings do not need explicit invalidation.

[baimengwei]

the iommu spec have the following sentences:

When an IOMMU is transitioned to Bare of Off state, the IOMMU may retain information cached from in-memory data structures such as page tables, DDT, PDT, etc. Software must use suitable invalidation commands to invalidate cached entries.

According to the previous dialogue abort this issue and translation process, switching from non-bare mode to bare mode seems still can get the correct result without invalidation command.

But the above quote still shows that it is necessary to use invalid. Is there any difference in principle between the ddtp.iommu_mode and the iosatp.mode ?

In a broader scenario, for a certain cache and its corresponding mode, when the mode switches from bare mode to non-bare mode, the invalid command must be used in order to make the switching result take effect. This is for sure. But is the invalid command still necessary when switching from non-bare mode to bare mode?

[ved-rivos]

This is discussing something completely different. When the iommu_mode is transitioned to Bare or Off, the IOMMU will not automatically invalidate any data cached from data structures referenced by ddtp such as the device context, process context, first-stage PTEs, second-stage PTEs, or MSI PTEs.

[baimengwei]

This is discussing something completely different. When the iommu_mode is transitioned to Bare or Off, the IOMMU will not automatically invalidate any data cached from data structures referenced by ddtp such as the device context, process context, first-stage PTEs, second-stage PTEs, or MSI PTEs.

You're right. However, it seems that iommu in bare mode does not involve any search, but just returns IOVA. So, after the current iommu switches from non-bare mode to bare mode, the current iommu can still get the correct result without invalidation command according to the translation process at step 2.

Any invalidate steps from software are there is to prevent unexpected translation results, but I can't find what could be wrong if I don't execute invalidate steps when the iommu_mode is transitioned to Bare or Off, even if the IOMMU will not automatically invalidate any data cached from data structures referenced by ddtp.

[ved-rivos]

That text is not asking SW to invalidate because mode was changed to Bare. It says that the IOMMU will not invalidate by itself and any invalidation if needed should be done by SW. So if you switched from non-Bare to Bare to non-Bare dont be surprised that IOMMU kept the cached translations.

[baimengwei]

the IOMMU will not invalidate by itself and any invalidation if needed should be done by SW.

It seems that if iommu is switched to Bare mode, the SW seems to only need to switch to bare mode, any invalidation is not needed.😐

If iommu is switched to Bare mode, invalid commands are not done by SW, which does not affect translation results. If iommu is switched to non-Bare mode again, invalid commands are necessary to ensure translation is correct. This process is universal.

[ved-rivos]

Yes. You got that right.

[ved-rivos]

Marking this as resolved. Please reopen if there are further questions.