Open MaistoV opened 2 months ago
the opae-sdk 2.12.0-5 which was released as part of OFS-2024.1 has several SRIOV/vfio fixes
https://github.com/OFS/opae-sdk/releases/tag/2.12.0-5
check this new update in opae-sdk 2.12.0-5 https://github.com/OFS/opae-sdk/compare/4cb9d59..31d4aba?diff=split&w=#diff-59969ebf5bd82fbc67352bfa9a72c999271f931f95a51842fc52bf16f127f72d
note that in OFS-2024.1, other OFS components like DFL kernel & OFS RTL were updated as well: https://github.com/orgs/OFS/discussions/62
Enviroment setup:
Design and use case:
Using the OFSS flow, we have multiple instances of the same AFU in a flat FIM. Leveraging the thread-safety capabilities of the SRIOV stack, we wish to use such AFUs as independent VFs in a multi-threaded/process setup:
Each AFU is used in a pretty standard OPAE access pattern:
Issue:
We observed a systematic error operating on more than two VFs/AFUs in parallel, effectively limiting the number of usable VFs to 2. This is very problematic to us, since our system-level design relies on the multi-threaded capabilities of the OPAE/SRIOV stack.
More in detail:
We replicated the issue below on two different host setups, both featuring IOMMU, VT-d and SRIOV support over a PCIe x16 Gen 4.0 slot:
More notes
Possible issues
Judging from the beaviour above, this seems to us as a software issue. We suspect that at some level of the OPAE library / VFIO driver stack, the IOMMU mapping is compromised, i.e. writes from the AFU are not propagated in the user pages. We have inspected the source code of the OPAE library, but were not able to find a clear cause for this beaviour.