SRIOV VFs Fault during Multi-threaded Operation

[MaistoV]

Enviroment setup:

• Software:
  • OPAE-SDK 2.8.0-1
  • Kernel Linux DFL 6.1.41-dfl
  • OS: RHEL 8.9
• Hardware:
  • Quartus 23.2.0 Build 94 06/14/2023 Patches 0.02,0.11,0.19,0.02iofs SC Pro Edition
  • OFS 2023.2-1
  • Hitek C220 PAC, FIM provided and verified by Hitek Systems

Design and use case:

Using the OFSS flow, we have multiple instances of the same AFU in a flat FIM. Leveraging the thread-safety capabilities of the SRIOV stack, we wish to use such AFUs as independent VFs in a multi-threaded/process setup:

• each process grubs and uses an AFU directly
• processes do not interfere with each other

Each AFU is used in a pretty standard OPAE access pattern:

1. We bind the target VF to the VFIO and the user with:
   • pcie_device
   • opae.io init
2. We grub a specific AFU with OPAE APIs in C (not C++), namely:
   • fpgaPropertiesSetObjectType()
   • fpgaPropertiesSetGUID ()
   • fpgaPropertiesSetSegment ()
   • fpgaPropertiesSetBus ()
   • fpgaPropertiesSetDevice ()
   • fpgaPropertiesSetFunction ()
   • fpgaEnumerate()
   • fpgaOpen()
   • fpgaMapMMIO()
3. We allocate two buffers:
   • with fpgaPrepareBuffer()
   • use fpgaGetIOAddress() to retrieve the addresss for the AFU
   • write such addresses either with mapped MMIO or fpgaWriteMMIO64() to AFU CSRs
4. We start the AFU, which reads inputs from the first buffer and writes outputs to the second
5. We wait for the AFU to complete either by polling on the status register or by mapped interrupt events on a file descriptor, i.e. with:
   • fpgaCreateEventHandle()
   • fpgaRegisterEvent()
6. Subsequently read results from the mapped output buffer.

Issue:

We observed a systematic error operating on more than two VFs/AFUs in parallel, effectively limiting the number of usable VFs to 2. This is very problematic to us, since our system-level design relies on the multi-threaded capabilities of the OPAE/SRIOV stack.

More in detail:

• The above sequence works flawlessly for each AFU/VF and buffer size, grubbing, allocating for and releasing one AFU/VF at the time
• The above sequence keeps working when grubbing and allocating for 2 VFs/AFUs
• As soon a third VF is grubbed and buffer are allocated, the oldest operating VFs shows the error:
  • The output buffer is no longer updated by the AFU
  • No page or segmetation fault, or any other memory-related error occurs
  • The input and output buffer can be safely accessed by the host
  • The AFU CSR space keeps working (read, write and even start/poll on status register) both in mapped MMIO and fpga[Write|Read]MMIO64()
• This same behaviour happens for subsequent AFUs:
  • Grubbing/mapping a 4th VF causes the 2nd to cease to working
  • Grubbing a 5th AFU, results in the 3rd to fail
• Such faulty pattern does not persist when releasing and re-acquiring a faulting AFU:
  • After releasing and re-acquiring, this AFU works properly, and the oldest accessed one stops writing on its output buffer

We replicated the issue below on two different host setups, both featuring IOMMU, VT-d and SRIOV support over a PCIe x16 Gen 4.0 slot:

1. Intel(R) Xeon(R) Gold 5418N HPE server host
2. 11th Gen Intel(R) Core(TM) i9, MSI workstation

More notes

• This happens mostly at 2 VFs, but after the host has been under such allocate/release testing, the maximum number of VFs reduces to 1.
• This happens for every AFU/VF, regardless of the utilization order.
• This behaviour is independent from:
  • buffer sizes: it happens from 128B to 16MB buffers
  • number of bound VFs: it happens even with just 3 VFs bound to VFIO
• To accomodate OPAE contiguous buffer allocation, we configured the OS with 16 Giga pages and 1024 Mega pages
• We have yet to probe the FIM design with Signal Tap to explore if there is any traffic on the FPGA and the PCIe interface

Possible issues

Judging from the beaviour above, this seems to us as a software issue. We suspect that at some level of the OPAE library / VFIO driver stack, the IOMMU mapping is compromised, i.e. writes from the AFU are not propagated in the user pages. We have inspected the source code of the OPAE library, but were not able to find a clear cause for this beaviour.

[umairsiddiqui-digitek]

the opae-sdk 2.12.0-5 which was released as part of OFS-2024.1 has several SRIOV/vfio fixes

https://github.com/OFS/opae-sdk/releases/tag/2.12.0-5

check this new update in opae-sdk 2.12.0-5 https://github.com/OFS/opae-sdk/compare/4cb9d59..31d4aba?diff=split&w=#diff-59969ebf5bd82fbc67352bfa9a72c999271f931f95a51842fc52bf16f127f72d

note that in OFS-2024.1, other OFS components like DFL kernel & OFS RTL were updated as well: https://github.com/orgs/OFS/discussions/62