IOCP on Windows to support SetFileIoOverlappedRange for OVERLAPPED structs...

[leonleon77]

Was wondering about various issues (pro/against) adding support for SetFileIoOverlappedRange on Windows... https://msdn.microsoft.com/en-us/library/windows/desktop/aa365540%28v=vs.85%29.aspx

[Ext3h]

Cons:

• It requires the SeLockMemoryPrivilege privilege on the process.
• In contrast du io_uring, it works on a per-file context rather than a "per IO context", so registration needs to be repeated for every handle the buffer is used with.
  • Same also applies to Windows implementation of IoRing!

Pros:

• Completely avoids page-mapping into kernel space.
  • No more global kernel side page table mutex contention.
  • No more global process side page table mutex contention.

Traps:

• The function name says "FileIo", and that's all it works for!
  • Socket stuff is handled exclusively by the RIO (see #784) instead.
• You need to call SetFileIoOverlappedRange both for the OVERLAPPED struct itself as well as on the buffer passed.
  • If you don't pool your OVERLAPPED structs, you are still going to hit the bottleneck eventually!
  • No need to register the WSABUF array - it's copied, not mapped.

When evaluating whether it's actually giving you any performance benefits, it's necessary to first ensure that you actually have some base load for the critical paths. A concurrent thread with lots of large dynamic memory allocations on the heap is a prime candidate, because it's hitting the same process / system global page table mutex.

On a perfectly idle system where only the IOCP APIs are used by a single thread, you will easily max out any other possible bottleneck, before SetFileIoOverlappedRange becomes relevant.

On a contended system, a fraction of of that becomes realistic when not using SetFileIoOverlappedRange - and a profiler points you to MmProbeAndLockPages and within to KeYieldProcessorEx - you are hitting a nasty spinlock. The IO APIs are not at fault for holding it too long, they just get hung up on it. And as with any spinlock - the more threads you throw at it, the worse it becomes.

---

So, to put it short: If you don't need SetFileIoOverlappedRange then you won't notice. When you need it, you will hit a wall without it.

It's only solving the "high bandwidth" part for direct I/O though.

You are still going to hit an IOPS limit eventually, and for that metric IoRing (see #1200) would be the way to go. Which coincidentally covers both registered buffers and high IOPS. But in return is not IOCP / Overlapped related at all.

[Emjayen]

Just in-case anyone stumbles upon this: much of the above is incorrect.

• SetFileIoOverlappedRange is an optimization for the I/O manager when accessing the ISB (OVERLAPPED is a superset in Win32 speak) during an I/O request's completion processing. The provided range of pages are locked and mapped into system address space. It does not apply I/O buffers as is being suggested (that's what RIO is for)

• As this is part of the common I/O manager, contrary to what's claimed: this is applicable irrespective of object type (file or socket)

• There are no "global page table mutexes", presumably referring to PTE access; this was somewhat the case in NT5, however that was more than 20 years ago. TLB shootdowns are still nasty though.

• The NT kernel represents I/O buffers by way of physical page descriptors (MDLs), which optionally are mapped into VA of system/process. The first step of building an MDL (again, responsibility of the I/O manager[^1]), is the cited routine MmProbeAndLockPages, of which yes, can be rather expensive (and again, is what RIO aims to solve[^2]

[^1]: This isn't strictly true, as it depends on the I/O buffering mode and negotiation with the uppermost driver. [^2]: RIO is rather borked in respect to performance, and has been for years (and seemingly not maintained anymore)