Remove lock contention in IOQueue

[benaadams]

Currently a lot of contention on the lock

Go through the lock if its not already doing work; otherwise skip the lock.

Submit work to threadpool outside the lock

After contention is much reduced

Also removes QUWIC allocations from the IOQueue on .NET Core 3.0

Resolves: https://github.com/aspnet/KestrelHttpServer/issues/3042

[halter73]

Both of the windows Jenkins runs failed due to the Sockets functional tests timing out 😟

https://ci3.dot.net/job/aspnet_KestrelHttpServer/job/master/job/windows-Configuration_Release_prtest/117/consoleFull https://ci3.dot.net/job/aspnet_KestrelHttpServer/job/master/job/windows-Configuration_Debug_prtest/117/consoleFull

[halter73]

Are there any benchmarks results for this? I see how in this reduces lock contention in Schedule() when DoWork() is in progress, but it does add a Volatile.Read() in Schedule() and the Volatile.Write() in DoWork() each time DoWork() needs to be restarted.

[benaadams]

Are there any benchmarks results for this?

Added a PR for 2.2 branch for easy benchmarking https://github.com/aspnet/KestrelHttpServer/pull/3096

I see how in this reduces lock contention in Schedule() when DoWork() is in progress, but it does add a Volatile.Read() in Schedule() and the Volatile.Write() in DoWork() each time DoWork() needs to be restarted.

The Volatile.Read and Volatile.Write should be fairly free on x64 (as there are no register benefits); but does allow it to more often skip Interlocked.CompareExchange; spinning and sleeping from the lock.

Arm would have more costs from the Volatile calls

[benaadams]

Actually, should be able to drop the lock entirely here

[muratg]

Kestrel work has moved to aspnetcore repo, and this repo will be archived. If you're still interested in pursuing this PR, please move it over to aspnetcore. Thanks.