While trying to get 1000s of listening simulated tentacles working, I was running into issues in which Octopus was unable to communicate with the Tentacle. Eventually I discovered that the CPU usage of Tentacle Simulator was high despite it not doing anything. Turns out it was spending all its time in SpinWait.
This was done because TcpListener.Stop hangs if Accept is in progresshttps://github.com/dotnet/runtime/issues/24513 . Which is claimed to be fixed, although I see an absolute disaster of reverts so it is not clear if it is fixed.
Working on the assumption TcpListener.Stop still hangs if Accept is in progress, the new optional path is to:
await waiting to accept a connection with the cancellation token.
when the SecureListener is disposed first the CancellationToken is cancelled, then we wait for the accept task to complete (which will be some sort cancellation exception), after that when we know we are not in the middle of Accept we then call TcpListener.Stop.
Note that the new behaviour must be opted in by setting:
UseAsyncListener = true;
on HalibutTimeoutsAndLimits.
Results
Before
CPU usage was half my cores when listening on 1000 sockets.
After
CPU usage is near nothing.
Tentacle Simulator can actually be used with 1000s of listening tentacles.
How to review this PR
Quality :heavy_check_mark:
Pre-requisites
[ ] I have read How we use GitHub Issues for help deciding when and where it's appropriate to make an issue.
[ ] I have considered informing or consulting the right people, according to the ownership map.
[ ] I have considered appropriate testing for my change.
Background
While trying to get 1000s of listening simulated tentacles working, I was running into issues in which Octopus was unable to communicate with the Tentacle. Eventually I discovered that the CPU usage of Tentacle Simulator was high despite it not doing anything. Turns out it was spending all its time in
SpinWait.This was done because
TcpListener.Stop hangs if Accept is in progresshttps://github.com/dotnet/runtime/issues/24513 . Which is claimed to be fixed, although I see an absolute disaster of reverts so it is not clear if it is fixed.Working on the assumption
TcpListener.Stopstill hangs if Accept is in progress, the new optional path is to:SecureListeneris disposed first the CancellationToken is cancelled, then we wait for the accept task to complete (which will be some sort cancellation exception), after that when we know we are not in the middle of Accept we then callTcpListener.Stop.Note that the new behaviour must be opted in by setting:
on
HalibutTimeoutsAndLimits.Results
Before
CPU usage was half my cores when listening on 1000 sockets.
After
CPU usage is near nothing.
Tentacle Simulator can actually be used with 1000s of listening tentacles.
How to review this PR
Quality :heavy_check_mark:
Pre-requisites