End-to-end Benchmarks for WebSockets

[CarnaViire]

We currently don't have any end-to-end WebSockets benchmarks, only SignalR one. We want to be able to benchmark WebSockets in itself, for example, to measure and improve performance for the compression feature.

The benchmark can be done in a similar way SignalR is implemented. Both client and server should be added. All metrics are collected and calculated in client code. We would also need a new WebSocket worker for benchmarking infra.

Main benchmarking repo: https://github.com/aspnet/Benchmarks Agent: https://github.com/dotnet/crank

For development purposes, Crank can be executed locally.

SignalR example: Client: https://raw.githubusercontent.com/aspnet/Benchmarks/main/src/SignalRClient/signalrclient.yml Worker: https://github.com/aspnet/Benchmarks/blob/main/src/BenchmarksClient/Workers/SignalRWorker.cs Scenario description: https://github.com/aspnet/Benchmarks/blob/main/scenarios/signalr.benchmarks.yml Server: https://github.com/aspnet/Benchmarks/tree/main/src/BenchmarksApps/SignalR

Other client example: https://raw.githubusercontent.com/dotnet/crank/main/src/Microsoft.Crank.Jobs.Bombardier/bombardier.yml

Crank docs: https://github.com/dotnet/crank/blob/main/docs/getting_started.md

Scenarios:

1. We should start with a simple WebSocket scenario similar to what SignalR is doing. We can use SignalR message format for a payload.
2. Second step would be to add a compression scenario. We can also use SignalR message format there, it should be somewhat compressible and it will be an estimate of how enabling compression will affect SignalR. We can also try random data, which should be uncompressible at all, to benchmark the overhead of compression. Need to think of other possible useful data formats.
3. We can also add some configurable scenario to help with #51231, e.g. a user sends a lot of messages of same size and we could change this size (including making it big, > 1MB, > 64MB possibly?), and we would apply there our auto-splitting to find out the best trade-off.

cc @BrennanConroy @zlatanov

[ghost]

Tagging subscribers to this area: @dotnet/ncl See info in area-owners.md if you want to be subscribed.

Issue Details

---

We currently don't have any end-to-end WebSockets benchmarks, only SignalR one. We want to be able to benchmark WebSockets in itself, for example, to measure and improve performance for the compression feature. The benchmark can be done in a similar way SignalR is implemented. Both client and server should be added. All metrics are collected and calculated in client code. We would also need a new WebSocket worker for benchmarking infra. Main benchmarking repo: https://github.com/aspnet/Benchmarks Agent: https://github.com/dotnet/crank For development purposes, Crank can be executed locally. SignalR example: Client: https://raw.githubusercontent.com/aspnet/Benchmarks/main/src/SignalRClient/signalrclient.yml Worker: https://github.com/aspnet/Benchmarks/blob/main/src/BenchmarksClient/Workers/SignalRWorker.cs Scenario description: https://github.com/aspnet/Benchmarks/blob/main/scenarios/signalr.benchmarks.yml Server: https://github.com/aspnet/Benchmarks/tree/main/src/BenchmarksApps/SignalR Other client example: https://raw.githubusercontent.com/dotnet/crank/main/src/Microsoft.Crank.Jobs.Bombardier/bombardier.yml Crank docs: https://github.com/dotnet/crank/blob/main/docs/getting_started.md Scenarios: 1. We should start with a simple WebSocket scenario similar to what SignalR is doing. We can use SignalR message format for a payload. 2. Second step would be to add a compression scenario. We can also use SignalR message format there, it should be somewhat compressible and it will be an estimate of how enabling compression will affect SignalR. We can also try random data, which should be uncompressible at all, to benchmark the overhead of compression. Need to think of other possible useful data formats. 3. We can also add some configurable scenario to help with #51231, e.g. a user sends a lot of messages of same size and we could change this size (including making it big, > 1MB, > 64MB possibly?), and we would apply there our auto-splitting to find out the best trade-off. cc @BrennanConroy @zlatanov

Author:	CarnaViire
Assignees:	-
Labels:	`area-System.Net`
Milestone:	Future

[kondratyev-nv]

I can take at least the first scenario if nobody minds 🙂

[BrennanConroy]

We've had baseline websocket benchmarks with 1000 bytes of random data since July 2021, we just added compression and different sizes (10, 100, 1,000, 10,000), still with random data.

Next step would be to add some sort of non-random data. One suggestion was a SignalR message, but I'm not sure how useful that would be as the SignalR framing is very small and the main part of the message is user data.