Proposal: make it a stress test tool

[apiraino]

Hi @vi

first of all, congrats and thanks for this project, it's interesting and I see a lot of potential. Namely, the are not many options to stress test a websocket service (at least my searches did not lead to many options). websocat can be automated with some scripting but it's a bit impractical. I think that it could deliver better with some additional features. For example, make it behave a little bit like ab:

$ websocat -n 5000 -c 200 -t 60 ws://localhost:1234

-n requests     Number of requests to perform
-c concurrency  Number of multiple requests to make at a time
-t timelimit    Seconds to max. to spend on benchmarking

What do you think?

[vi]

Currently the closest mode would probably be websocat -t tcp-l:127.0.0.1:2345 ws://localhost:1234 and stress-testing TCP server 127.0.0.1:2345.

I'm not sure how to I make a generic stress-tester for websockets for services more complicated than a simple "ping-pong". I would expect it to also analyse replies somehow.

Websocat can contain components to allow specifying stress testing rules on command line although. For example, there can be a "WebSocket message multiplier" to make a forwarder which duplicates each messages in one direction and collapses replies (as soon as they are the same and number matches) in the other direction. Or the same, but with parallel connections instead of sequentially duplicated messages.

---

What is your use case? How do you measure quality of service under stress?

[apiraino]

Hi, thanks for your thoughts on the matter.

Yes, as you correctly point out a better scenario would be more something like:

• connect X clients
• send Y messages in Z time (ex. customize msg/s ratio)
• measure the response time (total, p95, p99, mean response/sec)

For the moment I'm trying a bit Artillery.io but I'm not entirely satisfied (easy to use but a resource hog), but it should give an idea of what I have in mind.

My scenario is:

• connect X clients
• send messages, each message triggers something on the backend
• measure load on backend services
• measure response time on the clients

An idea could be adding to websocat one more feature:

• send a msg every s seconds, input can be stdio: or readfile:

In this way one can script websocat like:

# Start 10 clients
for i in 10;  do
     # Connect a new client, every 2 secs send a msg taken from messages.txt
     websocat -t --linemode-strip-newlines -s 2 readfile:messages.txt ws://localhost:1234
     sleep 1;
done

(the measuring part can be done later because it's more complex)

[vi]

Are all messages and replies assumed to be the same and each message is supposed to emit exactly one reply?

This is what I called a "ping-pong" type of service, which can be easier to test.

[apiraino]

Are all messages and replies assumed to be the same and each message is supposed to emit exactly one reply?

Yes, in my case we can assume that (not sure if this assumption covers other people use cases).

[vi]

To measure response time while shoveling multiple messages into one channel without waiting for replies, either IDs need to be assigned to messages or replies should be sequential.

---

In conclusion, very simple and primitive tools for stress-testing ws:// may be built into Websocat, but that would probably not enough for majority of real stress testing and measurements.

[apiraino]

Well, yes a stress test tool is way more complex, this is why I was mentioning to simply automate the user input. Anyway, nothing that some scripting can't solve...

$ cat send-timed-messages.sh
while read line; do
    echo $line
    sleep 1
done < bench/messages.txt

and then run $ sh bench/send-timed-messages.sh | websocat -t --linemode-strip-newlines ws://localhost

The above is not a stress tool :-) rather a dumb monkey bombing the server, which I guess is ok for now.

[apiraino]

closing as no actionable item emerged from the discussion.

thanks for your time :+1:

[vi]

I may still add msg-duplicate: and conn-duplicate: overlays, which would duplicate each message N times or make N connections from 1. For example, with

echo 1234 | \
     websocat -t - msg-duplicate:conn-duplicate:ws://127.0.0.1:1234 \
     --msg-duplicate-n=1000 \
     --msg-duplicate-delay-us=2000 \
     --conn-duplicate-n=100 \
  > /dev/null

it would send 100000 messages in 100 connections in 2 seconds.

It won't handle counting or measuring replies although, unless you replace /dev/null with something else. Would it be useful for you?

[apiraino]

Interesting suggestions.

--msg-duplicate-n and --msg-duplicate-delay-us are certainly a great addition, would you implement it also for readfile:? The usecase could also be:

websocat -t ws://127.0.0.1:1234 \
     readfile:messages.txt 
    --msg-delay-us=2000 \
> /dev/null

Notice how I've renamed --msg-duplicate-delay-us to --msg-delay-us to apply for a input file.

--conn-duplicate-n: this could be interesting for a "ramp up" scenario in which additional wave of new clients connect and start sending messages. However to make it more flexible I think it would also need a timer to spread the new connections (example instead of 100 new connections at once, let them connect gradually ten by ten). Rather than this parameter, I think in this case I would use another instance of websocat and keep one instance of websocat = one client. Also to keep simple the response analysis. What do you think?

[vi]

would you implement it also for readfile:

What's the difference between websocat -t readfile:messages.txt ... and cat messages.txt | websocat -t - ... in your case?

> /dev/null

readfile: endpoint already acts as a /dev/null analogue if you try to write into it (i.e. websocket server replies to messages read from the file).

---

Why not just drive Websocat from outside? Each line that arrives on Websocat's stdin get converted into a WebSocket message. You can regulate how fast you feed lines into Websocat, and that will regulate how fast WebSocket messages would be sent.

Example - send 1000 messages within 1 second:

perl -we 'for my $i (0..1000) { print "Qqq\n"; select undef,undef,undef,0.001; }' | websocat  ws://echo.websocket.org/

[apiraino]

What's the difference between websocat -t readfile:messages.txt ... and cat messages.txt | websocat -t - ... in your case?

No difference, I would just prefer using your overlay because I like it and it's cleaner :-)

Why not just drive Websocat from outside? Each line that arrives on Websocat's stdin get converted into a WebSocket message.

Correct and that's exacly what am I doing now (see https://github.com/vi/websocat/issues/77#issuecomment-606723222) and it's working fine (although a bit ugly), then you suggested to implement two more parameters to allow a bit less scripting for my case and that would work better.

So in the end, like you correctly mention I can reach my goal anyway with a bit of scripting and without abusing your time :-) A more integrated solution into websocat would be nicer and cleaner but I don't feel that you need to do additional work just to accomodate my usecase (and nobody else).