Bigger Receiving Througput with QoS 1 and 2

inovex / mqtt-stresser

Load testing tool to stress MQTT message broker

Apache License 2.0

259 stars 78 forks source link

Bigger Receiving Througput with QoS 1 and 2 #54

Open AlmostRuio opened 3 years ago

AlmostRuio commented 3 years ago

Hello! I'm stress testing a broker and and I noticed a lot bigger receiving througput with QoS 1 and 2. Otherwise same parameters. Broker is on AWS. No errors, all 100% published/received. Why is that?

./mqtt-stresser-linux-amd64 \
-broker tcp://localhost:1883 \
-num-clients 1000 \
-num-messages 10 \
-rampup-delay 5s \
-global-timeout 180s \
-timeout 20s \
-publisher-qos 2 \
-constant-payload $(cat payload40kb.txt) \
> mqtt_stresser_40kb_10_qos2.txt
/

QoS 0

# Receiving Througput
Fastest: 126 msg/sec
Slowest: 23 msg/sec
Median: 77 msg/sec

QoS 1

# Receiving Througput
Fastest: 49685 msg/sec
Slowest: 71 msg/sec
Median: 550 msg/sec

QoS 2

# Receiving Througput
Fastest: 31005 msg/sec
Slowest: 69 msg/sec
Median: 567 msg/sec

Sorry if is not a bug :)

hikhvar commented 3 years ago

I don't know if it is a bug, but it looks indeed suspicious. I will have a look at our QoS setup code. Maybe there is a bug. Do you have the corresponding sending throughput?

AlmostRuio commented 3 years ago

yes here:

QoS 0

# Publishing Throughput
Fastest: 25 msg/sec
Slowest: 12 msg/sec
Median: 18 msg/sec

# Receiving Througput
Fastest: 126 msg/sec
Slowest: 23 msg/sec
Median: 77 msg/sec

QoS 1

# Publishing Throughput
Fastest: 18 msg/sec
Slowest: 9 msg/sec
Median: 14 msg/sec

# Receiving Througput
Fastest: 49685 msg/sec
Slowest: 71 msg/sec
Median: 550 msg/sec

QoS 2

# Publishing Throughput
Fastest: 16 msg/sec
Slowest: 8 msg/sec
Median: 12 msg/sec

# Receiving Througput
Fastest: 31005 msg/sec
Slowest: 69 msg/sec
Median: 567 msg/sec

hikhvar commented 3 years ago

ok, thank you. I have to have a deeper look at the code. My current hypothesis is: If QoS>0 is set, we first send all messages and receive them afterwards in a complete bulk. If QoS==0 we receive as we sent. The fact that sending with QoS>0 is slower that QoS==0 supports this hypothesis. Moreover this result is expected since QoS > 0 should be slower since it requires more communication between sender and broker.

hikhvar commented 3 years ago

Can you give me some details on the used broker software? Is it a managed service? Can I setup a instance of that broker? This will help to find the issues with this.

AlmostRuio commented 3 years ago

The broker is a simple eclipse-mosquitto (1.6.12) docker image with default configuration. Docker compose:

broker:
    image: eclipse-mosquitto
    ports:
      - 1883:1883