Closed icebob closed 1 year ago
It looks that it's an issue with the nats
library version 2.x.x. With the previous 1.x.x version the performance is fine.
There is an open issue (2 years ago) about it in NATS repo: https://github.com/nats-io/nats.js/issues/438
For more context, libs:
"moleculer": "0.14.31",
"nats": "2.15.1",
Used OS for local testing: MacOS 13.5 Service images in the k8s cluster are based on node:18-alpine
Could you switch back to nats 1.4.12
to check the performance with this version as well?
Already tried. Initially, it was using nats 1.4.12
and the results were pretty similar, that's why I moved to 2.15.1
.
What is the NATS server version?
2.9.11
, if more concrete docker.io/bitnami/nats:2.9.11-debian-11-r0
plz try with the latest version 2.9.21
It appears that I've identified the root cause of the issue. Upon experimenting with various broker settings, I found that disabling the metrics feature resolved the problem. In my project, I use a StatsD reporter. Everything is functioning as anticipated with the Prometheus reporter. The StatsD reporter configuration looks like this
{
type: 'StatsD',
options: {
// Server host
host: 'localhost',
// Server port
port: 8125,
// Maximum payload size.
maxPayloadSize: 1300,
}
},
So it seems definitely not the NATS issue. I'll turn off StatsD for now.
It's strange because I can reproduce this issue without any metrics, only with 2.x.x nats lib.
I've found the problem inside the nats
library. The sending logic is changed to queue-based in 2.x.x version. Skipping this logic I could reach 30.000 msg/sec instead of 40 msg/sec. I've opened an issue in the NATS client repo: https://github.com/nats-io/nats.js/issues/581
This is a significant improvement, waiting for this fix then 😃
Pushed my tests just in case https://github.com/mrprigun/moleculer-benchmark-test There are two dedicated nodes and nats in docker-compose. With enabled StatsD reporter I receive ~170rps
, with disabled reporter it's ~2k rps
on my laptop.
Nats service: 2.9.21 Nats lib: 2.15.1 Node: v18.15.0 OS: MacOS 13.5
Discussed in https://github.com/moleculerjs/moleculer/discussions/1235