Closed kohlisid closed 7 months ago
Longer story but the reason is the selection mechanism is sync but the sorting mechanism works on data that is delivered async from the other servers around mostly HAAssets but also usage etc.
I will look into if we can improve.
Should be fixed now, will be in 2.10.11, next release which may go out this week.
@derekcollison Thanks for the prompt fix on this :D
Observed behavior
When creating a multiple streams (with replicas = 3) on a Jetstream cluster (with number of nodes > stream replica count), I have been observing a behaviour where the streams are not evenly distributed among the servers.
Some of the server instances end up getting a large chunk of the stream replicas.
On some testing, if we wait for some time (sleep = 3s) before creating consecutive streams we end up seeing a far balanced distribution.
Expected behavior
The expectation was to see a balanced distribution even without the wait between the create calls. In my use case I need to create multiple streams on a Jetstream cluster and such a behaviour might cause performance issues.
I could add a wait to help with the issue but that creates a long delay in the init process when creating a large number of streams.
It would be great if you could highlight if this is the expected behaviour or if there is some other way in which the issue can be remediated?
Server and client version
Nats Server Version:
nats-server: v2.10.9
Client version:nats --version 0.1.1
Host environment
CPU:
Apple M1 Pro arm64
Steps to reproduce
1) Create a Jetstream cluster with 5 server nodes, I'm using the following config for the nodes and starting the servers individually using
nats-server -js -c node.conf
. Each server having a unique name and the port is added to the cluster2) Once all the servers are up and running, Create multiple JS streams. All streams here have an identical configuration apart from having a unique name and subject.
I'm using the nats-cli to create 30 streams
The configuration of the streams are as follows
3) Once all streams are created check the Jetstream server report to find the stream count on each server node
4) If the same steps are followed but with the slight modification of adding a sleep interval between the stream creation we are able to see a well balanced system
node_configs.zip