STAN: Subscribe request timeout

[rutuja-clusters]

I have a NATS streaming server running on an AWS EC2 instance.

nats-streaming-server -a [ip] -p 4248 -store file -dir store_1

And I am using stan.go/examples/stan-bench on another AWS EC2 instance and using the above server.

cd stan.go/examples/stan-bench go run main.go -s "nats://[ip]:4248" -np 5 -ns 5 -n 10000 -ms 1024 foo

However, on running this, I get the following error: Subscriber benchmark-sub-0 can't subscribe: stan: subscribe request timeout exit status 1

Is there any limit on the number of subscribers? Or can I change the duration of request timeout?

[kozlovic]

First, let me just point out that -ns 5 means that the bench tool is going to create 5 subscribers on subject "foo". Which means that for the 10,000 messages sent (since you use -np 5, each publisher will send 10,000/5=2,000 messages), the server will send 50,000 messages total (10,000 to each subscriber). This number if not big in general, but I wanted to make sure you understand that this is what is happening.

Now to your questions:

Is there any limit on the number of subscribers?

Yes, there is a default of 1,000 subscribers per channel, so I don't think that you are hitting the limit, also, the error returned to the client would explicitly say that.

Or can I change the duration of request timeout?

Yes, although we use a global setting for all internal requests, which is the option ConnectWait() that you can pass when creating the connection: stan.Connect(clusterID, clientID, stan.ConnectWait(5*time.Second)).

Were there any error reported in the server log? Getting the timeout for the subscription but not the connection seem to indicate that there is no network issue (at least initially), and I hope that this is not a simple resource issue. I know that the server/clients can do way better than that. Check the CPU/network usage for your 2 AWS instances to see if for some reason you were hitting some limits.

[rutuja-clusters]

Thank you for the information!

Were there any error reported in the server log?

There were no errors in the server log.

I again tried running the same command a few times, it gave an output after the fourth try. Not sure exactly why this happened.

Also, wanted to clarify one more thing. Here, do subscribers receive messages as they are getting published? Or does the publish happen first, followed be subscribe

[kozlovic]

Here, do subscribers receive messages as they are getting published? Or does the publish happen first, followed be subscribe

The tool starts the subscription and then the publisher. As a message is published, the server will get matching subscriptions and deliver (if possible) to the subscriptions, then send the ACK back to the publisher. Note that if you restart the bench tool and reuse the same subject and there are pending messages, the server will have to deliver the old messages first, so new messages are likely going to be stored but not delivered as they are persisted by the server. There is an option -io that you can use in the bench tool to ignore old messages, the subscribers will receive only new messages published at the time they are started. If your storage is slow, it could be the issue you observed because the server has to go read messages at the beginning of the file (for the given channel) and append new messages to the end of the file. If the storage is not backed by a SSD, this is going to be quite slow.

[rutuja-clusters]

Thanks!