Subscribing to a large number of subjects takes a lot of time and memory consumption

[oncicaradupopovici]

Subcribing to:

• 100 subjects: takes 1 second and occupies 6 MB
• 1000 subjects: takes 12 seconds and occupies 48 MB
• 10000 subjects: takes 120 seconds and occupies 450 MB

We can see that it scales quite liniar but we are in a multi-tenant environment and we need to subscribe to a collection of subjects for every tenant, so the number gets large quickly.

Is there any tunning settings we can do to reduce this numbers? Is this behaviour specific to dotnet clr? I mean, should we try other languages/runtimes?

[ColinSullivan1]

@oncicaradupopovici , thank you for raising this issue. Each STAN subscription is backed by a number of underlying NATS subscriptions, so they take up quite a bit more memory than core NATS. I'll take a look.

In the meantime, reducing the # of tasks used by the underlying core NATS library should help. NATS defaults to providing the best performance, allocating a task for each subscription. This works very well for a smaller number of subscriptions, but at scale it can negatively impact performance. If you set the underlying NATS connection's SubscriberDeliveryTaskCount option to something like 5 or 10 you should see better memory usage and get better performance with a large number of subscriptions.

Here's a code snippet demonstrating how to do this:

            // Create core NATS options and set the delivery task count
            var nOpts = NATS.Client.ConnectionFactory.GetDefaultOptions();
            nOpts.SubscriberDeliveryTaskCount = 5;

            // Create a core NATS connection with the options.
            var nc = new NATS.Client.ConnectionFactory().CreateConnection(nOpts);

            var opts = StanOptions.GetDefaultOptions();

            // Tell the NATS streaming client to use the core NATS connection
            opts.NatsConn = nc; 
            var c = new StanConnectionFactory().CreateConnection(clusterID, clientID, opts);

[ColinSullivan1]

I ran a quick test, tried tweaking delivery task count and a few other options - unfortunately that doesn't appear to make a difference:

Note that with .NET core version 3.0.101 and a memory only streaming server I'm seeing better memory usage and performance.

Default Configuration:
100 subscriptions = 1.033 seconds with 1601200 bytes memory
1000 subscriptions = 2.325 seconds with 10732056 bytes memory
10000 subscriptions = 33.033 seconds with 101916024 bytes memory

I'll look to see if there are other places we can conserve memory. The NATS client will scale much better with JetStream (next generation streaming that is under development) since it uses only the underlying core NATS subscriptions.

[oncicaradupopovici]

We have managed to reduce the memory allocation from 450MB to 13Mb for 10000 subjects, using the TaskCount set to 5, witch is great! We will do some tests to see how it performs with this setting. Thank you very much for your help!

[ColinSullivan1]

Great! Glad it helped, and thanks for using NATS!