Support custom capacity per subscription

[nazar-pc]

Proposed change

I'd like to see something like Client::subscribe_with_options(), specifically for ability to customizing subscription capacity. This is especially important because some subscriptions might have larger messages and others might have smaller messages, hence one size fits all doesn't work well here.

[Jarema]

Sure, we can have this option, however keep in mind, we count messages, not bytes, as that relies on underlying mpsc capacity.

[nazar-pc]

I understand, this is why it is important to be able to customize it since with domain-specific knowledge developer may have a better understanding what message sizes to expect and what capacity is appropriate.

[Jarema]

We have to think about it. Adding additional restriction (not only on message count, but also on message size) means that client will likely drop more messages and cause client side slow consumer. Such limit does not restrict client from receiving the message from the server in the first place.

[nazar-pc]

It doesn't, but maybe it is expected that for some channels messages are like 1M in size, so default limit will result in 65G of buffered messages, while other messages are a few bytes and buffering more of them is fine.

Implementing size limit will be challenging, but allowing to customize number of messages with domain-specific knowledge is much less difficult. I just don't think there is one size fits all in this case.

[Jarema]

Yes, I understand the use case.

In some languages, like rust nats client, that is quite simple task (where the only challenge is to have least possible performance impact). However, features like this need to land in every nats client for parity, so it needs some discussions before moving forward.