Add Lazy Mode Support for Consumers

[ashupednekar]

Proposed change

Issue: Add Lazy Mode Support for Consumers

Feature Request

Add support for starting consumers in lazy mode to the NATS.rs client.

Description:

In this enhancement, we aim to introduce a feature that allows consumers to start listening to specific subjects in a lazy manner. The proposed implementation would allow consumers to listen on a pattern like some.pattern.*, but delay the creation of individual consumers until triggered by an internal event. This internal event could signal the base consumer to initiate a separate consumer for each relevant subject.

What is it

We have something called lazy mode for our consumers... what happens here is that the consumer process won't be subscribing to the subject pattern directly, instead it starts an internal consumer which listens for start event for every new unique subject and dynamically spawns a consumer thread subscribing to said subject.

This decision is made by the producer... before publishing the intended message, it checks with nats core if there is a consumer defined for said subject, and sends the start event accordingly.

Expected Outcome:

• Reduced overhead from inactive consumers.
• Better scalability and resource management for use cases with varying consumer requirements.

This feature would make the consumer system more dynamic and resource-efficient.

Use case

• Concurrent, independent consumers for individual users.
• Instead of starting all consumers at once, the system should spawn individual consumers only when required, optimizing resource usage.
• Useful in scenarios where specific users or events don't need continuous active listening, but can be initialized as needed.

Contribution

A similar system has already been built on top of NATS.rs, and the functionality proved useful. Additionally, this feature will require a lazy producer counterpart to send the initial start event, which can only be achieved using JetStream, not the NATS core.

[ashupednekar]

I'd like to take up this feature if it aligns with the project's direction. I've built something similar before and believe this could be a valuable addition to NATS.rs. Let me know if there's interest, and I can start working on a proposal or implementation.

[Jarema]

Hey.

Thanks for detailed explanation of the proposal.

However, I feel that this is very application-specific and use-case specific. Addtionally - Consumers are server side constructs, so it's not only about subscribing at given time, but also deciding when you want to create the on the server/cluster.

You should be able to achieve any of those goals with current client logic:

• if you want to postpone creation of the consumer, you can store a Future and await it when you want to create it.
• you can keep the config and call the create whenever needed
• if you want to have the consumer existing on the server, but not used on the client, you can create it, and just not stop messages call on it. You can also store the Future for later initiation if that's what you want.

[ashupednekar]

Cool, I already have it running at the application level.

Closing this issue then. Lmk if there's anything else in the roadmap that I can contribute to