search

nats-io / nats.go

Golang client for NATS, the cloud native messaging system.
https://nats.io
Apache License 2.0
5.29k stars 670 forks source link

Increase KeyValue store message history #1365

Open solita-cristian opened 10 months ago

solita-cristian commented 10 months ago

Feature Request

Increase the current maximum history message in a key-value store from 64 to 255.

Use Case:

With this change it will be possible to store more messages per key.

Proposed Change:

Change the KeyValueMaxHistory constant value from 64 to 255.

Who Benefits From The Change(s)?

Everyone that needs larger histories per KV entry.

Alternative Approaches

Let the maximum history value to be configurable

ripienaar commented 10 months ago

Conceptually the history isn’t a “store messages per key” it’s a revision history.

If you need to store messages we have a great message store and related APIs - just use a stream.

solita-cristian commented 10 months ago

Thanks for the reply. I might have expressed myself wrong, but the intent is to allow for a bigger revision history.

ripienaar commented 10 months ago

Would you mind explaining your use case a bit?

solita-cristian commented 10 months ago

In our project we are using the kv store to store revisions (we call them versions) of a particular payload. We have reached a point where the number of revisions is reaching the threshold, and we would like to not store older revisions in a separate stream/kv store, also because the kv store can potentially contain lots of keys.

This is the main reason for this change, as reaching the threshold would require extra work on our side, but there are no visible downsides increasing (or make configurable) the size of the history per kv entry in the NATS client itself.

ripienaar commented 10 months ago

From your description it doesn’t sound to me like a 256 limit will solve it. It just means you will hit it again later.

The problem is we may want to later support other backends than a stream - since while streams are “fine” (good for millions of keys etc) for this we can definitely improve things with a dedicated store.

So bumping it means such a future backend would need to be a stream again.

Inherently history is a audit trail of past changes the key goes through.

For your case it’s sounds like just using a actual stream will be a good choice and probably you will gain a bunch of new abilities that would also be beneficial.

These limits are about the abstraction and what we think it’s suitable for and bumping up to them probsbly tells you that your use case are better suited for streams. We recently released new APIs for JetStream there a lot more approachable and usable it might be worth giving tbem a look

We are discussing this internally though so any additional information you can provide regarding the revisions and why you need so many would be useful.