Support for document TTL/expiration

[sheinbergon]

Though not directly supported via the underlying MVStore, adding support for object TTL via some kind of maintenance thread could be a valuable feature.

[TareqK]

Nitirte has events that you can listen for, as per the documentation here. You can simply add a change listener for your collections, and use a ScheduledExecutorService from java concurrency and implement the TTL yourself.

[sheinbergon]

@TareqK that's not really how you address the issue of document expiration/TTL:

• It cannot rely on modification operations. In some scenarios you just need the expunging of documents without waiting for a modification to take place. For read-heavy workloads, this is the common case.
• It can be overloaded on the read operation (as a side effect), but that would make reads slower (which is undesirable) unless you're choosing an asynchronous approach. It would also require extra development.

[TareqK]

It cannot rely on modification operations

The method i suggested does not, you simply schedule the delete when the document is inserted(you cant delete a document that isnt inserted), by listening for an insertion event. Its that simple.

You simply need to check that the event type is insert,(ie, event type is ChangeType.INSERT) schedule the event, and let the application continue running. Everything is done in a non-blocking way, including the handling of the insertion event and the deletion, since

Event listener code always executes in a background thread in a non-blocking fashion.

and

An ObjectRepository also supports the same set of operations that NitriteCollection supports. It is also observable and tread-safe for concurrent use.

So i dont see any need to implement and test more code in the database itself for this use case, since the tools to do what you need are already there.

[sheinbergon]

TTLs can get more complicated than "schedule an in-memory deletion once the document is inserted" and forget about it":

• TTLs might change... so now you need to maintain this aspect as well.
• Do we want to be able to support disabling a TTL on a given document, without changing its contents?
• What about application/TTL persistence? I guess you need to store TTL values and make sure you load them after restarts?
• What about replication, which this project currently aims to support? should TTLs be set on both ends?
• Here, have a look at how Redis approaches this issue (high level).

TTL is a very common (and desirable) feature in Document/KV stores, though it's not always easy to implement correctly. The best approach to discussing such a feature is probably not doing it in the comments section of a github issue.

As for you seeing no need to add more code to the an open-source project... well just because you can roll your own partial solution with minimal effort (or a full one with way too much effort for a user-based solution) doesn't mean it shouldn't ever be part of the core project "because more code and tests". That's not a very open-source and fruitful approach to requested feature discussion.

Neither of us is the/a maintainer (unless I've missed something) for this project, and neither of us gets to decide if it's going to be a nofix or an enhancement. I've merely opened the issue to see if @anidotnet might consider it (or not), and it's up to him.

[anidotnet]

I had this very thought at the initial stage, but I backed off. Because, for embedded databases like nitrite, maintaining a ttl monitoring thread is costly. When your application is closed, nitrite has no way to delete expired documents or indices. It will only delete when your application starts up and database is loaded. This will increase your application startup time as you have to wait until the ttl monitoring thread expunge all the expired documents, for correctness.

In your application you can very well create a ScheduledExecutorService to monitor a specific collection/object repository and evict the expired documents at a specific stage/time of your choice. So it would be better off to delegate this task to the application rather than to the library.

But I am open to discussions. Please feel free to share your views. If it helps the community to have a ttl feature in nitrite, I can squeeze it in.

[anidotnet]

Closing this issue for now due to inactivity.