khonsulabs / bonsaidb

A developer-friendly document database that grows with you, written in Rust
https://bonsaidb.io/
Apache License 2.0
998 stars 37 forks source link

Add support for reserving a primary key #272

Open ecton opened 1 year ago

ecton commented 1 year ago

From a conversation on discord with @D1plo1d, a request was made to provide functionality similar to PostgreSQL's NEXTVAL.

We should have a way to get a reserved primary key for a given collection such that the next push to a collection would skip over any reserved keys.

This enables additional transactional flows without the need for interactive transactions.

Implementation-wise, I think the best approach would be to add a copy of the "last-value" key for a given collection to the header of the Nebari tree. Instead of looking at the last key when pushing a document, this last_key would be used instead.

A new function could be added that could increment the last_key and return its value without inserting a new document. I'm uncertain whether this should be a transactional function or atomic with persistence happening the next time the collection is updated. PostgreSQL's documentation has this caveat with their implementation:

If the database cluster crashes before committing a transaction containing a nextval or setval call, the sequence state change might not have made its way to persistent storage, so that it is uncertain whether the sequence will have its original or updated state after the cluster restarts. This is harmless for usage of the sequence within the database, since other effects of uncommitted transactions will not be visible either. However, if you wish to use a sequence value for persistent outside-the-database purposes, make sure that the nextval call has been committed before doing so.

ecton commented 1 year ago

I've added this to the Improve Transactional Performance checklist because this would alter the storage format -- might as well include that part in the format change that is coming with the adoption of the new file format.