Add a key-value store implementation

[TheRamBucket]

Describe the project you are working on

I've needed this feature in many projects but currently, I'm needing a way to store player statistical information such as in-game stats to be displayed on leaderboards. collections of item information such as stack count, description, valid uses, damage, durability, effects, multipliers, etc. I am looking to create a centralized system for storing compressing and or encrypting information so it cant be edited outside of the game and is stored in one place instead of needing files for each type of data independently.

Describe the problem or limitation you are having in your project

Currently saving and loading large amounts of data involves having each class control how the data for that class is stored and retrieved. It means that for each new project, I have to decide how the data will be stored, where to store it and create new implementations for each operation I want to do. While certain classes or objects do need their own implementations for the most part all of this data can be stored in a key-value store if it was available.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

Having the ability to store information in a central location available to be accessed from any location in code would greatly reduce the amount of work needed to create, store, and manage custom data. Having a system like a kv store in place would lighten my workload and provide a better alternative for managing statistical data in one location. Not all data needs to have its own methods to store and retrieve it which is where databases or key-value stores shine. Providing a central intuitive way to store this type of information would drastically improve my workflow.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

Users would interact with the kv store using a nosql or kv store class which provides methods to create or delete stores, insert, retrieve, update, and delete keys, as well as the ability to find keys based on filters to quickly sort through and provide a list of keys that match the filter. In my current design is using an LSM tree for the base of the persistently stored data along with a Red-Black tree, bloom filters, and write-ahead logging to improve both speeds and integrity of data. Im also considering adding an in-memory hash map for caching recently accessed key-value pairs to limit the number of io operations required to manipulate commonly accessed data. The kv store would work directly with Godot types foregoing the need to convert to and from c/c++ specific types like you'd need to do when using 3rd party databases like LevelDB or SQLite.

In short, when you call to store a key-value pair it would be added simultaneously to the cache, Red-Black tree and the write-ahead log. once a threshold of entries is reached the Red-Black tree would be flushed to disk as an sstable in an asynchronous operation. It then creates and uses a new empty RB-tree to continue with writes while the flush is happening. After the flush is completed the old RB-tree is removed.

Periodically these sstables are merged in the background following the LSM strategy.

When getting a key-value pair you would first check to see if theres a cache entry for it. if not you then check the RB-tree. If its absent from either you would load and check with the bloom filter info starting in the most recent sstable working to the oldest. If a bloom filter states the key may be contained in the file only then would you check the files index for the value. if a value is found it and the other keys within the block would be loaded into the cache and its value returned.

If needed I can provide more information on my current implementation and data structures.

If this enhancement will not be used often, can it be worked around with a few lines of script?

No to accomplish a similar task you would need to design your own data management scheme to manage the data of objects.

Is there a reason why this should be core and not an add-on in the asset library?

Given the nature of the kv store its general use and close integration into the engine I could see it as a feature that would find common and frequent use in projects of all kinds. While it could be made as an addon godot currently lacks a simple built-in alternative to solve this common issue of data management.

[TheRamBucket]

I am currently writing the implementation of this feature for use in my own projects though since I find it a useful tool to have I want to share it as a contribution for others to use in theirs. Regardless of whether it's determined to be fit enough for integration into the Godot engine, I will be making the code public in the form of a module you can add and build yourself

Repo

[Calinou]

• Related to https://github.com/godotengine/godot-proposals/issues/2691.

In general, when you need a multiplayer storage backend, you interact with a third-party solution such as Firebase using a REST API or similar. Therefore, you don't need key-value storage directly within the game.

[TheRamBucket]

though not all projects are intended to be multiplayer and still need a way to store statistical information in a centralized manner. usually, I see sqlite is used in that case. You also don't always want or have the ability to use, host, or manage a 3rd party solution for a game/project

In general, when you need a multiplayer storage backend, you interact with a third-party solution such as Firebase using a REST API or similar. Therefore, you don't need key-value storage directly within the game.

[ZynosStudios]

I'd prefer to use something like mongodb or some other nosql database over sql but i also cant see us paying to host a server on a game that may not sell well. its just another risk to take you know. Maybe self hosted servers would work but again more cost

[TheRamBucket]

I'd prefer to use something like mongodb or some other nosql database over sql but i also cant see us paying to host a server on a game that may not sell well. its just another risk to take you know. Maybe self-hosted servers would work but again more cost

I'm planning at some point to have a standalone server made that can be self-hosted to work with multiplayer. I'll start to work on that after the in-engine implementation is done. However, I'm also thinking that if you plan on hosting dedicated servers for a game the key-value store is right there alongside it so you wouldn't need to host anything separately unless you need data to be accessed from all servers.

[GersonFeDutra]

Adding for this proposal, I am on a situation that I don't want to expose the encrypt key from my game into the code repository. Setting an environment variable into, for example GitHub secrets, would be Ideal. The current work-around is to use scripts to construct the git-ignored export_presets.cfg. Not really a problem, but (maybe) an interesting use-case.

[Calinou]

Adding for this proposal, I am on a situation that I don't want to expose the encrypt key from my game into the code repository. Setting an environment variable into, for example GitHub secrets, would be Ideal.

This is not what a key-value store is for. You want support for using environment variables to override export options instead, which is already available as GODOT_SCRIPT_ENCRYPTION_KEY for the encryption key. (Despite having SCRIPT in its name, it affects the whole PCK encryption, as script-only encryption was replaced with whole PCK encryption in 4.0.)

This is also supported for Android export options: https://docs.godotengine.org/en/stable/tutorials/export/exporting_for_android.html#environment-variables