dpc
commented
1 year ago I was thinking that we should put modules data in a pre-prefix.
So we would hava a global u8 enum for everything, where one thing would be:
#[repr(u8)]
enum Prefix {
//...
Module,
// ...
}and then database would handle it somehow to prefix all operations from a module with Prefix::Module (1 bytes), ModuleKey (soon: ModuleInstanceId) (2 bytes) both on reads and writes.
This would isolate the modules from each other and put all the module data behind a common prefix.
We might create a wrapper struct ModuleDatabaseHandle and inject that instead of Database or DatabaseTransaction to modules. Internally it would just add these 3 bytes everywhere, so the module can't touch anything outside of its range.
m1sterc001guy
We don't currently have a great way of iterating through all of the keys in the database (i.e SELECT * in SQL land). This is because different parts of the code have reserved key prefixes, so to iterate through all the keys we need to know about all of the DbKeyPrefix enums in different parts of the code. This is what dbdump does currently.
I want to iterate through all the keys so that I can implement the db migrations by copying all the key/value pairs to a new db (we can also support an in-place upgrade too, but making a new db is safer).
One way to do this would be to refactor all of the prefixes into a giant enum and simply iterate through that. This is not great for modularization though since it requires all modules to reserve a key prefix space within the database.
A better solution is to use RocksDB's column families. We can partition the database for each module (each module can be a different column family). That means that key prefixes can then overlap between modules and we will still be able to read the key/value pairs properly from the database.
For the client, if its using SQLite instead of RocksDB, we can use different tables instead of Column Families.