Open Verneri opened 2 months ago
There are some constraints on the data format that might make such usage difficult:
dep_...
properties to represent dependencies, as part of the public API for tasksThat said, I'm interested in what you have in mind. A concrete case for storing task data in a different backend would help to clarify where these interfaces need to be changed.
Note also #372 is going to happen soon, and will modify the TC API pretty substantially.
I'll try to Create an example, it might take some time though
I will try to implement most of the system straight into SurrealDB to showcase the possibilities. Of course it might sometimes be better to use dedicated code for this but i think creating as much funtionality as possible, straight in the db shows the ideas better.
Also was thinking of trying out using BerkleyDB or rocksDB etc for the storage layer just to see if there is difference in the performance
Sounds good!
It might make more sense to create a struct that implements the Storage
(and StorageTxn
) traits, instead of a replica.
The set of methods provided in StorageTxn
probably deserves some scrutiny: lots of very common things, such as determining whether a task is BLOCKED or BLOCKING, currently require scanning all pending tasks, which is pretty expensive.
Let's see if this bug branches in either of those directions: creating a new storage backend using SurrealDB; or modifying the StorageTxn
methods to improve performance.
As some DBMSs (like SurrealDB) might be able to represent for example dependencies straight from the DB it might be a good idea to abstract the
Replica
andDependencyMap
to traits so that they can be given other Solutions than the way Rust implementation handles calculating dependencies, it might be a good idea to make Traits for these.In some cases this might help as Well with bindings to other languages/platforms