chipsenkbeil / entity-rs

A simplistic framework based on TAO, Facebook's distributed database for social graph
https://entity.rs
42 stars 5 forks source link

Support custom indexer for non-SQL databases #50

Open chipsenkbeil opened 3 years ago

chipsenkbeil commented 3 years ago

Need to devise some sort of custom indexer interface that can be used to support faster lookup of ent ids.

Use case in mind is for vimwiki-server where each ent has a region associated with it to denote where it is in a file. Before swapping to entity-rs, a special structure was maintained to provide fast lookup given an offset into the appropriate location (by using a tree).

The challenge is having a generic-enough indexer and how to pass it around as Rust isn't dynamic. We'd need to be able to attach an indexer type to the field attribute and then the database would know to create an indexer of that type for the given field.

Maybe an enum of types would be best such as

pub enum Indexer {
    Hash, // Will create a  hashmap of a field's value using `Value` as key and `Vec<Id>` as value
    Geo, // Will create a kd-tree of a field's value -- to support Region from vimwiki, we'd need a 3D search grid... how can we make this generic enough to enable someone to specify? 
}

Maybe the best way is to have a trait that is an indexer? It would take a Value as an argument and provide a Vec<Id> as the output?

trait Indexer {
    fn insert(&mut self, value: Value, id: Id);

    fn get(&self, value: Value) -> Vec<Id>;

    fn remove(&mut self, value: Value, id: Id) -> Option<Id>;
}
chipsenkbeil commented 3 years ago

If we go the trait route, we need some way to go from a type on a field to an instance in the database.

Only thought here right now is to hard-code the types and explicitly reference that in the databases themselves.

pub enum IndexerType {
    Hash,
    Geo, // We'd need some way to know how to convert Value -> something useful
}
chipsenkbeil commented 3 years ago

One thought would be to have the index attribute contain a function pointer that, when called, creates a new indexer. As defined here, the fn pointer we define should implement copy so we can pass it around as a parameter to the indexed field attribute.

#[derive(Copy, Clone, ...)]
#[...]
pub enum FieldAttribute {
    Indexed(fn() -> Box<dyn Indexer>),
    Mutable,
}