SuperDuperDB / superduperdb

🔮 SuperDuperDB: Bring AI to your database! Build, deploy and manage any AI application directly with your existing data infrastructure, without moving your data. Including streaming inference, scalable model training and vector search.
https://superduperdb.com
Apache License 2.0
4.54k stars 444 forks source link

[PORTABILITY] Installable plugins #2170

Open blythed opened 2 weeks ago

blythed commented 2 weeks ago

We would like to not be restricted to pre-installed code in the Dockerfile/ environment. Ideas:

@jieguangzhou to provide additional ideas.

blythed commented 2 weeks ago

After design discussion with @jieguangzhou and @kartik4949 we have decided on something like this:

class Plugin:
    _artifacts = [('cache_path', file)]
    path: '<some_local_path>'
    cache_path: t.Optional[str] = ''
    install_on_load: bool = False

    def __post_init__(self):
        if not self.cache_path:
            self.cache_path = '~/.superduperdb/plugins'

    def on_create(self, db):
        db.artifact_store.put_file(self.path)

    @ensure_initialized
    def install(self):
        os.system('pip install -r ~/.superduperdb/plugins/{self.identifier}')

    def on_load(self, db):
        sys.path.append('~/.superduperdb/plugins')
        if self.install_on_load:
            self.install()

On building connection, potentially load plugins:

def build_datalayer(...):
    ...  # existing things
    if CFG.load_plugins:
        for plugin in db.show('plugin'):
            db.load('plugin', plugin)