search

dm4ml / motion

Framework for building and maintaining self-updating prompts for LLMs
https://dm4ml.github.io/motion/
58 stars 4 forks source link

Enable switchable backends for storage #238

Open shreyashankar opened 1 year ago

shreyashankar commented 1 year ago

Redis will only take us so far, when sometimes it can be impractical to store everything in-memory. We want to support multiple backends---at least Redis and RocksDB for now, and if it becomes clearer to integrate another KV store-like DB, or if we need a relational model, we can consider other DBMSes.

shreyashankar commented 1 year ago

We can also have a Redis layer + RocksDB layer

shreyashankar commented 11 months ago

Can also consider s3 & filesystem

shreyashankar commented 11 months ago

We should have a storage interface, such that any storage system can be used for the backend.