SuperDuperDB / superduperdb

🔮 SuperDuperDB: Bring AI to your database! Build, deploy and manage any AI application directly with your existing data infrastructure, without moving your data. Including streaming inference, scalable model training and vector search.
https://superduperdb.com
Apache License 2.0
4.55k stars 445 forks source link

[CORE-0.3] Serialize classes so that they are accessible from all services #2130

Closed blythed closed 2 days ago

blythed commented 1 month ago

Currently, if a user creates his own custom Component:

class MyComponent(Component):
    ...

then, this code needs to be accessible in the installation of all environments using MyComponent. It would be good to remove this requirement. For instance, the source code code be serialized in the db.metadata, along with the parameters.

Possible solution

  1. Serialize class with dill
  2. Serialize the __repr__ of the class, to give users something to see
  3. Add this serialization to metadata store with _BaseEncodable and base64 encoding
  4. Reload the class definition to an in-memory module
  5. Save the parameters and artifacts as normal