superduper-io / superduper

Superduper: Integrate AI models and machine learning workflows with your database to implement custom AI applications, without moving your data. Including streaming inference, scalable model hosting, training and vector search.
https://superduper.io
Apache License 2.0
4.66k stars 449 forks source link

[PRE-RELEASE-0.4] Remove random mixin components which aren't necessary #2430

Open blythed opened 2 weeks ago

blythed commented 2 weeks ago
blythed commented 1 day ago

Plan for simplifying the queuing architecture:

  1. Each Table has a flag triggers=True/False.
  2. When a Trigger component is created, it will add this flag to the Table if not already there
  3. When an insert/ update/ delete comes in, we look in the meta-data store to see if triggers is True or not for query.table.
  4. The ids are sent to the table queue
  5. We iterate from children upwards through components which are triggered by that table in the consumer, running Component.run_jobs for each component.
  6. When db.apply is called, each event is sent to the apply queue (only one queue).
  7. The consumer of the db.apply queue runs Component.run_jobs for each event in the queue.