superduper-io / superduper

Superduper: Build end-to-end AI applications and agent workflows on your existing data infrastructure and preferred tools - without migrating your data.
https://superduper.io
Apache License 2.0
4.82k stars 466 forks source link

[DATATYPE] Create base `DataType` without all parameters and use it for "switch" `Vector` type #2629

Closed blythed closed 4 days ago

blythed commented 6 days ago

The base Datatype contains many extraneous parameters, which we don't necessarily need (encoder for example). We should add an intermediate class in the hierarchy to solve this, which we can subclass with Vector.

Vector will have its encode implementation based on this suggestion: https://github.com/superduper-io/superduper/issues/2562#issuecomment-2454060574