superduper-io / superduper

Superduper: Build end-to-end AI applications and agent workflows on your existing data infrastructure and preferred tools - without migrating your data.
https://superduper.io
Apache License 2.0
4.83k stars 467 forks source link

[DATATYPE] Migrate `CFG.bytes_encoding` to databackend #2628

Closed blythed closed 1 week ago

blythed commented 1 week ago

This prevents the need to reconfigure on a case-by-case basis.

class Databackend:
    bytes_encoding = 'bytes'

class IbisDatabackend:
    def __init__(self):
        if self.uri.startswith('snowflake'):
            self.bytes_encoding = 'str'
        ...

TODO: only do conversion between bytes and str if encodable == 'encodable'.