Closed jwills closed 1 year ago
Not sure if this exist yet but... is there a world where dbt sources support community made plugins? E.g: Write a subclass of Source
with the proper methods (casting to DataFrame/Arrow/...) and you got your source implemented.
@davidgasquez there isn't, but I think that's what this feature needs to support for it to be broadly useful so I was going to try to wire it up
Essentially, any external file format or storage system that can read/write a
pandas.DataFrame
(or an arrow record reader, or a polars data frame, or a JSON/CSV file) and that has Python library support should be usable as either a) a source or b) an external materialization target in dbt-duckdb.Reference: dbt-excel.com