search

StarRocks / dbt-starrocks

dbt-starrocks contains all of the code enabling dbt to work with StarRocks
16 stars 6 forks source link

Support External Catalogs in DBT Adapter #22

Open JadCham opened 1 year ago

JadCham commented 1 year ago

Feature request

Is your feature request related to a problem? Please describe.

No, but a blocker to use the DBT adapter.

Describe the solution you'd like

The SR DBT Connector supports reading from external tables/catalogs and writing materialized tables into SR internal storage.

We plan to use SR for querying our external Hive tables with disk caching instead of Presto/Athena. We have many processes around using DBT to transform data in Athena. It would be great to be able to replace them with SR instead.

Describe alternatives you've considered

No alternatives at the moment, DBT runs happen in Athena and we read the resulting data in SR.

geoHeil commented 7 months ago

Ideally nosql systems such as Elastic or Kafka can also be supported and even the ingestion from kafka controlled form DBT

alberttwong commented 7 months ago

@wangyouzhuo for visibility

alberttwong commented 6 months ago

I saw that there was a readme section on supporting hive external catalog. Can you try it out?