search

confluentinc / ksql

The database purpose-built for stream processing applications.
https://ksqldb.io
Other
48 stars 1.04k forks source link

Add DBT integration for better tests, documentation, and version-control of data transformations #9061

Open milicictanja opened 2 years ago

milicictanja commented 2 years ago

Is your feature request related to a problem? Please describe. My team is currently using DBT for batch data processing, versioning and documenting of our data transformations. The tool is doing a great job in batch processing, however it can't work with streams of data. KsqlDB is a great tool that can help us process real-time data, however we would need to migrate all of our models to ksqlDB format. It would be great if ksqlDB would have dbt adapter that would translate dbt models into ksqlDB streams. Another similar tool Materialize supports that, however it doesn't have the option for self-hosting.

Describe the solution you'd like Implement a DBT adapter similar to dbt-materialize

lcorneliussen commented 2 years ago

+1

Would love a good solution of using source control to maintain (and test + document) ksql. Dbt looks like a very natural fit.

daigotanaka commented 1 year ago

Hi all,

I started experimenting dbt-ksql here: https://github.com/anelendata/dbt-ksql I also attached an example project: https://github.com/anelendata/dbt-ksql/tree/main/example

I just coded this in a day, so it's still experimental, but my productivity has been boosted for sure.

Feedback and collaborations are welcome :)

jordanhamel commented 2 months ago

Any updates or thoughts on this open issue? @agavra thanks