faros-ai / faros-community-edition

BI, API and Automation layer for your Engineering Operations data
https://community.faros.ai
Apache License 2.0
423 stars 58 forks source link

Support dbt transforms in Airbyte #71

Closed thomas-gerber closed 2 years ago

thomas-gerber commented 2 years ago

Is your feature request related to a problem? Please describe. As a Faros CE user, I want to apply transformations on what Airbyte writes against the Faros Schema to capture my idiosyncrasies.

Describe the solution you'd like Support for dbt transforms in Airbyte: https://docs.airbyte.com/operator-guides/transformation-and-normalization/transformations-with-airbyte This would allow the ability to write lightweight transforms in SQL for one's own consumption, but also publish and share those with the community for common use cases. For example, transforming tasks that have a certain label or type into incidents.

Describe alternatives you've considered It is possible for Analysts in Metabase to transform the data with SQL, but they cannot write it back to the database. dbt transforms during ingestion allows users to leverage canned dashboards better, and provide better separation of concerns between the analyst and data engineer personas.

Additional context

thomas-gerber commented 2 years ago

Airbyte now supports this! Updated the documentation: https://community.faros.ai/docs/data-transforms-with-dbt.