search

Datavault-UK / automate-dv

A free to use dbt package for creating and loading Data Vault 2.0 compliant Data Warehouses (powered by dbt, an open source data engineering tool, registered trademark of dbt Labs)
https://www.automate-dv.com
Apache License 2.0
511 stars 131 forks source link

[FEATURE] Add support for spark processing #185

Open rishabh1815769 opened 1 year ago

rishabh1815769 commented 1 year ago
  1. We are trying to build lakehouse architecture on AWS S3 using dbtvault and dbt-glue. for this approach we need support with spark processing with dbtvault plugin. Currently, dbtvault only has support for snowflake, bigquery.
  2. Are there any future plans or timelines on this feature?

AB#5367

DVAlexHiggs commented 1 year ago

Hi. Where are you reading that dbtvault only supports BigQuery and Snowflake please? We currently support Postgres and Databricks as well. You should be able to use the databricks macros in place of spark

rishabh1815769 commented 1 year ago

Hi Alex,

Thank you for the quick response. Could you please direct me toward documentation regarding dbtvautl usage with databricks or spark preferably?

Thank you. Regards

On Thu, Feb 16, 2023 at 3:47 PM Alex Higgs @.***> wrote:

Hi. Where are you reading that dbtvault only supports BigQuery and Snowflake please? We currently support Postgres and Databricks as well. You should be able to use the databricks macros in place of spark

— Reply to this email directly, view it on GitHub https://github.com/Datavault-UK/dbtvault/issues/185#issuecomment-1433200036, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOBFD5YF7H5J43CMWCHEGY3WXY4W7ANCNFSM6AAAAAAU6INLOU . You are receiving this because you authored the thread.Message ID: @.***>

DVAlexHiggs commented 1 year ago

Hi Alex, Thank you for the quick response. Could you please direct me toward documentation regarding dbtvautl usage with databricks or spark preferably? Thank you. Regards … On Thu, Feb 16, 2023 at 3:47 PM Alex Higgs @.> wrote: Hi. Where are you reading that dbtvault only supports BigQuery and Snowflake please? We currently support Postgres and Databricks as well. You should be able to use the databricks macros in place of spark — Reply to this email directly, view it on GitHub <#185 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOBFD5YF7H5J43CMWCHEGY3WXY4W7ANCNFSM6AAAAAAU6INLOU . You are receiving this because you authored the thread.Message ID: @.>

There is no databricks specific documentation, as dbtvault uses dbt's multi-dispatch to work cross-platform. You may use dbtvault as-is with a databricks connection for Hubs, Links and Satellites at this time. We will be adding support for more structures in the future