search

Datavault-UK / automate-dv

A free to use dbt package for creating and loading Data Vault 2.0 compliant Data Warehouses (powered by dbt, an open source data engineering tool, registered trademark of dbt Labs)
https://www.automate-dv.com
Apache License 2.0
511 stars 131 forks source link

[FEATURE] support duckdb #171

Open chinahamu opened 1 year ago

chinahamu commented 1 year ago

Is your feature request related to a problem? Please describe.

A situation arose where I wanted to use duckdb in local development. I added duckdb as it does not seem to be currently supported.

Describe the solution you'd like

Please merge here https://github.com/Datavault-UK/dbtvault/pull/170

I have confirmed that dbt seed and dbt run work.

AB#5369

adumicic commented 1 year ago

I'm keen to try this with DuckDB. What is the holdup on the merge here? Is there something wrong with the code?

DVAlexHiggs commented 1 year ago

@adumicic Thanks for this question, you raise a really good point and a valid concern.

Our contribution process does not currently allow direct contributions via PRs due to our proprietary development and test environment. We have over 3000 tests which extensively test the AutomateDV macros with different patterns, across all of our currently supported platforms.

We are reluctant to approve PRs which have not gone through the test suite due to the nature of the product (data consistency, following the DV standards, etc.), however as public contributors do not have access to the test suite, it means we essentially cannot accept them. We do take contributions, but at this current time it involves working more collaboratively with us and going through a more formal process.

We want to work towards having a more streamlined and publicly accessible contribution process, as it will benefit the community massively. I do appreciate the current situation is awkward, and it is something we want to resolve sooner rather than later.

For now, leaving PRs open and making them visible to the community is what we are doing to compromise, rather than closing them.

For this particular PR, I did reach out to the author and let them know about the situation, and they were happy for us to leave it open in this way.

fithisux commented 7 months ago

DuckDB is getting more and more useful and popular. I think it makes sense.

dprovder commented 6 months ago

Of course it makes sense. Why wouldn't we want everything running in a python project, without depending on external connections?

ANelson82 commented 5 months ago

+1 for duckdb. I think it will be a great teaching tool to get people into datavault.