search

dbt-labs / dbt-labs-experimental-features

dbt support for database features which are not yet supported natively in dbt-core
Apache License 2.0
145 stars 43 forks source link

Improve installation options / documentations for materialized views #20

Closed Luttik closed 5 months ago

Luttik commented 2 years ago

The materialized-views readme. Very briefly addresses the essential step of the setup. Both seem problematic.

Copy-past into the macro's folders

This step could much more descriptive. But, more importantly, it doesn't seem to work. I've both tried the

macros
└── materialized-views
    └── bigquery
        └── adapters.sql
        └── materialized_view.sql
    └── default
        └── adapters.sql
        └── materialized_view.sql

as well as 

macros
└── bigquery
    └── adapters.sql
    └── materialized_view.sql
└── default
    └── adapters.sql
    └── materialized_view.sql

Both result in the following issue (before using the macro's anywhere)

Running with dbt=0.21.0
Encountered an error:
Compilation Error
  In adapter.dispatch, got a string packages argument ("None"), but packages should be None or a list.

Installation as a package

In my eyes the installation options for a package are from best & simples to worst & most problematic

  1. Installation via dbt hub
  2. Installation via github
  3. Installation via local files

Only the latter is currently supported. It would be great if we get more options for this as well, preferably the dbt hub one.

jtcohen6 commented 2 years ago

Hey @Luttik, thanks for your interest here! Two quick notes:

This feature is still "experimental": There are some known limitations with the implementation in this repository, as well as uncertainty about the intended use for / long-lived limitations of materialized views in each data warehouse. As such, we haven't added it as an officially supported package on the dbt Hub, and the code continues to be "use at your own risk." There's a bit of meta discussion about the future state of "experiments" in https://github.com/dbt-labs/dbt-utils/discussions/487.

When we are ready to officially support materialized views, I think the most likely move will be to dbt-core + adapter plugins. We know folks have wanted this for some time (https://github.com/dbt-labs/dbt-core/issues/1162, https://github.com/dbt-labs/dbt-bigquery/issues/18), so it's a question of feature maturity + readiness. In the meantime, feedback we get on the "experimental" implementation in this repo is a big help!

Luttik commented 2 years ago

Ah thanks this is a great response.

First, I tried everything I mentioned with the code from master as it was yesterday. Since #16 was merged on Januari the 5th that didn't solve this issue.

The subdirectory package config is a great addition. I'll propose a pr that extends the installation part a bit.

I understand that stuff like materialized views but also snowflake's streams and tasks is quite hard since the concepts vary wildly from tool to tool, but I also think that missing these features are the biggest argument in favor of rolling your own solution instead of using a solid framework like DBT. Therefore, I love any progress towards this goal, even if it is just experimental or at your own risk.

brabster commented 2 years ago

Hey there, not sure whether this is the best place to raise a problem with the install instructions, but I'm unable to install the package via git subdirectory:

packages.yml:

packages:
  - git: https://github.com/dbt-labs/dbt-labs-experimental-features
    subdirectory: materialized-views
> dbt deps
13:30:49  Running with dbt=1.0.1
13:30:50  Encountered an error:
Error checking out spec='None' for repo https://github.com/dbt-labs/dbt-labs-experimental-features
Cloning into '5b2c5462b916f7e61ceea92dbb107d2b'...
fatal: cannot change to 'https://github.com/dbt-labs/dbt-labs-experimental-features': No such file or directory
error: failed to initialize sparse-checkout

with debug:

3:23:28.123702 [debug] [MainThread]: STDOUT: "b'git version 2.25.1\n'"
13:23:28.124169 [debug] [MainThread]: STDERR: "b''"
13:23:28.124724 [debug] [MainThread]: Executing "git clone --depth 1 --filter=blob:none --sparse https://github.com/dbt-labs/dbt-labs-experimental-features 5b2c5462b916f7e61ceea92dbb107d2b"
13:23:28.141869 [debug] [MainThread]: STDOUT: "b''"
13:23:28.142548 [debug] [MainThread]: STDERR: "b"Cloning into '5b2c5462b916f7e61ceea92dbb107d2b'...\nfatal: cannot change to 'https://github.com/dbt-labs/dbt-labs-experimental-features': No such file or directory\nerror: failed to initialize sparse-checkout\n""

Am I doing something obvious wrong?

jtcohen6 commented 2 years ago

@brabster I think this may be an issue with your installed git version. Check out the convo in https://github.com/dbt-labs/dbt-labs-experimental-features/issues/25

brabster commented 2 years ago

@jtcohen6 thank you that was the problem!

dataders commented 5 months ago

closed by #20 (i think). regardless I'm closing