InfuseAI / piperider

Code review for data in dbt
https://www.piperider.io/
Apache License 2.0
480 stars 23 forks source link

Is this not available in dbt 1.6? #854

Closed pei0804 closed 1 year ago

pei0804 commented 1 year ago

Is your feature request related to a problem? Please describe.

The new manifest format does not seem to be supported. I am developing in the following environment. Do you plan to support this in the future?

❯ poetry run dbt --version
Core:
  - installed: 1.6.0
  - latest:    1.6.0 - Up to date!

Plugins:
  - snowflake: 1.6.0 - Up to date!
❯ piperider compare-reports --base /tmp/piperider-base/run.json --target /tmp/piperider-target/run.json -o /tmp/piperider-comparison
────────────────────────────────────────────────────────────────────────────────── Comparison report ──────────────────────────────────────────────────────────────────────────────────
Selected reports:
  Base:   /tmp/piperider-base/run.json
  Target: /tmp/piperider-target/run.json
Warning: Runtime Error
  Expected a schema version of "https://schemas.getdbt.com/dbt/manifest/v9.json" in input file, but found "https://schemas.getdbt.com/dbt/manifest/v10.json". Are you running with a
different version of dbt?. Got problem to generate changeset.
popcornylu commented 1 year ago

We will support dbt 1.6 in the next release (ETA 8/24)

Please try the nightly version as a temporary solution, in which dbt 1.6 support has already been added

$ pip install piperider-nightly   
$ piperider version                                                                                                                                                                                        
0.32.0.20230815
GitCommit: bf26319bb87dd83859d623d2aba5ad95b774f335
pei0804 commented 1 year ago

Change detection is working, but no summary report is generated.

Column additions are detected. image

It does not appear in the summary report. image

Execution context

piperider version.

piperider version
0.32.0.20230815
GitCommit: bf26319bb87dd83859d623d2aba5ad95b774f335

at main branch.

PIPERIDER_SKIP_TARGET_CHECK=1 piperider run --dbt-target-path target/ -o /tmp/piperider-base

at future branch.

PIPERIDER_SKIP_TARGET_CHECK=1 piperider run --dbt-target-path target/ -o /tmp/piperider-target
piperider compare-reports --debug --base /tmp/piperider-base/run.json --target /tmp/piperider-target/run.json -o /tmp/piperider-comparison
────────────────────────────────────────────────────────────────────────────────── Comparison report ──────────────────────────────────────────────────────────────────────────────────
Selected reports:
  Base:   /tmp/piperider-base/run.json
  Target: /tmp/piperider-target/run.json
Warning: Runtime Error
  Expected a schema version of "https://schemas.getdbt.com/dbt/manifest/v9.json" in input file, but found "https://schemas.getdbt.com/dbt/manifest/v10.json". Are you running with a
different version of dbt?. Got problem to generate changeset.

Comparison report: /tmp/piperider-comparison/index.html

Soulution

So, I changed the dbt_schema_version in the following file from https://schemas.getdbt.com/dbt/manifest/v10.json to https://schemas.getdbt.com/dbt/ manifest/v9.json and now I get a summary report.

Selected reports:
  Base: /tmp/piperider-base/run.json
  Target: /tmp/piperider-target/run.json
popcornylu commented 1 year ago

Thank you for your feedback, we will troubleshoot the problem you mentioned.

popcornylu commented 1 year ago

Hi, after investigation, i can reproduce your case only when

  1. base report is https://schemas.getdbt.com/dbt/ manifest/v9.json
  2. target report is https://schemas.getdbt.com/dbt/ manifest/v10.json
  3. The environment to run piperider is dbt-core=1.5.x

To solve this, just also upgrade the dbt package to 1.6 in your environment running the PipeRider

Here is my reproducing steps.

Run compare-reports in dbt 1.5.5, target report is generated in dbt 1.6.0 environment.

$ pip list | grep 'piperider\|dbt'
dbt-core                  1.5.5
dbt-duckdb                1.5.2
dbt-extractor             0.4.1
piperider-nightly         0.32.0.20230817

[notice] A new release of pip is available: 23.0.1 -> 23.2.1
[notice] To update, run: pip install --upgrade pip

$ piperider compare-reports --last
 Comparison report 
Selected reports:
  Base:   /private/tmp/jaffle_shop/.piperider/outputs/dev-20230818082235/run.json
  Target: /private/tmp/jaffle_shop/.piperider/outputs/dev-20230818082336/run.json
Warning: Runtime Error
  Expected a schema version of "https://schemas.getdbt.com/dbt/manifest/v9.json" in input file, but found "https://schemas.getdbt.com/dbt/manifest/v10.json". Are you running with a different version of dbt?. Got problem to generate changeset.

Comparison report: /private/tmp/jaffle_shop/.piperider/comparisons/latest/index.html

Upgrade to dbt 1.6

$ source ./dbt16/bin/activate
$ pip list | grep 'piperider\|dbt'3.0.0
dbt-core                 1.6.0
dbt-duckdb               1.6.0
dbt-extractor            0.4.1
dbt-semantic-interfaces  0.2.0
piperider-nightly        0.32.0.20230817

[notice] A new release of pip is available: 23.0.1 -> 23.2.1
[notice] To update, run: pip install --upgrade pip

$ piperider compare-reports --last
Comparison report
Selected reports:
  Base:   /private/tmp/jaffle_shop/.piperider/outputs/dev-20230818082235/run.json
  Target: /private/tmp/jaffle_shop/.piperider/outputs/dev-20230818082336/run.json

Impact Summary:
  Code Changes: added=0, removed=0, modified=0
  Resource Impact: potentially_impacted=0, assessed=0, skipped=0, impacted=0

Comparison report: /private/tmp/jaffle_shop/.piperider/comparisons/latest/index.html
pei0804 commented 1 year ago

I think i can solve this problem. Great thank you.