dbt-labs / dbt-common

Apache License 2.0
13 stars 15 forks source link

Support DIFF as a new mode for recording #149

Closed emmyoop closed 5 months ago

emmyoop commented 5 months ago

resolves #144

Description

Adds a DIFF mode to record.

Expected format: DBT_RECORDER_MODE=DIFF DBT_RECORDER_FILE_PATH=recording_old.json dbt run --no-partial-parse

Diff will record using the same types as the input file and then diff the two files. It ouputs a diff.json with the results. Current state is just json that's reasonably readable but not what we'd necesary want to shwo as output. But we could parse it out later.

example `diff.json`

``` { "GetEnvRecord": {}, "LoadFileRecord": { "values_changed": { "root[13]['result']['contents']": { "new_value": "select 1 as some new id", "old_value": "select 1 as id" }, "root[24]['result']['contents']": { "new_value": "-- Edit the comments: Use the `ref` function to select from other models\n\nselect *\nfrom {{ ref('my_first_dbt_model') }}\nwhere id = 1", "old_value": "-- Use the `ref` function to select from other models\n\nselect *\nfrom {{ ref('my_first_dbt_model') }}\nwhere id = 1", "new_path": "root[23]['result']['contents']", "diff": "--- \n+++ \n@@ -1,4 +1,4 @@\n--- Use the `ref` function to select from other models\n+-- Edit the comments: Use the `ref` function to select from other models\n \n select *\n from {{ ref('my_first_dbt_model') }}" } }, "iterable_item_removed": { "root[23]": { "params": { "path": "/Users/emily/projects/basic-dbt/models/example/my_first_dbt_model.sql", "strip": true }, "result": { "contents": "/*\n Welcome to your first dbt model!\n Did you know that you can also configure models directly within SQL files?\n This will override configurations stated in dbt_project.yml\n\n Try changing \"table\" to \"view\" below\n*/\n\n{{ config(materialized='table') }}\n\nwith source_data as (\n\n select 1 as id\n union all\n select null as id\n\n)\n\nselect *\nfrom source_data\n\n/*\n Uncomment the line below to remove records with null `id` values\n*/\n\n-- where id is not null" } } } }, "FindMatchingRecord": { "iterable_item_removed": { "root[0]": { "params": { "root_path": "/Users/emily/projects/basic-dbt", "relative_paths_to_search": [ "macros" ], "file_pattern": "[!.#~]*.sql" }, "result": { "matches": [ { "searched_path": "macros", "absolute_path": "/Users/emily/projects/basic-dbt/macros/lots_of_macros.sql", "relative_path": "lots_of_macros.sql", "modification_time": 1717083600.9987388 } ] } } } }, "WriteFileRecord": {}, "QueryRecord": {} } ```

Checklist

github-actions[bot] commented 5 months ago

Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the contributing guide.

github-actions[bot] commented 5 months ago

Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the contributing guide.

codecov-commenter commented 5 months ago

Codecov Report

Attention: Patch coverage is 21.79487% with 61 lines in your changes missing coverage. Please review.

Project coverage is 63.03%. Comparing base (e30f65f) to head (aef3c30).

Files Patch % Lines
dbt_common/record.py 21.79% 61 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #149 +/- ## ========================================== - Coverage 64.04% 63.03% -1.01% ========================================== Files 51 51 Lines 3151 3222 +71 ========================================== + Hits 2018 2031 +13 - Misses 1133 1191 +58 ``` | [Flag](https://app.codecov.io/gh/dbt-labs/dbt-common/pull/149/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dbt-labs) | Coverage Δ | | |---|---|---| | [unit](https://app.codecov.io/gh/dbt-labs/dbt-common/pull/149/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dbt-labs) | `63.03% <21.79%> (-1.01%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dbt-labs#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.