search

Dashlane / dbt-invoke

A CLI for creating, updating, and deleting dbt property files
Apache License 2.0
70 stars 8 forks source link

Modify existing description of the model, if exists #11

Open Gregory108 opened 3 years ago

Gregory108 commented 3 years ago

Current behaviour: dbt-invoke properties --models model_name creates a dedicated model_name.yml file if it does not exist. If the model is already described in some other .yml file, it causes error on dbt test run as dbt cannot handle duplicates.

Desired behaviour:

  1. Either check if a model is already documented somewhere in the project, then enrich it where it is (and as it Is)
  2. Or check if model is already documented somewhere in the project, then crop its description from that file and add it into newly generated model_name.yml

Probably, it should be a toogle option. Some dbt projects stick to "one SQL - one YML" convention and other play by "many SQLs - one schema.yml to rule them all"

robastel commented 3 years ago

Some dbt projects stick to "one SQL - one YML" convention and other play by "many SQLs - one schema.yml to rule them all"

Yes, these seem to be the two predominant conventions when it comes to dbt property files. We happen to be a "one SQL - one YML" shop at Dashlane and created the package with that in mind. Considering our implementation, I believe it would take a serious amount of work to create either of the desired behaviors you mentioned.

I hear where you are coming from on this, but it's not something I anticipate being able to dedicate the necessary time to develop and support.

I'm happy to keep this issue open for a while to gather more thoughts and to see if there is an idea for how to implement it that is less intensive than what I am picturing.

BeadW commented 1 year ago

I have some use cases where we want to handle the "Migration" element of this. IE we have a heap of definitions in schema.yml files and want to migrate to one yaml file per model as a convention.

I have implemented this functionality and am happy to contribute it.

As for making a toggle that you can leave the definition where it is I don't see much issue implementing that at a later date as it is a bit of a natural progression now that the code is there to identify current definitions.

The method I have at the moment maps the results from dbt ls into the manifest.json artifact and works from there.