Consider adding passthrough columns

[kristin-bagnall]

In some instances, customers use multiple shopify connectors. Our solution is to union all of the data together prior to running the transformation, however there may not be a way to differentiate the different connectors/shops. Users would need to add an additional column to do this, and we would then need to allow customers to pass through these columns to the final models.

Related slack convo: https://getdbt.slack.com/archives/C0VLZPLAE/p1610571493343300

[MartinGuindon]

I'd be happy to work on this and contribute a PR as part of the project I'm working on. I think this should be pretty straightforward to implement. Here's the solution I'm thinking:

• introduce a configuration variable where a developer can set the name of one or more columns that are common to all tables, eg.:

vars:
    shopify_passthrough_columns: ["shop_identifier", "shop_url"]

• in each model, add a Jinja condition that checks if passthrough columns are defined in the project config, and if so, loop over them to add them in the select statement.

Any thoughts?

[kristin-bagnall]

@MartinGuindon that would be great! A good example of how we've done this in the pass is in our salesforce model.

Once you identify the models you'd like to include passthrough columns to, the steps include the following. I've linked examples of this work for each item.

Source package:

• Update dbt_project.yml file to include passthrough variables
• Update staging models to utilize passthrough columns. Note this needs to be done in two places - here and here. - Update README configuration to call out use of passthrough columns

Transform package:

• Update dbt_project.yml file to include passthrough columns
• Update any applicable models so that the passthrough columns flow through to final models. Example of final model here.
• Update README

[MartinGuindon]

Cool, pretty much the solution I had in mind, except with a per-model config which makes it even more flexible.