Feature Request: Support for Dynamic Profile Switching and Default Model Profiles in dbt build Command

[tommyhutcheson]

Is this your first time submitting a feature request?

• [X] I have read the expectations for open source contributors
• [X] I have searched the existing issues, and I could not find an existing issue for this feature
• [X] I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

Describe the feature

Overview: Currently, dbt allows setting a default profile in dbt_project.yml, but lacks the ability to dynamically switch profiles within a single run of dbt build. Additionally, there is no provision to set default profiles for specific models or directories.

Use Case: In our dbt project, we manage multiple datasets across different environments in BigQuery. Each dataset may require specific configurations and permissions. We would like to run models with different profiles based on these requirements within the same dbt build command.

profile.yml

my_dbt_project:
  target: dev
  outputs:
    dev:
      dataset: shop
      job_execution_timeout_seconds: 600
      job_retries: 1
      location: EU
      method: oauth
      priority: interactive
      project: project-dev
      threads: 10
      type: bigquery

    explore:
      dataset: bakery
      job_execution_timeout_seconds: 600
      job_retries: 1
      location: EU
      method: oauth
      priority: interactive
      project: project-explore
      threads: 10
      type: bigquery

Then within the dbt_project.yml we could set the default profile for a specific model

models:
  my_dbt_project:
    model_a:
      +materialized: view
    model_b:
      +materialized: view
      +profile: explore

so now when I run dbt build I would like to build the model_a model within project-dev and model model_b will be built within project-explore.

Describe alternatives you've considered

No response

Who will this benefit?

Benefits: Dynamic profile switching would streamline our deployment process, enabling us to manage development, staging, and production environments more effectively. The ability to set default profiles for specific models or directories would simplify configuration management and improve workflow flexibility.

Are you interested in contributing this feature?

No response

Anything else?

No response

[yu-iskw]

I am for this idea. There are some segregated dbt Models in our dbt project. We choose a profile based on the model selection. So, we have to separate the command execution today. However, if we can dynamically switch profiles based on the model configs, we can aggregate the commands. Specifically, we can dynamically switch the permission to run partial dbt models in the same dbt command with the same service account using service account impersonation.

# profiles.yml
default:
  outputs:
    model_group1:
      type: bigquery
      method: oauth
      project: "xxxx"
      impersonate_service_account: "dbt-model-group1@example-google-project.iam.gserviceaccount.com"
    model_group2:
      type: bigquery
      method: oauth
      project: "xxxx"
      impersonate_service_account: "dbt-model-group1@example-google-project.iam.gserviceaccount.com"

# dbt_project.yml
models:
  my_dbt_project:
    model_group1:
      +profile: model_group1
    model_b:
      +profile: model_group2

[j0pr8]

Great idea!