databricks / bundle-examples

Examples of Databricks Asset Bundles
Other
55 stars 20 forks source link

Inaccurate reference to Databricks.yml? #25

Open johalnes opened 2 months ago

johalnes commented 2 months ago

Hi,

I'm working on deploying a DBT project trough Databricks Asset bundles, but somewhat confused on how to get environment variables into the task. In this example, how is {{ env_var("DBT_ACCESS_TOKEN") }} defined as an environment variable?

Is the databricks.yml configuration working as it is now? I get it to work without passing environment variables, but setting job parameters doesn't seem to work.

Would be super grateful for some help 🙂

PatrickLeahey commented 2 months ago

Hey @johalnes you will need to create an environment variable for your GitHub repo by going to settings > secrets and variables > actions

Create a variable called DBT_ACCESS_TOKEN and set it to your Databricks PAT

{{ env_var('DBT_ACCESS_TOKEN') }} specifically parses a GitHub Actions env var

more info on PAT auth

johalnes commented 2 months ago

Thanks for replying @PatrickLeahey ! But not really what I was wondering about.

I know how secrets and environment variables within Github works. But what is the link between environment variables and how can one use these within Workflows? I've created a environment variable {{ env_var("DBT_SCHEMA") }}. And even when defined as workflow parameters, and defined in my environment where I run databricks bundle deploy , Databricks Workflows don't seem to get this variable.

How would you defined this without using --vars "{ dev_schema: ${workspace.current_user.short_name} }"' like in this example repo?

johalnes commented 2 months ago

The inaccurate part of the headline was regarding you writing :

# The workspace host / token are provided by Databricks
# see databricks.yml for the host used for 'dev' 

in the code, without any references to these variables within the actual databricks.yml file 🙂