[Regression] Unable to access environment variables

[DanMawdsleyBA]

Is this a new bug in dbt-core?

• [X] I believe this is a new bug in dbt-core
• [X] I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

Environment variables are no longer working. I was using dbt-core 1.7.3 previously where I use environment varaibles for my profiles.yml file. Since upgrading to 1.8.3 dbt is no longer able to access these same environement variables.

I have my environement varaibles setup in vscode under the settings json file example below:

    "terminal.integrated.env.windows": {
        "local_user":"dan")}

Expected Behavior

Expect DBT to be able to read my environement variables like in the previous version of dbt 1.7 .

Steps To Reproduce

Upgrade to dbt core 1.8 . Use environment variables as part of dbt. In the profiles file example below:

dbt_profile:
  target: snowflake_local
  outputs:
    snowflake_local:
      type: snowflake
      account: ba.eu-west-1
      user: "{{ env_var('local_user') }}"
      authenticator: externalbrowser
      role: "{{ env_var('local_role') }}"
      database: "{{ env_var('local_database') }}"
      schema: "{{ env_var('local_schema') }}"
      threads: 1
      warehouse: DEVELOPER

Relevant log output

dbt parse
10:16:51  Running with dbt=1.8.3
10:16:51  Encountered an error:
Parsing Error
  Env var required but not provided: 'local_user'

Environment

- OS: Windows
- Python: 3.11
- dbt: 1.8.3

Which database adapter are you using with dbt?

snowflake

Additional Context

No response

[dbeatty10]

Thanks for reaching out @DanMawdsleyBA !

Are you sure that the local_user environment variable was defined when you ran dbt 1.8?

I tried this in both 1.7 and 1.8, and I got the same error when it was not defined:

16:54:38  Encountered an error:
Parsing Error
  Env var required but not provided: 'local_user'

But things worked as expected for me in both 1.8 and 1.7 when it was defined.

[DanMawdsleyBA]

``Yes all of my environment variables were setup before and working in dbt 1.7 .

I have setup a new venv running with dbt 1.7 which does work so it only seems to only be introduced when I switch to 1.8.

Logs of 1.7:

dbt parse
08:51:39  Running with dbt=1.7.0
08:51:41  Registered adapter: snowflake=1.7.0
08:51:41  Performance info: C:\Users\u236618\Documents\Github\olympus-utils\tests\sql\snowflake\dbt\jaffle_shop\target\perf_info.json

I'm wondering if the OS or something else is different in your testing. I'm currently running on Windows .

Logs of command on 1.7: 09:58:45.380152 [debug] [MainThread]: running dbt with arguments {'printer_width': '80', 'indirect_selection': 'eager', 'log_cache_events': 'False', 'write_json': 'True', 'partial_parse': 'True', 'cache_selected_only': 'False', 'warn_error': 'None', 'debug': 'False', 'fail_fast': 'False', 'log_path': 'C:\\Users\\u236618\\Documents\\Github\\olympus-utils\\tests\\sql\\snowflake\\dbt\\jaffle_shop\\logs', 'version_check': 'True', 'profiles_dir': 'C:\\Users\\u236618\\Documents\\Github\\olympus-utils\\tests\\sql\\snowflake\\dbt\\jaffle_shop', 'use_colors': 'True', 'use_experimental_parser': 'False', 'no_print': 'None', 'quiet': 'False', 'warn_error_options': 'WarnErrorOptions(include=[], exclude=[])', 'static_parser': 'True', 'invocation_command': 'dbt parse', 'introspect': 'True', 'target_path': 'None', 'log_format': 'default', 'send_anonymous_usage_stats': 'True'}

Same command on 1.8: 09:55:30.356674 [debug] [MainThread]: running dbt with arguments {'printer_width': '80', 'indirect_selection': 'eager', 'write_json': 'True', 'log_cache_events': 'False', 'partial_parse': 'True', 'cache_selected_only': 'False', 'warn_error': 'None', 'debug': 'False', 'version_check': 'True', 'log_path': 'C:\\Users\\u236618\\Documents\\Github\\olympus-utils\\tests\\sql\\snowflake\\dbt\\jaffle_shop\\logs', 'fail_fast': 'False', 'profiles_dir': 'C:\\Users\\u236618\\Documents\\Github\\olympus-utils\\tests\\sql\\snowflake\\dbt\\jaffle_shop', 'use_colors': 'True', 'use_experimental_parser': 'False', 'empty': 'None', 'quiet': 'False', 'no_print': 'None', 'log_format': 'default', 'introspect': 'True', 'warn_error_options': 'WarnErrorOptions(include=[], exclude=[])', 'invocation_command': 'dbt parse', 'target_path': 'None', 'static_parser': 'True', 'send_anonymous_usage_stats': 'True'}

[dbeatty10]

I'm using the zsh shell in macOS to compare v1.7 and v1.8, and loading my environment variables using direnv. It is working the same way in both environments for me.

For clarity, you can see the see the value of the environment variable in both your v1.7 and v1.8 environments when you print out the value using one of the following methods?

I'm wondering if local_user is set in your v1.7 environment, but somehow not set in your v1.8 environment.

Using Command Prompt (cmd):

echo %local_user%

Using PowerShell:

echo $env:local_user

[DanMawdsleyBA]

Here is a screenshot with an echo to the local user here:

[dbeatty10]

@DanMawdsleyBA I'm not sure what's going on here. We did make an update within env_var a handful of months ago within https://github.com/dbt-labs/dbt-core/pull/9489, but nothing is standing out to me that would affect only Windows.

Could you try running each of the following Python scripts to see if each recognizes your local_user environment variable?

Show just one environment variable:

import os

var = "local_user"

print("\nSpecific variable:\n")

if var in os.environ:
    print(f"Environment variable '{var}': {os.environ[var]}")
else:
    print(f"Environment variable '{var}' not found")

Show all environment variables:

import os

print("\nAll environment variables:\n")

for key, value in os.environ.items():
    print(f"{key}: {value}")

[DanMawdsleyBA]

Running the first version here:

Running the second script here:

[aranke]

@DanMawdsleyBA Could it be something to do with the casing? Does "{{ env_var('LOCAL_USER') }}" work?

[DanMawdsleyBA]

That seems to work:

Although strange this behavior only changed between 1.7 and 1.8 .

[dbeatty10]

@DanMawdsleyBA glad that we discovered a workaround by changing the variable name to uppercase.

We're wondering if something about the caching implementation of https://github.com/dbt-labs/dbt-core/pull/9489 lost Windows' case-insensitivity of environment variable names and that explains the difference between v1.7 and v1.8.

In terms of testing, @aranke is going to try something like the following:

Windows:

    "terminal.integrated.env.windows": {
        "local_user":"dan")}

1. {{ env_var("local_user") }} yields "dan"
2. {{ env_var("LOCAL_USER") }} yields "dan"
3. {{ env_var("lOcAl_UsEr") }} yields "dan"

Unix-like:

export local_user=dan

1. {{ env_var("local_user") }} yields "dan"
2. {{ env_var("LOCAL_USER") }} yields ""
3. {{ env_var("lOcAl_UsEr") }} yields ""

[dvanbolt]

I am experiencing the same issue and having to roll back all projects to 1.7. Confirmed with echo $env:x that the variables are loaded into the environment. Is there any path forward or workaround for this?