search

dbt-labs / dbt-core

dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.
https://getdbt.com
Apache License 2.0
10.01k stars 1.63k forks source link

[Feature] Skip creating a log directory when file logging is disabled #10948

Open ericfreese opened 3 weeks ago

ericfreese commented 3 weeks ago

Is this a new bug in dbt-core?

Current Behavior

I ran into this while deploying DBT core in an environment where dbt's current working directory is not writeable. I had set DBT_LOG_PATH to a separate writeable directory, but then realized recently that we actually want to entirely disable DBT's file logging because we store the stdout/stderr output separately.

After some digging through the source (because I couldn't find documentation of how to disable the file logging), I found this line:

https://github.com/dbt-labs/dbt-core/blob/dd77210756e6432ef2737ab59a65c238e3abfdda/core/dbt/events/logging.py#L95

A bit more digging and I found that I could set a DBT_LOG_LEVEL_FILE=none environment variable to effectively disable the file logging. I figured that after disabling file logging, I would no longer need to set DBT_LOG_PATH so I removed that environment variable.

Upon testing, I was surprised to see that dbt debug now provided no output and only exited with error code 2. I'd run into this issue before when it couldn't open the log file for writing, so I added DBT_LOG_PATH=/tmp back to the environment, and it ran successfully.

Expected Behavior

It seems to me that if file logging is disabled, dbt should not try to make the logging directory.

Steps To Reproduce

  1. In an environment where DBT cannot make a directory in its current working directory (you may also be able to make the mkdir fail by creating logs as a regular file rather than a directory e.g. touch logs)
  2. With env vars: 
    DBT_TARGET_PATH=/tmp # or some writeable directory
DBT_PACKAGES_DIR=/tmp # or some writeable directory
DBT_LOG_LEVEL_FILE=none
  3. Run dbt debug
  4. See no output and exit code 2 (via echo $?)

Instead, I would expect dbt debug to succeed and print its output. Nothing should be logged to the file system.

Relevant log output

No response

Environment

- OS: Amazon Linux 2023
- Python: 3.11.6
- dbt: 1.8.7

Which database adapter are you using with dbt?

postgres

Additional Context

I dug back through the git history a bit and I think the make_log_dir_if_missing call should have been moved under the if statement when the none option was added in commit a2213abbc0ba932ab164160ce98f8b2c63e74d5c

dbeatty10 commented 3 weeks ago

Thanks for reporting this and digging into the git history @ericfreese 🤩

At the very least, there should be documentation that describes the LOG_LEVEL_FILE config, so I've opened this docs issue: https://github.com/dbt-labs/docs.getdbt.com/issues/6404

I don't think we ever intentionally designed or documented dbt to not create a logging directory in this scenario, so I'm going to switch this to a feature request for further consideration for your PR: https://github.com/dbt-labs/dbt-core/pull/10949