search

ramonvermeulen / dbt-toolkit

The dbt-toolkit is an early-stage plugin designed to enhance your experience working with dbt-core projects in JetBrains IDEs.
GNU General Public License v3.0
13 stars 0 forks source link

Bug: DBT Not Detected in Project-Specific Python Environment #113

Closed tim-watcha closed 2 weeks ago

tim-watcha commented 1 month ago

Description

The plugin is failing to detect and use DBT installed in project-specific Python environments (e.g., Poetry virtual environments). Instead, it's attempting to execute DBT from the global environment, leading to execution failures when DBT is not installed globally.

Steps to Reproduce

  1. Set up a project using Poetry for dependency management.
  2. Install DBT in the project's virtual environment.
  3. Ensure DBT is not installed in the global Python environment.
  4. Attempt to run a DBT command through the plugin.

Current Behavior

The plugin fails to execute DBT, throwing an error:

java.io.IOException: Cannot run program "dbt" (in directory "/Users/choongjaelee/Codes/kiosk"): error=2, No such file or directory

Expected Behavior

The plugin should detect and use the DBT installation in the project-specific Python environment, allowing successful execution of DBT commands.

Impact

This bug significantly impacts users who manage their project dependencies using tools like Poetry, preventing them from using the plugin effectively and requiring manual workarounds.

Possible Fix

Modify the plugin's environment detection logic to:

  1. First check for DBT in the project-specific Python environment.
  2. If not found, check the project's configured Python interpreter.
  3. Only fall back to the global environment if the above checks fail.

Thank you for your attention to this issue. Resolving it would greatly improve the plugin's usability for many developers using modern Python project management tools.

tim-watcha commented 1 month ago

I tested again, the error message appears only when open the project. After open the project, it seems dbt works correctly.

Anyway thank you for hard working on this project! I'm not familiar with js or kotlin, which is sad.

ramonvermeulen commented 1 month ago

It is very important to have the Python venv that contains the dbt version and executable that you want to use set as project python interpreter (for IntelliJ users this requires installation of the Python IntelliJ plugin).

See logic defined here: https://github.com/ramonvermeulen/dbt-toolkit/blob/ace5cddcd1e58ba1be6eb1ae66f9a58a90523073/src/main/kotlin/com/github/ramonvermeulen/dbtToolkit/services/VenvInitializerService.kt#L59

So basically if there is a project interpreter set, it tries to find a dbt installation/executable within that specific venv. If there is no project interpreter set, or it cannot find dbt it will fall back to the global installation (if this exists). Let me know if this solves your problem, otherwise I will try to see if I can reproduce this.

tim-watcha commented 1 month ago

I appreciate your explanation, but I'd like to provide some additional context from a Mac user's perspective. On macOS, I installed dbt using Homebrew (brew), which is a popular package manager for Mac. For those unfamiliar, Homebrew simplifies the installation and management of software on macOS, similar to how package managers work on Linux systems.

Using Homebrew to install dbt is actually a very common and recommended approach for Mac users. In fact, dbt Cloud also suggests this method for macOS installations. Given how prevalent this setup is among Mac users, I think it would be beneficial if the plugin could consider Homebrew-installed packages as a potential source for dbt.

Interestingly, even though I encountered the error message you mentioned, I noticed that features like lineage and compiled SQL are still visible and functional. This suggests that parts of the plugin are working correctly, despite the initial error.

timvancann commented 2 weeks ago

It is very important to have the Python venv that contains the dbt version and executable that you want to use set as project python interpreter (for IntelliJ users this requires installation of the Python IntelliJ plugin).

See logic defined here:

https://github.com/ramonvermeulen/dbt-toolkit/blob/ace5cddcd1e58ba1be6eb1ae66f9a58a90523073/src/main/kotlin/com/github/ramonvermeulen/dbtToolkit/services/VenvInitializerService.kt#L59

So basically if there is a project interpreter set, it tries to find a dbt installation/executable within that specific venv. If there is no project interpreter set, or it cannot find dbt it will fall back to the global installation (if this exists). Let me know if this solves your problem, otherwise I will try to see if I can reproduce this.

Thanks for this, that setup my project correctly. Though I had to reboot the IDE before it took effect. Maybe it's worth rechecking the interpreter before a dbt command? On big mono-repos, or when having multiple independent code locations, it's also common to switch interpreters occasionally. It'd be nice if the pluging would work without rebooting the IDE.

ramonvermeulen commented 2 weeks ago

It is very important to have the Python venv that contains the dbt version and executable that you want to use set as project python interpreter (for IntelliJ users this requires installation of the Python IntelliJ plugin). See logic defined here: https://github.com/ramonvermeulen/dbt-toolkit/blob/ace5cddcd1e58ba1be6eb1ae66f9a58a90523073/src/main/kotlin/com/github/ramonvermeulen/dbtToolkit/services/VenvInitializerService.kt#L59

So basically if there is a project interpreter set, it tries to find a dbt installation/executable within that specific venv. If there is no project interpreter set, or it cannot find dbt it will fall back to the global installation (if this exists). Let me know if this solves your problem, otherwise I will try to see if I can reproduce this.

Thanks for this, that setup my project correctly. Though I had to reboot the IDE before it took effect. Maybe it's worth rechecking the interpreter before a dbt command? On big mono-repos, or when having multiple independent code locations, it's also common to switch interpreters occasionally. It'd be nice if the pluging would work without rebooting the IDE.

Thanks for this update, I think this is indeed something to consider. Not sure why I didn't implement it like that in the first place, maybe I thought there could be a performance overhead but I think in reality this overhead will be minimal and outweighs the convenience of not restarting your IDE. When I have time I will test this out, I think it is quite a big win for minimal code change.

Currently it calls this initializeEnvironment() just one time when executing the first dbt command during an IDE session, and from then on re-uses that dbt executable path for the lifetime of the IDE session. https://github.com/ramonvermeulen/dbt-toolkit/blob/b46f3b42ffdf130b08fd800ebd8651407df5ab1c/src/main/kotlin/com/github/ramonvermeulen/dbtToolkit/services/DbtCommandExecutorService.kt#L37

In general (especially the Kotlin) codebase is a bit messy and suffered at some places because it was the first time for me doing something with Kotlin (trying to improve on that 🫠).

ramonvermeulen commented 2 weeks ago

Solved in v0.2.0 https://github.com/ramonvermeulen/dbt-toolkit/pull/191