Multiple Projects, Unable to Locate Files

[hc1wwilger]

I have a singular dbt repo with multiple projects within. I have set the DBT_PROJECT_DIR and DBT_PROFILE_DIR environment variables both in terminal and in the dbt plugin settings. The issue is that the venv is not in the same directory as the project. We run a single venv in the root of the repo, with each project in a folder/subfolder structure also in the root. Is there a way to make this work?

When I try to compile SQL I am met with:

  Database Error in model ingest_result_flattened (models/ingest/ingest_result_flattened.sql)
    [Errno 2] No such file or directory: '*****'
Exception ignored in: <function Pool.__del__ at 0x106ae2700>
Traceback (most recent call last):
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 268, in __del__
    self._change_notifier.put(None)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/multiprocessing/queues.py", line 378, in put
    self._writer.send_bytes(obj)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes
    self._send(header + buf)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/multiprocessing/connection.py", line 373, in _send
    n = write(self._handle, buf)
OSError: [Errno 9] Bad file descriptor

[ramonvermeulen]

Hi @hc1wwilger,

Thanks for reporting this issue. I've never tested the plugin with multiple dbt projects sharing one venv, when I have time I will see if there is anything I can do to make it work. Would be nice to receive a little bit more context about your directory structure so that I might be able to reproduce the issue. For example do all these dbt projecsts share the same profiles.yml?

I assume you have something like:

venv: ~/projects/my-repository/.venv profiles: ~/projects/my-repository/profiles.yml dbt-project-a: ~/projects/my-repository/dbt-project-a dbt-project-b: ~/projects/my-repository/dbt-project-b dbt-project-c: ~/projects/my-repository/dbt-project-c

And then you open your IDE with ~/projects/my-repository as "project" folder, and select ~/projects/my-repository/.venv as active venv?

Some things you can try to solve the issue for the time being (please let me know if one of these works):

• Simply open the IDE directly with one of the dbt project directories as project root, then manually select the venv from the parent directory
• Ensure the dbt project root and target directory settings are set to the dbt project instead of the parent directory

Also could you try to copy the command that is ran by the plugin from the console window (the line that is logged with >>> <command>, take the command and see what happens if it runs manually in your terminal (try both from parent directory as dbt project directory, trying to figure out if it is a working dir related issue).

[hc1wwilger]

HI @ramonvermeulen thanks so much for responding! You are close to my project structure, with the exception that the profiles live within the projects as we a re a multi-tenant shop:

venv: ~/projects/my-repository/.venv dbt-project-a: ~/projects/my-repository/dbt-project-a/dbt-project-a profiles: ~/projects/my-repository/profiles.yml dbt-project-b: ~/projects/my-repository/dbt-project-b profiles: ~/projects/my-repository/dbt-project-b/profiles.yml dbt-project-c: ~/projects/my-repository/dbt-project-c profiles: ~/projects/my-repository/dbt-project-c/profiles.yml

As far as paths:

1. Ensure the dbt project root and target directory settings are set to the dbt project instead of the parent directory: This was what I was trying initially but could not get it to work. In addition to this, I set env vars for the project dir as well to see if that helped; It did not
2. Simply open the IDE directly with one of the dbt project directories as project root, then manually select the venv from the parent directory I'm woirking through this one now. It seems to be recognizing the models subdirectories now (no unused configuration paths), which it was not in the first attempt, however, I am still being met with the Errno2:

   13:43:26  Encountered an error:
   Runtime Error
   Database Error in model egest_aggregated_results (models/egest/egest_aggregated_results.sql)
   [Errno 2] No such file or directory: '*****'

   If I run the command in the terminal I am met with success: /Users/wwilger/IdeaProjects/dbt_insights/venv/bin/dbt --no-use-colors compile --no-populate-cache --select egest_aggregated_results

I suspect it still has something to do with the location of the venv, as if it is cd'ing to that directory or running the code from that location instead of the location of the project: Located dbt installation within the virtual environment at: /Users/wwilger/IdeaProjects/dbt_hc1_insights/venv/bin/dbt

I suspect that is due to the lines of code here, but I am not a java dev so my understanding is a bit limited: https://github.com/ramonvermeulen/dbt-toolkit/blob/f7c7d4c07ae4fb038a9336e67a9824945c902221/src/main/kotlin/com/github/ramonvermeulen/dbtToolkit/services/VenvInitializerService.kt#L56

[ramonvermeulen]

HI @ramonvermeulen thanks so much for responding! You are close to my project structure, with the exception that the profiles live within the projects as we a re a multi-tenant shop:

venv: ~/projects/my-repository/.venv dbt-project-a: ~/projects/my-repository/dbt-project-a/dbt-project-a profiles: ~/projects/my-repository/profiles.yml dbt-project-b: ~/projects/my-repository/dbt-project-b profiles: ~/projects/my-repository/dbt-project-b/profiles.yml dbt-project-c: ~/projects/my-repository/dbt-project-c profiles: ~/projects/my-repository/dbt-project-c/profiles.yml

As far as paths:

1. Ensure the dbt project root and target directory settings are set to the dbt project instead of the parent directory: This was what I was trying initially but could not get it to work. In addition to this, I set env vars for the project dir as well to see if that helped; It did not
2. Simply open the IDE directly with one of the dbt project directories as project root, then manually select the venv from the parent directory I'm woirking through this one now. It seems to be recognizing the models subdirectories now (no unused configuration paths), which it was not in the first attempt, however, I am still being met with the Errno2:

13:43:26  Encountered an error:
Runtime Error
  Database Error in model egest_aggregated_results (models/egest/egest_aggregated_results.sql)
    [Errno 2] No such file or directory: '*****'

If I run the command in the terminal I am met with success: /Users/wwilger/IdeaProjects/dbt_insights/venv/bin/dbt --no-use-colors compile --no-populate-cache --select egest_aggregated_results

I suspect it still has something to do with the location of the venv, as if it is cd'ing to that directory or running the code from that location instead of the location of the project: Located dbt installation within the virtual environment at: /Users/wwilger/IdeaProjects/dbt_hc1_insights/venv/bin/dbt

I suspect that is due to the lines of code here, but I am not a java dev so my understanding is a bit limited:

https://github.com/ramonvermeulen/dbt-toolkit/blob/f7c7d4c07ae4fb038a9336e67a9824945c902221/src/main/kotlin/com/github/ramonvermeulen/dbtToolkit/services/VenvInitializerService.kt#L56

Sorry for the late reply, I'm quite busy lately so unfortunately I have less time to work on dbt-toolkit, but as soon as I have time I will try to make this issue reproducable and see if I can do anything to solve it!

I suspect it still has something to do with the location of the venv, as if it is cd'ing to that directory or running the code from that location instead of the location of the project: Located dbt installation within the virtual environment at: /Users/wwilger/IdeaProjects/dbt_hc1_insights/venv/bin/dbt

I suspect that is due to the lines of code here, but I am not a java dev so my understanding is a bit limited:

Yes so what happens is that the plugin is trying to find a dbt executable within your virtual environment, and if it found that dbt executable it will log the line that you just mentioned. Then it will run that executable as "dbt" with the dbt project root directory that is configured in your plugin settings as working directory for the execution (this is by default the repository root).

https://github.com/ramonvermeulen/dbt-toolkit/blob/f7c7d4c07ae4fb038a9336e67a9824945c902221/src/main/kotlin/com/github/ramonvermeulen/dbtToolkit/services/ProcessExecutorService.kt#L25

In theory it should then work if you select the correct "sub" project as dbt projects root in the settings, but apparently something is not working correctly given you already tried that.