[Feature] Provide compatibility with local dbt packages

[haoranlee]

Description

Similar to #715, in our environment we avoid dbt packages from external sources, and limit the use of packages to local packages. Our local dbt packages are located at a parent directory, with the dbt projects that share use of the package located in subdirectories, and these dbt projects use relative paths when specifying the package in the packages.yml file. Unlike #715 however, we do not have full control over the build process, and it is unfeasible to modify the build process for manually installing our local dbt packages.

The concern here is that local packages currently do not work with Cosmos because the project directory is copied or symlinked out to a temporary directory prior to performing any dbt operation. As a result, when the local dbt package is subsequently installed, the symlink from dbt deps points out of the temporary directory to a nonexistent location relative to the project directory, and subsequent dbt operations continue to insist that 0 package(s) installed in dbt_packages.

Use case/motivation

Be able to use local dbt packages with Cosmos, even when

• they are located in parent directories relative to the dbt projects that use them (which makes sense for sharing of resources in the package by multiple dbt projects)
• the packages.yml files use relative paths instead of absolute paths when pointing to the packages

Unsure if this use case might be too niche, hence have not started working on a PR.

Related issues

No response

Are you willing to submit a PR?

• [X] Yes, I am willing to submit a PR!

[tatiana]

Hi @haoranlee, This is a valid use case; thanks for logging it and offering to implement a solution! We'd love to see this contribution, please, let us know if you'd like to discuss it any further or if have any issues with the local development setup.

[haoranlee]

Thanks @tatiana. We actually have just found a workaround, so I'll be closing this issue. It basically involves making use of environmental variables to expand the relative path out into the corresponding absolute path, which can differ between the local and production environments. So the packages.yml file would have the following format for local packages:

packages:
  - local: "{{ env_var('DBT_PROFILES_DIR') }}/../../dbt/packages/my_package"