Closed haoranlee closed 1 month ago
Hi @haoranlee, This is a valid use case; thanks for logging it and offering to implement a solution! We'd love to see this contribution, please, let us know if you'd like to discuss it any further or if have any issues with the local development setup.
Thanks @tatiana. We actually have just found a workaround, so I'll be closing this issue.
It basically involves making use of environmental variables to expand the relative path out into the corresponding absolute path, which can differ between the local and production environments.
So the packages.yml
file would have the following format for local packages:
packages:
- local: "{{ env_var('DBT_PROFILES_DIR') }}/../../dbt/packages/my_package"
Description
Similar to #715, in our environment we avoid dbt packages from external sources, and limit the use of packages to local packages. Our local dbt packages are located at a parent directory, with the dbt projects that share use of the package located in subdirectories, and these dbt projects use relative paths when specifying the package in the
packages.yml
file. Unlike #715 however, we do not have full control over the build process, and it is unfeasible to modify the build process for manually installing our local dbt packages.The concern here is that local packages currently do not work with Cosmos because the project directory is copied or symlinked out to a temporary directory prior to performing any dbt operation. As a result, when the local dbt package is subsequently installed, the symlink from
dbt deps
points out of the temporary directory to a nonexistent location relative to the project directory, and subsequent dbt operations continue to insist that0 package(s) installed in dbt_packages
.Use case/motivation
Be able to use local dbt packages with Cosmos, even when
packages.yml
files use relative paths instead of absolute paths when pointing to the packagesUnsure if this use case might be too niche, hence have not started working on a PR.
Related issues
No response
Are you willing to submit a PR?