[BUG] dbt deps permission denied for dbt-expectations

christopherekfeldt commented 1 year ago

Is this a new bug in dbt-expectations?

[x] I believe this is a new bug in dbt-expectations
[x] I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

When running poetry run dbt deps it fails during installing of the packages on dbt-expectations giving error messages regarding permissions denied.

Expected Behavior

To dbt deps to run successfully and install all packages specified in the packages.yml

Steps To Reproduce

Our packages.yml config:

packages:

package: dbt-labs/dbt_utils version: 1.0.0
package: dbt-labs/codegen version: 0.9.0
package: elementary-data/elementary version: 0.7.1
package: calogica/dbt_expectations version: 0.8.2

run dbt deps or poetry run dbt deps

Relevant log output

08:40:51 Installed from version 0.7.1 08:40:51 Updated version available: 0.7.3
08:40:51 Installing calogica/dbt_expectations 08:40:56 Encountered an error: [Errno 13] Permission denied: 'integration_tests' 08:40:56 Traceback (most recent call last): File "/usr/lib/python3.8/shutil.py", line 791, in move os.rename(src, real_dst) PermissionError: [Errno 13] Permission denied: 'dbt_packages/dbt-expectations-0.8.2' -> 'dbt_packages/dbt_expectations' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/dmw199/.cache/pypoetry/virtualenvs/bica-cli-qaq_ldT7-py3.8/lib/python3.8/site-packages/dbt/main.py", line 136, in main results, succeeded = handle_and_check(args) File "/home/dmw199/.cache/pypoetry/virtualenvs/bica-cli-qaq_ldT7-py3.8/lib/python3.8/site-packages/dbt/main.py", line 206, in handle_and_check task, res = run_from_args(parsed) File "/home/dmw199/.cache/pypoetry/virtualenvs/bica-cli-qaq_ldT7-py3.8/lib/python3.8/site-packages/dbt/main.py", line 253, in run_from_args results = task.run() File "/home/dmw199/.cache/pypoetry/virtualenvs/bica-cli-qaq_ldT7-py3.8/lib/python3.8/site-packages/dbt/task/deps.py", line 75, in run package.install(self.config, renderer) File "/home/dmw199/.cache/pypoetry/virtualenvs/bica-cli-qaq_ldT7-py3.8/lib/python3.8/site-packages/dbt/deps/registry.py", line 60, in install self._install(project, renderer) File "/home/dmw199/.cache/pypoetry/virtualenvs/bica-cli-qaq_ldT7-py3.8/lib/python3.8/site-packages/dbt/deps/base.py", line 117, in _install connection_exception_retry(download_untar_fn, 5) File "/home/dmw199/.cache/pypoetry/virtualenvs/bica-cli-qaq_ldT7-py3.8/lib/python3.8/site-packages/dbt/utils.py", line 616, in _connection_exception_retry return fn() File "/home/dmw199/.cache/pypoetry/virtualenvs/bica-cli-qaq_ldT7-py3.8/lib/python3.8/site-packages/dbt/deps/base.py", line 129, in download_and_untar system.untar_package(tar_path, deps_path, package_name) File "/home/dmw199/.cache/pypoetry/virtualenvs/bica-cli-qaq_ldT7-py3.8/lib/python3.8/site-packages/dbt/clients/system.py", line 497, in untar_package dbt.clients.system.rename(downloaded_path, desired_path, force=True) File "/home/dmw199/.cache/pypoetry/virtualenvs/bica-cli-qaq_ldT7-py3.8/lib/python3.8/site-packages/dbt/clients/system.py", line 485, in rename shutil.move(from_path, to_path) File "/usr/lib/python3.8/shutil.py", line 809, in move rmtree(src) File "/usr/lib/python3.8/shutil.py", line 718, in rmtree _rmtree_safe_fd(fd, path, onerror) File "/usr/lib/python3.8/shutil.py", line 659, in _rmtree_safe_fd onerror(os.rmdir, fullname, sys.exc_info()) File "/usr/lib/python3.8/shutil.py", line 657, in _rmtree_safe_fd os.rmdir(entry.name, dir_fd=topfd) PermissionError: [Errno 13] Permission denied: 'integration_tests'

Environment

- OS: Windows leveraging WSL
- Python: 3.8.10
- dbt: 1.4.4
- dbt-expectations: 0.8.2

Which database adapter are you using with dbt?

dbt-bigquery 1.4.1 Note: dbt-expectations currently does not support database adapters other than the ones listed below.

Postgres
Snowflake
BigQuery

Additional Context

We have an ugly solution now that works. Steps to make it work:

Remove dbt-expectations from packages.yml
Run dbt deps
Successfull installation
Add dbt-expectations 0.8.2 to packages.yml
Run dbt deps again
Succesfull installation

clausherther commented 1 year ago

Hi, thanks for opening this issue! I can't repro this locally, so just a couple of comments:

this may or may not be related, but I don't recommend ever pinning to point releases (e.g. 0.8.2), but instead I recommend pinning to major release ranges for all packages, not just dbt-expectations, e.g.
```
packages:
- package: calogica/dbt_expectations
version: [">=0.8.0", "<0.9.0"]
```
The default instructions on dbt-hub are misleading there imo. That way you get automatic bug fixes etc.
if you're getting permissions errors during dbt deps, I think that's more likely an issue with dbt-core, or something else locally, since dbt-expectations is just a git repo you're cloning during dbt deps.
curious, what if you remove the elementary package, do you still get the error? They have some integrations with dbt-expectations and I wonder if that's causing it. They're on 0.7.5 btw.

christopherekfeldt commented 1 year ago

Thanks @clausherther So that range you put out there, how is it translated exactly?

Does it say take some version of dbt_expectations between 0.8.0 and 0.9.0 which is compatible? Or does it always select the latest? Or does it try with the latest, and if that doesn't work it tries with an older version?

clausherther commented 1 year ago

Using the range ensures you're automatically getting the latest minor release for each package, and that you don't end up with weird incompatibilities between packages.

calogica / dbt-expectations