dbt-labs / dbt-core

dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.
https://getdbt.com
Apache License 2.0
9.93k stars 1.63k forks source link

[Bug] Running `dbt docs generate` gives "Access is denied" error with a read-only asset #10631

Open yuichiotsuka opened 2 months ago

yuichiotsuka commented 2 months ago

Is this a new bug in dbt-core?

Current Behavior

I have a read-only .png file that is configured in the assets folder. On the first execution of dbt docs generate there is no issue. But when succeeding executions on dbt docs generate returns an error. It seems that it cannot delete the read-only file under /targets/assets

Expected Behavior

I expect the dbt docs generate to run with no errors.

Steps To Reproduce

  1. Try this using VSCode in Windows
  2. Configure assets in dbt_project.yml
  3. Add a .png file, and set it to Read-Only
  4. Run dbt docs generate, it should work as expected
  5. For a second time, run dbt docs generate. I get a permission error.

Relevant log output

(env) PS C:\Users\Yuichi\Desktop\course\dbtlearn> dbt docs generate
08:20:54  Running with dbt=1.8.5
08:20:55  Registered adapter: snowflake=1.8.3
08:20:55  Found 8 models, 1 snapshot, 1 analysis, 9 data tests, 1 seed, 3 sources, 1 exposure, 566 macros
08:20:55
08:20:58  Concurrency: 4 threads (target='dev')
08:20:58
08:21:01  Encountered an error:
[WinError 5] Access is denied: 'C:\\Users\\Yuichi\\Desktop\\course\\dbtlearn\\target\\assets\\input_schema.png'
08:21:01  Traceback (most recent call last):
  File "C:\Users\Yuichi\Desktop\course\env\Lib\site-packages\dbt\cli\requires.py", line 138, in wrapper
    result, success = func(*args, **kwargs)
                      ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Yuichi\Desktop\course\env\Lib\site-packages\dbt\cli\requires.py", line 101, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Yuichi\Desktop\course\env\Lib\site-packages\dbt\cli\requires.py", line 218, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Yuichi\Desktop\course\env\Lib\site-packages\dbt\cli\requires.py", line 247, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Yuichi\Desktop\course\env\Lib\site-packages\dbt\cli\requires.py", line 294, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Yuichi\Desktop\course\env\Lib\site-packages\dbt\cli\requires.py", line 332, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Yuichi\Desktop\course\env\Lib\site-packages\dbt\cli\main.py", line 273, in docs_generate
    results = task.run()
              ^^^^^^^^^^
  File "C:\Users\Yuichi\Desktop\course\env\Lib\site-packages\dbt\task\docs\generate.py", line 242, in run
    shutil.rmtree(to_asset_path)
  File "C:\Progs\Python\Python311\Lib\shutil.py", line 759, in rmtree
    return _rmtree_unsafe(path, onerror)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Progs\Python\Python311\Lib\shutil.py", line 622, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "C:\Progs\Python\Python311\Lib\shutil.py", line 620, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 5] Access is denied: 'C:\\Users\\Yuichi\\Desktop\\course\\dbtlearn\\target\\assets\\input_schema.png'

Environment

- OS: Windows 11
- Python: 3.11.3
- dbt: 1.8.5

Which database adapter are you using with dbt?

snowflake

Additional Context

No response

dbeatty10 commented 2 months ago

Thanks for reaching out @yuichiotsuka !

It sounds like you are configuring asset-paths which allows you to do things like include an image from your repo in your descriptions.

Root cause and possible solution

It looks like we might be able to change the copy_function for shutil.copytree to be copyfile rather than the default of copy2. The former says it ignores metadata (like the file's permission mode) whereas the latter preserves it.

i.e., change this to be:

                shutil.copytree(asset_path, to_asset_path, copy_function=copy)

Summary and workaround

I don't sense that this would be a high priority for us, so I'd recommend avoid using read-only files in the meantime.