search

huggingface / huggingface_hub

The official Python client for the Huggingface Hub.
https://huggingface.co/docs/huggingface_hub
Apache License 2.0
1.82k stars 470 forks source link

`huggingface_hub download` failing when grandparent of the destination folder is not writable #2359

Open kyujin-cho opened 4 days ago

kyujin-cho commented 4 days ago

Describe the bug

While trying to download the model by the help of download command, I have found that setting model_dir path to a folder residing right under the root (e.g. /models) makes the process to fail.

Seems like the specific line inside the implementation of _chmod_and_move() function, which assumes that the grandparent folder of the original destination (dst.parent.parent) is always writable, causes the error: https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/file_download.py#L1936

Reproduction

Create a writable folder under root (e.g. /models) and try to store the artifact under the destination like huggingface_hub download --model-dir /models https://huggingface.co/openai/whisper-tiny

Logs

Fetching 16 files:   0%|          | 0/16 [00:00<?, ?it/s]Downloading 'generation_config.json' to '/models/.cache/huggingface/download/generation_config.json.4b26dd66b8f7bca37d851d259fdc118315cacc62.incomplete'
Downloading 'model.safetensors' to '/models/.cache/huggingface/download/model.safetensors.7ebd0e69e78190ffe1438491fa05cc1f5c1aa3a4c4db3bc1723adbb551ea2395.incomplete'
Downloading '.gitattributes' to '/models/.cache/huggingface/download/.gitattributes.3476589a58942b0e9ef94ae0efb29355707f95bb.incomplete'
Downloading 'merges.txt' to '/models/.cache/huggingface/download/merges.txt.6038932a2a1f09a66991b1c2adae0d14066fa29e.incomplete'
Downloading 'config.json' to '/models/.cache/huggingface/download/config.json.417aa9de49a132dd3eb6a56d3be2718b15f08917.incomplete'
Downloading 'added_tokens.json' to '/models/.cache/huggingface/download/added_tokens.json.e3d256c988462aa153dcabe2aa38b8e9b436c06f.incomplete'
Downloading 'README.md' to '/models/.cache/huggingface/download/README.md.0ebbf1f524ea117e0fd826f593cde4be3897f70f.incomplete'
Downloading 'flax_model.msgpack' to '/models/.cache/huggingface/download/flax_model.msgpack.a695f537d9f018eb43a9eb05bc22f4524351061bbf2890b2c01ac9b51b43ff4e.incomplete'
Download complete. Moving file to /models/generation_config.json
Download complete. Moving file to /models/.gitattributes

Fetching 16 files:   0%|          | 0/16 [00:00<?, ?it/s]
Download complete. Moving file to /models/config.json
Download complete. Moving file to /models/added_tokens.json
Download complete. Moving file to /models/README.md
Downloading 'normalizer.json' to '/models/.cache/huggingface/download/normalizer.json.dd6ae819ad738ac1a546e9f9282ef325c33b9ea0.incomplete'
Downloading 'preprocessor_config.json' to '/models/.cache/huggingface/download/preprocessor_config.json.c2048dfa9fd94a052e62e908d2c4dfb18534b4d2.incomplete'
Download complete. Moving file to /models/preprocessor_config.json
Download complete. Moving file to /models/normalizer.json
Download complete. Moving file to /models/merges.txt
Download complete. Moving file to /models/flax_model.msgpack
Download complete. Moving file to /models/model.safetensors
Traceback (most recent call last):
  File "/home/work/.local/lib/python3.9/site-packages/huggingface_hub/file_download.py", line 1938, in _chmod_and_move
    tmp_file.touch()
  File "/usr/lib/python3.9/pathlib.py", line 1315, in touch
    fd = self._raw_open(flags, mode)
  File "/usr/lib/python3.9/pathlib.py", line 1127, in _raw_open
    return self._accessor.open(self, flags, mode)
PermissionError: [Errno 13] Permission denied: '/tmp_92b4d9da-96a0-4f3e-a780-977f39964607'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/work/.local/bin/huggingface-cli", line 8, in <module>
    sys.exit(main())
  File "/home/work/.local/lib/python3.9/site-packages/huggingface_hub/commands/huggingface_cli.py", line 53, in main
    service.run()
  File "/home/work/.local/lib/python3.9/site-packages/huggingface_hub/commands/download.py", line 146, in run
    print(self._download())  # Print path to downloaded files
  File "/home/work/.local/lib/python3.9/site-packages/huggingface_hub/commands/download.py", line 180, in _download
    return snapshot_download(
  File "/home/work/.local/lib/python3.9/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
  File "/home/work/.local/lib/python3.9/site-packages/huggingface_hub/_snapshot_download.py", line 299, in snapshot_download
    thread_map(
  File "/home/work/.local/lib/python3.9/site-packages/tqdm/contrib/concurrent.py", line 69, in thread_map
    return _executor_map(ThreadPoolExecutor, fn, *iterables, **tqdm_kwargs)
  File "/home/work/.local/lib/python3.9/site-packages/tqdm/contrib/concurrent.py", line 51, in _executor_map
    return list(tqdm_class(ex.map(fn, *iterables, chunksize=chunksize), **kwargs))
  File "/home/work/.local/lib/python3.9/site-packages/tqdm/std.py", line 1181, in __iter__
    for obj in iterable:
  File "/usr/lib/python3.9/concurrent/futures/_base.py", line 608, in result_iterator
    yield fs.pop().result()
  File "/usr/lib/python3.9/concurrent/futures/_base.py", line 445, in result
    return self.__get_result()
  File "/usr/lib/python3.9/concurrent/futures/_base.py", line 390, in __get_result
    raise self._exception
  File "/usr/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/work/.local/lib/python3.9/site-packages/huggingface_hub/_snapshot_download.py", line 273, in _inner_hf_hub_download
    return hf_hub_download(
  File "/home/work/.local/lib/python3.9/site-packages/huggingface_hub/utils/_deprecation.py", line 101, in inner_f
    return f(*args, **kwargs)
  File "/home/work/.local/lib/python3.9/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
  File "/home/work/.local/lib/python3.9/site-packages/huggingface_hub/file_download.py", line 1220, in hf_hub_download
    return _hf_hub_download_to_local_dir(
  File "/home/work/.local/lib/python3.9/site-packages/huggingface_hub/file_download.py", line 1505, in _hf_hub_download_to_local_dir
    _download_to_tmp_and_move(
  File "/home/work/.local/lib/python3.9/site-packages/huggingface_hub/file_download.py", line 1912, in _download_to_tmp_and_move
    _chmod_and_move(incomplete_path, destination_path)
  File "/home/work/.local/lib/python3.9/site-packages/huggingface_hub/file_download.py", line 1942, in _chmod_and_move
    tmp_file.unlink()
  File "/usr/lib/python3.9/pathlib.py", line 1354, in unlink
    self._accessor.unlink(self)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp_92b4d9da-96a0-4f3e-a780-977f39964607'

System info

- huggingface_hub version: 0.24.0.dev0
- Platform: Linux-5.15.0-107-generic-x86_64-with-glibc2.35
- Python version: 3.11.9
- Running in iPython ?: No
- Running in notebook ?: No
- Running in Google Colab ?: No
- Token path ?: /home/work/.cache/huggingface/token
- Has saved token ?: False
- Configured git credential helpers: 
- FastAI: N/A
- Tensorflow: N/A
- Torch: N/A
- Jinja2: N/A
- Graphviz: N/A
- keras: N/A
- Pydot: N/A
- Pillow: N/A
- hf_transfer: N/A
- gradio: N/A
- tensorboard: N/A
- numpy: N/A
- pydantic: N/A
- aiohttp: N/A
- ENDPOINT: https://huggingface.co
- HF_HUB_CACHE: /home/work/.cache/huggingface/hub
- HF_ASSETS_CACHE: /home/work/.cache/huggingface/assets
- HF_TOKEN_PATH: /home/work/.cache/huggingface/token
- HF_HUB_OFFLINE: False
- HF_HUB_DISABLE_TELEMETRY: False
- HF_HUB_DISABLE_PROGRESS_BARS: None
- HF_HUB_DISABLE_SYMLINKS_WARNING: False
- HF_HUB_DISABLE_EXPERIMENTAL_WARNING: False
- HF_HUB_DISABLE_IMPLICIT_TOKEN: False
- HF_HUB_ENABLE_HF_TRANSFER: False
- HF_HUB_ETAG_TIMEOUT: 10
- HF_HUB_DOWNLOAD_TIMEOUT: 10

{'huggingface_hub version': '0.24.0.dev0', 'Platform': 'Linux-5.15.0-107-generic-x86_64-with-glibc2.35', 'Python version': '3.11.9', 'Running in iPython ?': 'No', 'Running in notebook ?': 'No', 'Running in Google Colab ?': 'No', 'Token path ?': '/home/work/.cache/huggingface/token', 'Has saved token ?': False, 'Configured git credential helpers': '', 'FastAI': 'N/A', 'Tensorflow': 'N/A', 'Torch': 'N/A', 'Jinja2': 'N/A', 'Graphviz': 'N/A', 'keras': 'N/A', 'Pydot': 'N/A', 'Pillow': 'N/A', 'hf_transfer': 'N/A', 'gradio': 'N/A', 'tensorboard': 'N/A', 'numpy': 'N/A', 'pydantic': 'N/A', 'aiohttp': 'N/A', 'ENDPOINT': 'https://huggingface.co', 'HF_HUB_CACHE': '/home/work/.cache/huggingface/hub', 'HF_ASSETS_CACHE': '/home/work/.cache/huggingface/assets', 'HF_TOKEN_PATH': '/home/work/.cache/huggingface/token', 'HF_HUB_OFFLINE': False, 'HF_HUB_DISABLE_TELEMETRY': False, 'HF_HUB_DISABLE_PROGRESS_BARS': None, 'HF_HUB_DISABLE_SYMLINKS_WARNING': False, 'HF_HUB_DISABLE_EXPERIMENTAL_WARNING': False, 'HF_HUB_DISABLE_IMPLICIT_TOKEN': False, 'HF_HUB_ENABLE_HF_TRANSFER': False, 'HF_HUB_ETAG_TIMEOUT': 10, 'HF_HUB_DOWNLOAD_TIMEOUT': 10}
Wauplin commented 1 day ago

Thanks @kyujin-cho for investigating and reporting this bug! This should definitely be fixed. I opened https://github.com/huggingface/huggingface_hub/pull/2367 to add a try/except around tmp_file.unlink(), hence preventing any IO issue when trying to chmod the file.

kyujin-cho commented 1 day ago

Thanks for the response. Glad to hear that!