search

OpenCSGs / csghub-sdk

3 stars 2 forks source link

2.2G 的模型文件下载了4G 多，最终提示文件不完整 #8

Closed Rader closed 1 day ago

Rader commented 1 week ago

python load_model.py

== 错误提示

pycsghub.errors.FileDownloadError: File pytorch_model.bin download incomplete, content_length: 2391002734 but the file downloaded length: 4062432878, please download again

== 完整日志

Downloading .gitattributes: 100%|████████████████████████| 1.66k/1.66k [00:00<00:00, 1.81MB/s]
Downloading README.md: 100%|█████████████████████████████████| 465/465 [00:00<00:00, 1.31MB/s]
Downloading added_tokens.json: 100%|████████████████████████| 60.0/60.0 [00:00<00:00, 230kB/s]
Downloading config.json: 100%|███████████████████████████████| 692/692 [00:00<00:00, 2.70MB/s]
Downloading generation_config.json: 100%|█████████████████████| 125/125 [00:00<00:00, 553kB/s]
Downloading image.png: 100%|███████████████████████████████| 263k/263k [00:00<00:00, 4.54MB/s]
Downloading pytorch_model.bin:  67%|██████████████▊       | 1.50G/2.23G [37:56<18:33, 705kB/s]
Downloading pytorch_model.bin:  70%|█████████████▎     | 1.56G/2.23G [17:31<3:10:58, 62.8kB/s]
Downloading pytorch_model.bin: 2.65GB [16:23, 1.20MB/s]                                       
Downloading pytorch_model.bin: 2.65GB [16:23, 1.20MB/s]
Downloading pytorch_model.bin: 3.78GB [33:36, 1.19MB/s]^[[A^[[B
Traceback (most recent call last):
  File "/private/tmp/csghub-sdk-demo/load_model.py", line 10, in <module>
    model = AutoModelForCausalLM.from_pretrained(repo_id)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/leida/miniconda3/lib/python3.11/site-packages/pycsghub/repo_reader/model/huggingface/model_auto.py", line 19, in from_pretrained
    path = Path(snapshot_download(pretrained_model_name_or_path, token=token))
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/leida/miniconda3/lib/python3.11/site-packages/pycsghub/snapshot_download.py", line 87, in snapshot_download
    http_get(
  File "/Users/leida/miniconda3/lib/python3.11/site-packages/pycsghub/file_download.py", line 199, in http_get
    raise FileDownloadError(msg)
pycsghub.errors.FileDownloadError: File pytorch_model.bin download incomplete, content_length: 2391002734 but the                             file downloaded length: 4062432878, please download again
Rader commented 1 week ago

http_get 方法在下载失败重试时一直往文件后面追加写入，没有做 offset 的回溯