padeoe / hf-mirror-site

a huggingface mirror site.
https://hf-mirror.com
220 stars 26 forks source link

hf-mirror.com should forward the "Content-Length" header from huggingface.co #31

Closed blurgyy closed 3 months ago

blurgyy commented 3 months ago

As the title notes, hf-mirror.com should forward the "Content-Length" header. The missing header is causing huggingface-cli to emit errors like huggingface_hub.utils._errors.FileMetadataError: Distant resource does not have a Content-Length., and later a less helpful error huggingface_hub.utils._errors.LocalEntryNotFoundError: An error happened while trying to locate the file on the Hub and we cannot find the requested files in the local cache. Please check your connection and try again or make sure your Internet connection is on.

To reproduce (tested with huggingface-hub 0.23.3):

$ huggingface-cli download microsoft/Florence-2-large --resume-download

It will first download some files (17 on my machine) and aborts due to the above mentioned error.

To verify that hf-mirror.com currently does not forward the "Content-Length" header:

  1. Verify that the "Content-Length" header exists from huggingface.co:

    $ curl -IL https://huggingface.co/microsoft/Florence-2-large/resolve/aec3bc57822662953317bc228d279bad98b7819c/modeling_florence2.py | grep -i "content-length"
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
      0  123k    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
    Content-Length: 126723
  2. Request the same file from hf-mirror.com and observe no "Content-Length" header:

    $ curl https://hf-mirror.com/microsoft/Florence-2-large/resolve/aec3bc57822662953317bc228d279bad98b7819c/modeling_florence2.py -IL | grep -i content-length
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
      0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
padeoe commented 3 months ago

OK, I will fix it

padeoe commented 3 months ago

Fixed!