Downloads the whole hf repo

pytorch-labs / gpt-fast

Simple and efficient pytorch-native transformer text generation in <1000 LOC of python.

BSD 3-Clause "New" or "Revised" License

5.37k stars 488 forks source link

Open das-projects opened 7 months ago

das-projects commented 7 months ago

The scripts/download.py downloads the whole hf repo including the .safetensors files, even though it seems only the .bin files are required.

yhyu13 commented 7 months ago

It usually a bad practice for repo creator to make multiple copies of the same file just different serialization format, like .bin and .safetensor.

usually, hf transformer perfers .safetensor format becuase loading .bin could lead to executing harmful code if .bin is hacked.

The hf transformer provides python inferface for downloading specific formats from hf repo, but I don't think this project use that interface

bohea commented 7 months ago

snapshot_download(repo_id="", ignore_patterns=["*. safetensors"])

will do