current `device_map` had weights offloaded to the disk

openlm-research / open_llama

OpenLLaMA, a permissively licensed open source reproduction of Meta AI’s LLaMA 7B trained on the RedPajama dataset

Apache License 2.0

7.29k stars 372 forks source link

current `device_map` had weights offloaded to the disk #37

Open xding2 opened 1 year ago

xding2 commented 1 year ago

Can someone help me to know how to solve this issue, thanks so much in advanced!

ValueError: The current device_map had weights offloaded to the disk. Please provide an offload_folder for them. Alternatively, make sure you have safetensors installed if the model you are using offers the weights in this format.

The code that is causing my problem is:

model = LlamaForCausalLM.from_pretrained(
    model_path, local_files_only=True, torch_dtype=torch.float16, device_map="auto"
)

mhenrichsen commented 1 year ago

The model probably doesn't fit in your VRAM and therefore is offloaded to a folder.