error need either state_dict or a save folder

ABDULLAHHG commented 1 month ago

System Info

transformers version: 4.43.3
Platform: Linux-5.15.0-113-generic-x86_64-with-glibc2.35
Python version: 3.11.7
Huggingface_hub version: 0.24.5
Safetensors version: 0.4.3
Accelerate version: 0.21.0
Accelerate config: not found
PyTorch version (GPU?): 2.4.0+cu121 (False)
Tensorflow version (GPU?): not installed (NA)
Flax version (CPU?/GPU?/TPU?): not installed (NA)
Jax version: not installed
JaxLib version: not installed
Using distributed or parallel set-up in script?:

Who can help?

This is code


import transformers
import torch

def Response(text):
    model_id = "meta-llama/Meta-Llama-3.1-8B"
    pipeline = transformers.pipeline(
    "text-generation", model=model_id, 
    model_kwargs={"torch_dtype": torch.bfloat16}, 
    device_map="auto",
    offload_folder="save_folder" # the error will show up with this code or without this code 
    )

    output = pipeline(text)

    return output[0]['generated_text']

print(Response("hi"))

This is error

i got this error on llama3.1

/home/aboud/anaconda3/envs/pt/lib/python3.11/site-packages/bitsandbytes/cextension.py:34: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable. warn("The installed version of bitsandbytes was compiled without GPU support. " /home/aboud/anaconda3/envs/pt/lib/python3.11/site-packages/bitsandbytes/libbitsandbytes_cpu.so: undefined symbol: cadam32bit_grad_fp32 Loading checkpoint shards: 100%|██████████████████| 4/4 [00:00<00:00, 15.72it/s] Loading checkpoint shards: 100%|██████████████████| 4/4 [00:00<00:00, 13.49it/s] Traceback (most recent call last): File "/home/aboud/Desktop/testLlama.py", line 20, in print(Response("hi")) ^^^^^^^^^^^^^^ File "/home/aboud/Desktop/testLlama.py", line 8, in Response pipeline = transformers.pipeline( ^^^^^^^^^^^^^^^^^^^^^^ File "/home/aboud/anaconda3/envs/pt/lib/python3.11/site-packages/transformers/pipelines/init.py", line 895, in pipeline framework, model = infer_framework_load_model( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/aboud/anaconda3/envs/pt/lib/python3.11/site-packages/transformers/pipelines/base.py", line 296, in infer_framework_load_model raise ValueError( ValueError: Could not load model meta-llama/Meta-Llama-3.1-8B with any of the following classes: (<class 'transformers.models.auto.modeling_auto.AutoModelForCausalLM'>, <class 'transformers.models.llama.modeling_llama.LlamaForCausalLM'>). See the original errors:

while loading with AutoModelForCausalLM, an error is thrown: Traceback (most recent call last): File "/home/aboud/anaconda3/envs/pt/lib/python3.11/site-packages/transformers/pipelines/base.py", line 283, in infer_framework_load_model model = model_class.from_pretrained(model, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/aboud/anaconda3/envs/pt/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py", line 564, in from_pretrained return model_class.from_pretrained( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/aboud/anaconda3/envs/pt/lib/python3.11/site-packages/transformers/modeling_utils.py", line 3990, in from_pretrained dispatch_model(model, device_map_kwargs) File "/home/aboud/anaconda3/envs/pt/lib/python3.11/site-packages/accelerate/big_modeling.py", line 364, in dispatch_model weights_map = OffloadedWeightsLoader( ^^^^^^^^^^^^^^^^^^^^^^^ File "/home/aboud/anaconda3/envs/pt/lib/python3.11/site-packages/accelerate/utils/offload.py", line 150, in init raise ValueError("Need either a state_dict or a save_folder containing offloaded weights.") ValueError: Need either a state_dict or a save_folder containing offloaded weights.

while loading with LlamaForCausalLM, an error is thrown: Traceback (most recent call last): File "/home/aboud/anaconda3/envs/pt/lib/python3.11/site-packages/transformers/pipelines/base.py", line 283, in infer_framework_load_model model = model_class.from_pretrained(model, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/aboud/anaconda3/envs/pt/lib/python3.11/site-packages/transformers/modeling_utils.py", line 3990, in from_pretrained dispatch_model(model, device_map_kwargs) File "/home/aboud/anaconda3/envs/pt/lib/python3.11/site-packages/accelerate/big_modeling.py", line 364, in dispatch_model weights_map = OffloadedWeightsLoader( ^^^^^^^^^^^^^^^^^^^^^^^ File "/home/aboud/anaconda3/envs/pt/lib/python3.11/site-packages/accelerate/utils/offload.py", line 150, in init raise ValueError("Need either a state_dict or a save_folder containing offloaded weights.") ValueError: Need either a state_dict or a save_folder containing offloaded weights.

the only way that it fix it by change device_map to cpu and comment the offload

# fix it code
import transformers
import torch

def Response(text):
    model_id = "meta-llama/Meta-Llama-3.1-8B"
    pipeline = transformers.pipeline(
    "text-generation", model=model_id, 
    model_kwargs={"torch_dtype": torch.bfloat16}, 
    device_map="cpu", # just change it to cpu 
    # offload_folder="save_folder" 
    )

    output = pipeline(text)

    return output[0]['generated_text']

print(Response("hi"))

i dont know why my gpu is integrated intel 3000 hd so it should ignore it i try it on linux arm64 system and it work well after 10 mintues of waiting and device_map set on auto

Information

[ ] The official example scripts
[ ] My own modified scripts

Tasks

[ ] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
[ ] My own task or dataset (give details below)

Reproduction

I just run the code

Expected behavior

It should give a clear error about what should i do I know it was because my gpu cause i run the same code on my linux arm64 system and its work if i wasn't know idk what i will do next I will continue searching on this error

And tyvm

amyeroberts commented 1 month ago

cc @muellerzr @SunMarc

github-actions[bot] commented 2 days ago

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

huggingface / transformers