search

shailja-thakur / VGen

Apache License 2.0
114 stars 8 forks source link

RuntimeError: Expected one of cpu, cuda, xpu, mkldnn, opengl, opencl, ideep, hip, msnpu, xla, vulkan device type at start of device string: meta #4

Closed chennyang closed 1 year ago

chennyang commented 1 year ago

Hi, I tryed to run with cpu. when i execute the following command： device = torch.device("cpu") model = AutoModelForCausalLM.from_pretrained(model_name).to(device) error occured:

Traceback (most recent call last): File "", line 1, in File "/home/cyang.zhou/anaconda3/envs/pytorch/lib/python3.9/site-packages/transformers/models/auto/auto_factory.py", line 493, in from_pretrained return model_class.from_pretrained( File "/home/cyang.zhou/anaconda3/envs/pytorch/lib/python3.9/site-packages/transformers/modeling_utils.py", line 2903, in from_pretrained ) = cls._load_pretrained_model( File "/home/cyang.zhou/anaconda3/envs/pytorch/lib/python3.9/site-packages/transformers/modeling_utils.py", line 3061, in _load_pretrained_model id_tensor = id_tensor_storage(tensor) if tensor.device != torch.device("meta") else id(tensor) RuntimeError: Expected one of cpu, cuda, xpu, mkldnn, opengl, opencl, ideep, hip, msnpu, xla, vulkan device type at start of device string: meta

Can you tell me what should I do?

shailja-thakur commented 1 year ago

They are memory intensive models, cpu will not be enough. Try on GPUs, Infact I have mentioned the particular GPUs which works in case of these models

Thanks

On Fri, Aug 11, 2023, 5:33 a.m. chennyang @.***> wrote:

Hi, I tryed to run with cpu. when i execute the following command：

device = torch.device("cpu") model = AutoModelForCausalLM.from_pretrained(model_name).to(device) error occured:

Traceback (most recent call last): File "", line 1, in File "/home/cyang.zhou/anaconda3/envs/pytorch/lib/python3.9/site-packages/transformers/models/auto/auto_factory.py", line 493, in from_pretrained return model_class.from_pretrained( File "/home/cyang.zhou/anaconda3/envs/pytorch/lib/python3.9/site-packages/transformers/modeling_utils.py", line 2903, in from_pretrained ) = cls._load_pretrained_model( File "/home/cyang.zhou/anaconda3/envs/pytorch/lib/python3.9/site-packages/transformers/modeling_utils.py", line 3061, in _load_pretrained_model id_tensor = id_tensor_storage(tensor) if tensor.device != torch.device("meta") else id(tensor) RuntimeError: Expected one of cpu, cuda, xpu, mkldnn, opengl, opencl, ideep, hip, msnpu, xla, vulkan device type at start of device string: meta

Can you tell me what should I do?

— Reply to this email directly, view it on GitHub https://github.com/shailja-thakur/VGen/issues/4, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAXKPBN65TYFGRBXHG3Y7FTXUX36RANCNFSM6AAAAAA3MVE2JQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>

chennyang commented 1 year ago

Thank you for your reply, but our condition is not satisfied now, so we can only try it under the CPU. And could you please tell me is there a way to solve this problem?

shailja-thakur commented 1 year ago

Hi, LLMs are known to work because they are ginormous in size and demand intensive computing. And the problem of bringing down the compute and memory requirement is an open problem, let alone using it on merely CPUs. Some ways help you to optimize memory and resource usage such as deepspeed, accelerators, but none of them bring the usage down to just using CPUs for training/inference, especially if the model has billions of parameters.

Hope that helps answer your question.

Thank You

On Fri, Aug 11, 2023 at 8:54 PM chennyang @.***> wrote:

Thank you for your reply, but our condition is not satisfied now, so we can only try it under the CPU. And could you please tell me is there a way to solve this problem?

— Reply to this email directly, view it on GitHub https://github.com/shailja-thakur/VGen/issues/4#issuecomment-1675579416, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAXKPBMYVH4HBBZ6AN7SI43XU3H37ANCNFSM6AAAAAA3MVE2JQ . You are receiving this because you commented.Message ID: @.***>

chennyang commented 1 year ago

Thanks a lot for your reply, and it will be very useful for me. I have tried it on GPU，and I found that the error I mentioned at the beginning is caused by the lower version of the pytorch, now the problem has been solved.

Thank you