search

efeslab / Atom

[MLSys'24] Atom: Low-bit Quantization for Efficient and Accurate LLM Serving
259 stars 21 forks source link

error：same device #8

Closed muzi0111 closed 6 months ago

muzi0111 commented 6 months ago

"I would like to ask if there is a solution to this problem, as the error occurred without any changes to the code." :RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument mat2 in method wrapper_CUDA_mm)

happierpig commented 6 months ago

Hi @muzi0111,

Thanks for your interest in our project. However, it will be better if you can provide more information to help us locate the issue. Could you please give some scripts on how to reproduce the issue? Like what is the exact command or which kernel is causing the RuntimeError. Thanks!