RuntimeError: Creating a Parameter from an instance of type Params4bit requires that detach() returns an instance of the same type, but return type Tensor was found instead. To use the type as a Parameter, please correct the detach() semantics defined by its __torch_dispatch__() implementation.

[bai1535]

运行baichuan2量化模型加载的时候出现的报错，有没有大佬知道怎么解决 You shouldn't move a model when it is dispatched on multiple devices. Traceback (most recent call last): File "/home/ls/anaconda3/envs/aichat/bin/lmdeploy", line 8, in sys.exit(run()) File "/home/ls/anaconda3/envs/aichat/lib/python3.9/site-packages/lmdeploy/cli/entrypoint.py", line 37, in run args.run(args) File "/home/ls/anaconda3/envs/aichat/lib/python3.9/site-packages/lmdeploy/cli/serve.py", line 283, in api_server run_api_server(args.model_path, File "/home/ls/anaconda3/envs/aichat/lib/python3.9/site-packages/lmdeploy/serve/openai/api_server.py", line 1191, in serve VariableInterface.async_engine = pipeline_class( File "/home/ls/anaconda3/envs/aichat/lib/python3.9/site-packages/lmdeploy/serve/async_engine.py", line 212, in init self._build_pytorch(model_path=model_path, File "/home/ls/anaconda3/envs/aichat/lib/python3.9/site-packages/lmdeploy/serve/async_engine.py", line 276, in _build_pytorch self.engine = Engine(model_path=model_path, File "/home/ls/anaconda3/envs/aichat/lib/python3.9/site-packages/lmdeploy/pytorch/engine/engine.py", line 131, in init self.model_agent = AutoModelAgent.from_pretrained( File "/home/ls/anaconda3/envs/aichat/lib/python3.9/site-packages/lmdeploy/pytorch/engine/model_agent.py", line 462, in from_pretrained return build_model_agent(pretrained_model_name_or_path, File "/home/ls/anaconda3/envs/aichat/lib/python3.9/site-packages/lmdeploy/pytorch/engine/model_agent.py", line 1116, in build_model_agent model_agent = TPModelAgent(model_path, File "/home/ls/anaconda3/envs/aichat/lib/python3.9/site-packages/lmdeploy/pytorch/engine/model_agent.py", line 959, in init model, cache_engine, cache_config = self._build_model( File "/home/ls/anaconda3/envs/aichat/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, kwargs) File "/home/ls/anaconda3/envs/aichat/lib/python3.9/site-packages/lmdeploy/pytorch/engine/model_agent.py", line 1028, in _build_model model, cache_engine, cache_config = _tp_build_model( File "/home/ls/anaconda3/envs/aichat/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, *kwargs) File "/home/ls/anaconda3/envs/aichat/lib/python3.9/site-packages/lmdeploy/pytorch/engine/model_agent.py", line 745, in _tp_build_model raise e File "/home/ls/anaconda3/envs/aichat/lib/python3.9/site-packages/lmdeploy/pytorch/engine/model_agent.py", line 719, in _tp_build_model param_model = param_model.to('meta') File "/home/ls/anaconda3/envs/aichat/lib/python3.9/site-packages/accelerate/big_modeling.py", line 416, in wrapper return fn(args, kwargs) File "/home/ls/anaconda3/envs/aichat/lib/python3.9/site-packages/transformers/modeling_utils.py", line 2556, in to return super().to(*args, **kwargs) File "/home/ls/anaconda3/envs/aichat/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1160, in to return self._apply(convert) File "/home/ls/anaconda3/envs/aichat/lib/python3.9/site-packages/torch/nn/modules/module.py", line 810, in _apply module._apply(fn) File "/home/ls/anaconda3/envs/aichat/lib/python3.9/site-packages/torch/nn/modules/module.py", line 810, in _apply module._apply(fn) File "/home/ls/anaconda3/envs/aichat/lib/python3.9/site-packages/torch/nn/modules/module.py", line 810, in _apply module._apply(fn) [Previous line repeated 3 more times] File "/home/ls/anaconda3/envs/aichat/lib/python3.9/site-packages/torch/nn/modules/module.py", line 841, in _apply out_param = Parameter(param_applied, param.requires_grad) File "/home/ls/anaconda3/envs/aichat/lib/python3.9/site-packages/torch/nn/parameter.py", line 44, in new raise RuntimeError(f"Creating a Parameter from an instance of type {type(data).name} " RuntimeError: Creating a Parameter from an instance of type Params4bit requires that detach() returns an instance of the same type, but return type Tensor was found instead. To use the type as a Parameter, please correct the detach() semantics defined by its __torch_dispatch__() implementation.

[korsar4eg]

same issue