Open lomoonmoonbird opened 5 months ago
通过TGI托管的模型，启动方式为： BNB_CUDA_VERSION=122 CUDA_VISIBLE_DEVICES=0,1 text-generation-launcher --model-id /data/llms/codeshell-7b-chat --tokenizer-config-path /data/llms/codeshell-7b-chat/tokenizer_config.json --sharded true --trust-remote-code --port=8080
CUDA_VISIBLE_DEVICES=0,1 和 --sharded true 设置后报错： RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1!
通过TGI托管的模型，启动方式为： BNB_CUDA_VERSION=122 CUDA_VISIBLE_DEVICES=0,1 text-generation-launcher --model-id /data/llms/codeshell-7b-chat --tokenizer-config-path /data/llms/codeshell-7b-chat/tokenizer_config.json --sharded true --trust-remote-code --port=8080
CUDA_VISIBLE_DEVICES=0,1 和 --sharded true 设置后报错： RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1!