npu 推理和部署怎么设置多卡

modelscope / ms-swift

Use PEFT or Full-parameter to finetune 350+ LLMs or 90+ MLLMs. (LLM: Qwen2.5, Llama3.2, GLM4, Internlm2.5, Yi1.5, Mistral, Baichuan2, DeepSeek, Gemma2, ...; MLLM: Qwen2-VL, Qwen2-Audio, Llama3.2-Vision, Llava, InternVL2, MiniCPM-V-2.6, GLM4v, Xcomposer2.5, Yi-VL, DeepSeek-VL, Phi3.5-Vision, ...)

Apache License 2.0

3.68k stars 315 forks source link

我的测试脚本：

NPROC_PER_NODE=8 \
ASCEND_RT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
HCCL_SOME_VARIABLE=value
swift infer \
    --model_type '/data2/dxc/Qwen1.5-32B-Chat' \
    --load_args_from_ckpt_dir true \
    --ckpt_dir '/data2/dxc/Qwen1.5-32B-Chat/v17-20240920-141406/checkpoint-6/' \
    --load_dataset_config true \
    --tensor_parallel_size 8 \
    --merge_lora false \
    --model_kwargs '{"device_map": ["npu:0", "npu:1", "npu:2", "npu:3", "npu:4", "npu:5", "npu:6", "npu:7"]}' \
    --dataset '/data2/dxc/blossom-math-v2/transformed_dataset.jsonl#100'

运行之后显示device_count:8，但是model_kwargs: {'device_map': 'npu:0'}

没找到Swift infer和Swift deploy中有关NPU的多显卡设置参数

modelscope / ms-swift

npu 推理和部署怎么设置多卡 #2084