search

modelscope / ms-swift

Use PEFT or Full-parameter to finetune 300+ LLMs or 80+ MLLMs. (Qwen2, GLM4v, Internlm2.5, Yi, Llama3.1, Llava-Video, Internvl2, MiniCPM-V-2.6, Deepseek, Baichuan2, Gemma2, Phi3-Vision, ...)
https://swift.readthedocs.io/zh-cn/latest/Instruction/index.html
Apache License 2.0
3.39k stars 290 forks source link

infer Qwen1.5-72B出现问题 #681

Closed HJT9328 closed 5 months ago

HJT9328 commented 5 months ago

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \

swift infer \ --model_id_or_path /data/LLM_model/Qwen1.5-72B-Chat \ --use_flash_attn true \ --model_cache_dir /data/LLM_model/Qwen1.5-72B-Chat \ --model_type qwen1hflaf-72b-Chat 这个是我的脚本， Traceback (most recent call last): File "/root/anaconda3/envs/baichuan2/lib/python3.8/site-packages/swift/cli/infer.py", line 5, in infer_main() File "/root/anaconda3/envs/baichuan2/lib/python3.8/site-packages/swift/utils/run_utils.py", line 25, in x_main args, remaining_argv = parse_args(args_class, argv) File "/root/anaconda3/envs/baichuan2/lib/python3.8/site-packages/swift/utils/utils.py", line 98, in parse_args args, remaining_args = parser.parse_args_into_dataclasses( File "/root/anaconda3/envs/baichuan2/lib/python3.8/site-packages/transformers/hf_argparser.py", line 338, in parse_args_into_dataclasses obj = dtype(**inputs) File "", line 53, in init File "/root/anaconda3/envs/baichuan2/lib/python3.8/site-packages/swift/llm/utils/argument.py", line 649, in __post_init__ set_model_type(self) File "/root/anaconda3/envs/baichuan2/lib/python3.8/site-packages/swift/llm/utils/argument.py", line 982, in set_model_type assert args.model_cache_dir is None

hjh0119 commented 5 months ago 
export modelscope_cache = /data/LLM_model
HJT9328 commented 5 months ago

解决了这个问题 使用本地模型 RAY_memory_monitor_refresh_ms=0 CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 swift infer \ --model_type qwen1half-72b-chat \ --model_id_or_path /data/xxxxx \ --infer_backend vllm --tensor_parallel_size 8