Open yileld opened 6 months ago
如果想用cpu运行,请确保CUDA_VISIBLE_DEVICES=空
如果想用cpu运行,请确保CUDA_VISIBLE_DEVICES=空
是想用GPU运行的,但是有quant 8,所以AutoModel.from_pretrained()一开始是在CPU上吧
quant 8暂时不支持overwrite_arge={'model_parallel_size'}
quant 8暂时不支持overwrite_arge={'model_parallel_size'}
那难道是我记错了。。。所以目前quant是不支持多卡推理的是吧
另外我改成bf16报错
Traceback (most recent call last):
File "/ssd/ylying/CogVLM/basic_demo/infer_dataset.py", line 164, in <module>
main()
File "/ssd/ylying/CogVLM/basic_demo/infer_dataset.py", line 36, in main
model, model_args = AutoModel.from_pretrained(
File "/usr/local/lib/python3.10/dist-packages/sat/model/base_model.py", line 368, in from_pretrained
reset_random_seed(6)
File "/usr/local/lib/python3.10/dist-packages/sat/arguments.py", line 572, in reset_random_seed
assert _GLOBAL_RANDOM_SEED is not None, "You have not set random seed. No need to reset it."
AssertionError: You have not set random seed. No need to reset it.
是的,因为quant切分的状态我也不知道怎么均分到不同卡上……取决于量化算法
原来能跑起来的,现在又不行了,是sat又更新了吗? 目前版本torch=2.1.2,sat=0.4.11,transformers=4.38.2