How to use multi gpus in qwen2/quantize.py?

Tlntin / Qwen-TensorRT-LLM

MIT License

563 stars 51 forks source link

How to use multi gpus in qwen2/quantize.py? #105

Closed qy1026 closed 4 months ago

qy1026 commented 4 months ago

Single gpu(48GB) can't support qwen1.5-72B-awq-int4 to be converted to .pt format, which will trigger an OOM error. How can I use multi gpus in qwen2/quantize.py?

Tlntin commented 4 months ago

unmark the line 113 and 114 may work it. link

Tlntin commented 4 months ago

already fix