Open KrisWongz opened 4 months ago
@KrisWongz Can you please share the arguments to the command you used when you ran into this issue?
Sure @magdyksaleh
docker run --gpus all \ --shm-size 1g \ -p 8081:80 \ -v /home/unionlab001/Model/qwen-32b:/data ghcr.io/predibase/lorax:latest \ --model-id /data/Qwen1_5-32B-Chat \ --trust-remote-code \ --quantize bitsandbytes-nf4 \ --max-batch-prefill-tokens 1024 \ --max-input-length 1024 \ --max-total-tokens 1512 \ --num-shard 4 \
Looks like this may be an issue specific to Qwen + nf4 quantization. We'll take a look.
System Info
lorax latest docker, 2 A100, unbuntu 22.04
Information
Tasks
Reproduction
When loading qwen-32b-chat.
Expected behavior
none